subject:"\[jira\] \[Commented\] \(MAPREDUCE\-4980\) Parallel test execution of hadoop\-mapreduce\-client\-core"

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-09-15 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168439#comment-16168439
 ] 

Jason Lowe commented on MAPREDUCE-4980:
---

bq. it's patching code that isn't (last I checked) in branch-2.

I'm confused by that statement.  The latest 636K patch applies about 591K  of 
it as-is on branch-2.  Granted I didn't try to build it, but I'm curious what 
part makes this patch unapplicable for branch-2.  It's interesting to note that 
some of the patch conflicts are context issues because JIRAs like HADOOP-14729 
and HADOOP-10392 were applicable to branch-2 but not applied there.

There is a _lot_ of whitespace changes in this patch, so I did a comparison of 
file sizes both with and without the whitespace changes.  If the patch focuses 
just on fixing the problems with the tests wrt. making them run parallel, the 
patch is around 293K.  (I can post it if interested.)  That's less than half 
the size of the current patch, making it easier to review, easier to port to 
other branches, and more durable against other checkins to trunk.  If we really 
want to clean up whitespace that's seems better done and discussed in a 
separate JIRA given the extent of the changes rather than negatively impacting 
this one.

Other comments on patch 015:

Commented out code in the following places should be removed:
{code}
@@ -130,8 +135,8 @@ public void testMRTimelineEventHandling() throws Exception {
   + cluster.getApplicationHistoryServer().getPort());
   TimelineStore ts = cluster.getApplicationHistoryServer()
   .getTimelineStore();
-  String localPathRoot = System.getProperty("test.build.data",
-  "build/test/data");
+  String localPathRoot = GenericTestUtils.getRandomizedTempPath();
+//getTempPath(TestMRTimelineEventHandling.class.getSimpleName());
   Path inDir = new Path(localPathRoot, "input");
   Path outDir = new Path(localPathRoot, "output");
   RunningJob job =
@@ -176,9 +181,9 @@ public void testMRTimelineEventHandling() throws Exception {
   public void testMRNewTimelineServiceEventHandling() throws Exception {
 LOG.info("testMRNewTimelineServiceEventHandling start.");
 
-String testDir =
-new File("target", getClass().getSimpleName() +
-"-test_dir").getAbsolutePath();
+String testDir = GenericTestUtils.getRandomizedTestDir().getAbsolutePath();
+//new File("target", getClass().getSimpleName() +
+//"-test_dir").getAbsolutePath();
 String storageDir =
 testDir + File.separator + "timeline_service_data";
{code}

There's some TBD stuff that was added in TestMultipleLevelCaching and TestPipes 
should either be addressed or removed if unnecessary, i.e.:
{code}
-mr.waitUntilIdle();
-mr.shutdown();
+// TBD mr.waitUntilIdle();
+mr.stop();

and

-  mr.waitUntilIdle();
+  //TBD mr.waitUntilIdle();
{code}

I really like your idea of unsetting HADOOP_CONF_DIR in the pom if we can get 
away with that for the tests.  Will help keep dev environments from 
contaminating the unit tests in unexpected ways.  Best done as a separate JIRA? 
 Seems like that would apply to more than just the jobclient tests.

Thanks again for the hard work pushing this forward.  It's far from glamorous 
work, but it really is appreciated.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.015.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-09-14 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167062#comment-16167062
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

bq.  A lot of the changes are essentially whitespace and indentation reformats, 
and those changes aren't necessary to fix the unit test behaviors. Do we really 
need those changes as part of this patch?

IMO, this is an *ideal* patch to fix the checkstyle issues with, especially 
since a) it's already a huge patch and b) it's patching code that isn't (last I 
checked) in branch-2.

I'll do a rebase etc etc and post an updated version later tonight if I get a 
chance.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.015.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-09-14 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166993#comment-16166993
 ] 

Jason Lowe commented on MAPREDUCE-4980:
---

bq. Would folks be amenable if we commit a patch that basically 
fixes/modernizes the unit test code but doesn't actually enable parallel tests 
by default? 

Yes, that sounds great!  I'd hate for perfect to be the enemy of good here.  
There's some lingering issues with the parallel stuff, but in general I think 
there's great stuff here as far as making them better unit tests.

My apologies for not having had a chance to get back to this for a while, but 
I'd like to take a closer look at the patch tomorrow.  Just had a chance to 
glance at the patch so far, but one concern is that the patch is a lot bigger 
than it needs to be.  A lot of the changes are essentially whitespace and 
indentation reformats, and those changes aren't necessary to fix the unit test 
behaviors.  Do we really need those changes as part of this patch?

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.015.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-09-14 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166968#comment-16166968
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

Given the whole unit test discussion in common-dev@ today, I had a thought on 
this issue.  Would folks be amenable if we commit a patch that basically 
fixes/modernizes the unit test code but doesn't actually enable parallel tests 
by default?  

My thinking is that would at least push us significantly forward and reduce the 
amount of code to have to churn through while working out the 5 or so 
unreliable tests.  I really don't want to lose any ground on this one again.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.015.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-09-02 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151641#comment-16151641
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 144 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 24s{color} 
| {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
 generated 2 new + 95 unchanged - 48 fixed = 97 total (was 143) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 137 new + 2573 unchanged - 305 fixed = 2710 total (was 
2878) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 32m  0s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.TestMRJobClient |
|   | hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12885111/MAPREDUCE-4980.015.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 72fd12ecac1e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 275980b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7120/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| checkstyle |

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-18 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133175#comment-16133175
 ] 

Jason Lowe commented on MAPREDUCE-4980:
---

I'm not sure adding more time to TestMapReduceJobControl is going to help, as 
it looked like a surefire timeout rather than a junit timeout.  MAPREDUCE-6747 
is tracking the issue.  What I believe is happening is that the code is trying 
to kill a local job (which involves trying to interrupt a thread) and that 
interrupt gets swallowed at some point.  It then spins waiting for a job to 
exit that isn't going to exit since it wasn't successfully interrupted.  
Eventually surefire steps in and kills the whole mess without a lot of 
helpfulness.  For this JIRA I'd be tempted to slap a reasonable junit timeout 
on testJobControlWithKillJob so we don't wait the full surefire timeout when it 
is going to hang.  We can fix that test in MAPREDUCE-6747.

Still haven't been able to reproduce the TestNNBench issue, but I've only been 
able to take a couple of cracks at it in my spare time.  Will try to get a few 
more tries in the next few days.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-17 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131410#comment-16131410
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

bq. hdfs.TestNNBench 

YES I KNOW.

bq. mapreduce.lib.jobcontrol.TestMapReduceJobControl

Clock ran out:

{code}
2017-08-17 21:35:50,778 WARN  [main] util.NativeCodeLoader 
...
java.lang.InterruptedException
...
2017-08-17 21:36:48,364 INFO  [communication thread] mapred.LocalJobRunner 
(LocalJobRunner.java:statusUpdate(628)) - map > map
{code}

So I'll put more time on it for the next patch.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.013.patch, MAPREDUCE-4980.014.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131379#comment-16131379
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 144 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 34s{color} 
| {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
 generated 2 new + 95 unchanged - 48 fixed = 97 total (was 143) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
44s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 124 new + 2574 unchanged - 305 fixed = 2698 total (was 
2879) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 36s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m  6s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestNNBench |
| Timed out junit tests | 
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12882437/MAPREDUCE-4980.014.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux cab09933ee25 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / ab1a8ae |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7083/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7083/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| unit |

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131251#comment-16131251
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 144 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 26s{color} 
| {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
 generated 2 new + 95 unchanged - 48 fixed = 97 total (was 143) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
49s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 124 new + 2574 unchanged - 305 fixed = 2698 total (was 
2879) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 55s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMRTimelineEventHandling |
|   | hadoop.mapreduce.v2.TestMRJobs |
|   | hadoop.mapreduce.v2.TestUberAM |
|   | hadoop.mapred.TestLocalMRNotification |
|   | hadoop.mapred.TestMRCJCFileInputFormat |
|   | hadoop.hdfs.TestNNBench |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12882424/MAPREDUCE-4980.013.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 88cb0fc686b0 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / dd7916d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7081/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| checkstyle |

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-17 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130792#comment-16130792
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

Awesome, thanks!

I've been scratching my head going through everything wondering how your 
experience was so radically different than what I'd seen on my runs and on 
Jenkins!  I wonder if the pom.xml should unset HADOOP_CONF_DIR?  Hmm.

bq.  it was a clean run except for one "didn't cleanup" error in 
TestLocalMRNotification

I currently dream about this error. I don't think that's healthy.


> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-17 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130711#comment-16130711
 ] 

Jason Lowe commented on MAPREDUCE-4980:
---

bq. I'd need to see stack traces or something else to help diagnose further.

Doh, looks like I accidentally ran the tests with HADOOP_CONF_DIR set in the 
environment.  A lot of the tests were failing because the jobs running on the 
miniclusters failed due to picking up those configs.  I reran the unit tests 
without any HADOOP* env vars, and it was a clean run except for one "didn't 
cleanup" error in TestLocalMRNotification and a surefire timeout in 
TestMapReduceJobControl#testJobControlWithKillJob.  The latter is already 
tracked by MAPREDUCE-6747.  Sorry for the false alarm.

I'll see if I can get some time later today to try to reproduce the NNBench 
issue again.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-16 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129582#comment-16129582
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

bq. I wasn't able to reproduce the TestNNBench failure locally

It's extremely racy and seems to only show up under heavy load. Fun!  You might 
try increasing the thread count.

bq. Looks like lots of things want to scribble on the same areas in 
target/test-dir.

The test-dir/x/dfs thing is a red herring.  The test dirs get re-used by 
different threads at different times by assigning the various system properties 
when they are launched.  You'll see the same sort of thing when running 
parallel in hadoop-hdfs; it's a product of the MiniDFSCluster.  It's a little 
confusing and I sort of wish MiniDFSCluster always unique'd the directory (but 
in a smarter way than MiniYARNCluster).  On the flip side, I'm too lazy to set 
the property that gives each test it's own HDFS base dir. ;)

I'd need to see stack traces or something else to help diagnose further. If you 
see any "x didn't cleanup in 20 secs" or whatever, those are the timing issues 
I'm still working out. Finding a not-insane multiplier that works on Jenkins 
reliably has been hard. Suspect there are a lot of dead processes on those 
machines. :(

BTW, there are a TON of expected and/or "useless" stack traces (I'm looking at 
you, metrics.) in the test output.  The logging doesn't provide a lot of hints 
and so be careful about that too.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-16 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129542#comment-16129542
 ] 

Jason Lowe commented on MAPREDUCE-4980:
---

Thanks a ton for pushing this forward, Allen!

I wasn't able to reproduce the TestNNBench failure locally, but I did run into 
a lot of other failures.  Still digging through them, but a lot appear to be 
due to mini clusters colliding in the local filesystem.  For example:
{noformat}
$ grep -l 
'hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/1/dfs/name-0-1
 has been successfully formatted' target/surefire-reports/*-output.txt  
 
target/surefire-reports/org.apache.hadoop.conf.TestNoDefaultsJobConf-output.txt
target/surefire-reports/org.apache.hadoop.mapred.TestJobName-output.txt
target/surefire-reports/org.apache.hadoop.mapred.TestLazyOutput-output.txt
target/surefire-reports/org.apache.hadoop.mapred.TestMiniMRChildTask-output.txt
target/surefire-reports/org.apache.hadoop.mapred.TestMiniMRClasspath-output.txt
target/surefire-reports/org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers-output.txt
target/surefire-reports/org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge-output.txt
target/surefire-reports/org.apache.hadoop.mapreduce.TestMRJobClient-output.txt
target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestUberAM-output.txt
{noformat}

Looks like lots of things want to scribble on the same areas in target/test-dir.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-16 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128981#comment-16128981
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

Other than what are "wait enough time" problems (and that's just a matter of 
tuning), the only test that really seems troublesome is TestNNBench.

If I run at -DtestThreadCount=8, I can break this test consistently.  I'm 
completely stumped as to why though.  Whichever test runs second fails the 
create_write assert.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126748#comment-16126748
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 144 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 21s{color} 
| {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
 generated 2 new + 96 unchanged - 48 fixed = 98 total (was 144) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
47s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 124 new + 2574 unchanged - 305 fixed = 2698 total (was 
2879) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m  
0s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12881858/MAPREDUCE-4980.012.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux bf5f1858c5f8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0446511 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7070/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7070/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7070/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 U:

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-14 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126716#comment-16126716
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

-012:
* fixed all the easy checkstyle problems in code that was modified or connected 
to modified code
* changed the namenode code in builder to also take URI and FileSystem
* changed most of the tests to use FileSystem as the constructor as a result
* Added some code to scale UtilsForTests  timeout for tests that was randomly 
failing on ASF jenkins based on the number of threads being forked while in 
parallel mode.  Let's see what happens. ;)  Might still need to tweak it a bit.
* Removed the exists check from TestMapperReducerCleanup


> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.012.patch, MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-13 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125214#comment-16125214
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

bq. Could MiniMRClientClusterBuilder implement Closeable.close()? As then we 
can use IOUtils.cleanupWithLogger() to close them in teardowns. Robust & logs 
problems. Also works in try-with-resources code like TestPipes

I dug into this a bit.  Through a very long and convoluted path 
MiniMRClientClusterBuilder (including a trip through MiniYARNCluster) is an 
AbstractService which implements Closeable.close().  So this should already be 
there.  However... tests that build new clusters for every test will lose a 
chunk of their NM logs.

bq. I'd steer clear of checkstyles, as, while irritating, they're unrelated.

I'm fixing the trivial/low-effort ones.  (Like, I'm not even bothering to fix 
the StaticVarName ones.)


> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-13 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124971#comment-16124971
 ] 

Steve Loughran commented on MAPREDUCE-4980:
---

I'd steer clear of checkstyles, as, while irritating, they're unrelated. On 
those classes where you are changing the timeouts of all methods though: just 
go for a single rule & make it bigger.

The hadoop-aws ITest timeouts are property file driven, esp the scale ones...if 
you are test uploading 4GB files, you need a timeout > 1h so your test doesn't 
fail 45 minutes in.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-12 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124753#comment-16124753
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

Thanks!

bq. Is the test failure related?

Sort of.  From what I've observed is that the ASF infra can get IO bound.  This 
means that operations that normally complete quickly can get stuck in IO wait.  
If you have a threaded test that has a portion of it dependent upon IO, it may 
end up where some threads are completing but others appear "stuck". This made 
worse by noisy neighbors on the same node.  Given that this test "failed" while 
doing IO, there's a good chance that due to the extra load this test will 
appear as flaky.  In fact, for this particular test, we've seen it before: 
MAPREDUCE-4753. 

These are the sort of problems I've been fighting.  Many of the tests in 
mr-jobclient are threaded.  They bring up full or partial clusters in 
@before/@after clauses with inappropriate timeouts for these situations. 
NotificationTestCase.java (which is what TestLocalMRNotification really is) 
follows this exact same pattern.

I wouldn't be surprised if unit tests in other parts of the code base don't 
suffer from the same problem.  We might have a test "anti-pattern" happening 
and not realize it.

bq. w.r.t @Test(timeout), I much prefer having a timeout rule per class; if we 
do that rather than simply upping the value in the @Test clauses we set things 
up better for the future.

I've been debating raising the default limit in general for jobclient rather 
than fight these one by one.  But overall, yeah, I agree. Per class timing 
rules really seem like a smarter choice.

bq. Could MiniMRClientClusterBuilder implement Closeable.close()? 
bq. If MiniMRClientClusterBuilder.namenode() was overriddent to take a URI or 
FileSystem, it'd be easier to setup

In the case of closeable, my hunch is that it will be highly dependent upon 
what the YARN minicluster code does, since it's effectively a wrapper around 
it.  

I've been hesitant to fix "too much" here since the tests, despite being slow, 
are (mostly) reliable.  As I go through the patch to fix the checkstyle issues 
(mainly caused by using machine translation), the patch is getting pretty big. 
:(


> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-12 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124568#comment-16124568
 ] 

Steve Loughran commented on MAPREDUCE-4980:
---

good piece of maintenance on the tests here, they often get neglected "not a 
feature", but given how they get run by us developers more than any other piece 
of code, fast tests are wonderful.
# Is the test failure related? 
# w.r.t @Test(timeout), I much prefer having a timeout rule per class; if we do 
that rather than simply upping the value in the @Test clauses we set things up 
better for the future.
# Could {{MiniMRClientClusterBuilder}} implement {{Closeable.close()}}? As then 
we can use {{IOUtils.cleanupWithLogger()}} to close them in teardowns. Robust & 
logs problems. Also works in try-with-resources code like {{TestPipes}}
# If {{MiniMRClientClusterBuilder.namenode()}} was overriddent to take a URI or 
FileSystem, it'd be easier to setup
# {{TestMapperReducerCleanup}} no need to do an exists() check before delete(); 
its harmless if the path isn't there (and a serious regression if it does 
suddenly start complaining...)


other than that, LGTM. fine piece of work

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.011.patch, 
> MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
> MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-12 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124511#comment-16124511
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 141 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 27s{color} 
| {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
 generated 2 new + 96 unchanged - 48 fixed = 98 total (was 144) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
52s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 194 new + 2595 unchanged - 240 fixed = 2789 total (was 
2835) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 32s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 43s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestLocalMRNotification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12881602/MAPREDUCE-4980.011.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 04b91daa6d80 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8b242f0 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7067/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7067/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| unit |

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-11 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123088#comment-16123088
 ] 

Steve Loughran commented on MAPREDUCE-4980:
---

If its going through configs & is splnning up a new MR app for each one, so 
that bit does need to be isolated. And it does its best with static HDFS & YARN 
clusters


Looking at the code, the job it is starting is "SleepJob", which does as it 
says: sleeps a long time. Which is why the tests take so long. It is a series 
of tests which sleep(). That's also why they set such a big test timeout of 
their own.

It does go into the integration test category in terms of processes & work, 
except then there's a risk of it not being run on Yetus tests of MAPREDUCE 
patches, isn't it? And that could miss things.

FWIW, in hadoop-aws & soon hadoop-azure, we have a "scale" profile which runs 
the slower tests: more files, bigger files. Maybe we could have a standard 
-Dscale profile, and put this in. So if you don't run with scale, this test 
gets skipped.

thx for doing this work BTW. Welcome to the QA team

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.1.patch, 
> MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, 
> MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-10 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122733#comment-16122733
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

A few notes:

* TestMRJobs doesn't actually appear to be a unit test.  It looks more like an 
integration test.
* It currently takes about 18 minutes on my laptop to run.  It's significantly 
longer on Jenkins.
* Most of that time is spend starting up and bringing down clusters.
* Is there any reason for it to do that? i.e., why can't can't it just bring up 
one cluster, do the tests, then shut it down?



> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.1.patch, 
> MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, 
> MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-08 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118656#comment-16118656
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

On OS X/JDK 1.8.0_111:

{code}
hadoop-mapreduce-client-jobclient aw$ git rev-parse --abbrev-ref HEAD
trunk
hadoop-mapreduce-client-jobclient aw$ git diff apache/trunk
hadoop-mapreduce-client-jobclient aw$ mvn clean test -Dtest=TestMRJobs
[INFO] Scanning for projects...
[INFO] 

Running org.apache.hadoop.mapreduce.v2.TestMRJobs
Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 440.537 sec 
<<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestMRJobs)  Time 
elapsed: 43.235 sec  <<< FAILURE!
java.lang.AssertionError: No thread dump
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1215)
...
[INFO] Total time: 07:29 min
...
{code}

There's probably more going wrong with TestMRJobs in general.  (and that 
runtime yikes.)

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.1.patch, 
> MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, 
> MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-08 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118153#comment-16118153
 ] 

Steve Loughran commented on MAPREDUCE-4980:
---

if you look at the S3A (and soon WASB) parallel patches, we split into "things 
which parallelise" and "things which don't", the latter running sequentially. 
TestMRJobs probably goes into the "don't" category

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.1.patch, 
> MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, 
> MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-07 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117530#comment-16117530
 ] 

Allen Wittenauer commented on MAPREDUCE-4980:
-

Hmm.  org.apache.hadoop.mapreduce.v2.TestMRJobs is apparently a problematic 
test.  I get the same failure in trunk (without this patch) on my Mac. :(  
Looking at the past few nightly ASF trunk builds, it passes. I'm guessing this 
test is making file system assumptions that aren't true? (On the Mac, it's 
likely because /tmp isn't actually a real directory.)

I wonder if it's worthwhile to get what we can in by dropping the changes to 
pom.xml, TestMRJobs, and TestUberAM.  Then we'd just need to work on fixing 
those two jobs without having to worry about losing the rest of the changes 
again.

> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
> Attachments: MAPREDUCE-4980.010.patch, MAPREDUCE-4980.1.patch, 
> MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, 
> MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, 
> MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115948#comment-16115948
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 133 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 16s{color} 
| {color:red} root generated 10 new + 1364 unchanged - 54 fixed = 1374 total 
(was 1418) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m 
36s{color} | {color:red} root: The patch generated 234 new + 2532 unchanged - 
170 fixed = 2766 total (was 2702) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 41s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
15s{color} | {color:green} hadoop-extras in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 58s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.mapreduce.v2.TestMRJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880577/MAPREDUCE-4980.010.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 163b74ad84de 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / d91b7a8 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| javac | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7046/artifact/patchprocess/diff-compile-javac-root.txt
 |
| checkstyle |

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2017-08-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109305#comment-16109305
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} MAPREDUCE-4980 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-4980 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12611165/MAPREDUCE-4980--n8.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7034/console |
| Powered by | Apache Yetus 0.4.0   http://yetus.apache.org |


This message was automatically generated.



> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Andrey Klochkov
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
> MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
> MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the 
> tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2015-05-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524728#comment-14524728
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  1s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12611165/MAPREDUCE-4980--n8.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5533/console |


This message was automatically generated.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi Ozawa
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, 
 MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, MAPREDUCE-4980--n7.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, MAPREDUCE-4980.1.patch, 
 MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-30 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809611#comment-13809611
]

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12611165/MAPREDUCE-4980--n8.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 125
new or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4157//console

This message is automatically generated.

Parallel test execution of hadoop-mapreduce-client-core
---

Key: MAPREDUCE-4980
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch,
MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch,
MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch,
MAPREDUCE-4980.patch

The maven surefire plugin supports parallel testing feature. By using it, the
tests can be run more faster.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-30 Thread Andrey Klochkov (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809636#comment-13809636
 ] 

Andrey Klochkov commented on MAPREDUCE-4980:


The build failed due to OOM while processing native code. Not related to the 
patch.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n7.patch, MAPREDUCE-4980--n8.patch, 
 MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-22 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802190#comment-13802190
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12609712/MAPREDUCE-4980--n7.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 120 
new or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The test build failed in 
hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-extras 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4129//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4129//console

This message is automatically generated.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-22 Thread Jonathan Eagles (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802553#comment-13802553
 ] 

Jonathan Eagles commented on MAPREDUCE-4980:


Looks like many failures like this in the unit tests runs as show in the 
console log

{quote}
---
 T E S T S
---
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
{quote}

Also, this

{quote}
[WARNING] The requested profile native could not be activated because it does 
not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-mapreduce-client-jobclient: ExecutionException; nested exception 
is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The 
forked VM terminated without saying properly goodbye. VM crash or System.exit 
called ?
{quote}

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-22 Thread Jonathan Eagles (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802573#comment-13802573
 ] 

Jonathan Eagles commented on MAPREDUCE-4980:


Sorry. Those seems to with the build itself and not due to this patch.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-10-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798642#comment-13798642
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12609036/MAPREDUCE-4980--n7.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 119 
new or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix:

  org.apache.hadoop.mapred.TestJobCleanup

  The following test timeouts occurred in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix:

org.apache.hadoop.mapreduce.v2.TestUberAM

  The test build failed in 
hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-extras 

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4124//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4124//console

This message is automatically generated.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980--n6.patch, 
 MAPREDUCE-4980--n7.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-05-24 Thread Andrey Klochkov (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1303#comment-1303
 ] 

Andrey Klochkov commented on MAPREDUCE-4980:


Vinod, can you please point me to downstream components depending on 
MiniMRClientClusterFactory? Can't find such by quick grepping through Core, 
HBase, Ping, Hive.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-05-24 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1321#comment-1321
]

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12584744/MAPREDUCE-4980--n5.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 119
new or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3677//console

This message is automatically generated.

Parallel test execution of hadoop-mapreduce-client-core
---

The maven surefire plugin supports parallel testing feature. By using it, the
tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-05-12 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655746#comment-13655746
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-4980:


bq. Does this depend on HDFS-4491?
Looked at the patch. Answering my own question - Yes.

Looking at the patch based on your overview above.

One quick point: we cannot break MiniMRClientClusterFactory as that is used by 
downstream components. We should leave it alone, may be make it extend 
MiniMRClientClusterBuilder and thus limit the test-changes. Can that be done?

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-05-12 Thread Tsuyoshi OZAWA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655754#comment-13655754
 ] 

Tsuyoshi OZAWA commented on MAPREDUCE-4980:
---

Thanks, Andrey and Vinod!

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-04-17 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633846#comment-13633846
]

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12579103/MAPREDUCE-4980--n4.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 119
new or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3531//console

This message is automatically generated.

Parallel test execution of hadoop-mapreduce-client-core
---

Key: MAPREDUCE-4980
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch,
MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch

The maven surefire plugin supports parallel testing feature. By using it, the
tests can be run more faster.

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-04-17 Thread Andrey Klochkov (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633849#comment-13633849
 ] 

Andrey Klochkov commented on MAPREDUCE-4980:


The failure is expected due to dependency on HDFS-4491

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-03-29 Thread Andrey Klochkov (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617813#comment-13617813
 ] 

Andrey Klochkov commented on MAPREDUCE-4980:


I believe something's wrong with the QA robot. The patch is perfectly 
applicable to the current trunk.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, 
 MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-03-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599265#comment-13599265
 ] 

Hadoop QA commented on MAPREDUCE-4980:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12573117/MAPREDUCE-4980.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 119 
new or modified test files.

  {color:red}-1 one of tests included doesn't have a timeout.{color}

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3399//console

This message is automatically generated.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-03-11 Thread Andrey Klochkov (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599286#comment-13599286
 ] 

Andrey Klochkov commented on MAPREDUCE-4980:


Build fails due to dependency on HDFS-4491 which is not in trunk yet.

The patch affects large number of tests, setting timeout for all of them 
shouldn't be done as part of this patch.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-02-06 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572553#comment-13572553
 ] 

Steve Loughran commented on MAPREDUCE-4980:
---

Are you trying to run all MR jobs in parallel? Because I worry about conditions 
where the tests are making some assumption about unique access to the 
filesystem or mini clusters. The MiniMR and MiniDFS clusters do come up on new 
ports -but I'm not sure about the local filesystem test directories

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

45 matches

Mail list logo