[
https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906548#comment-14906548
]
Chris Nauroth commented on HADOOP-11984:
----------------------------------------
[~vinayrpet], thank you very much for your help with rebasing this and trying
to complete it.
bq. But I am hitting some other different failures everytime.
Yes, this is exactly what I had been seeing. We're currently on revision 21 of
this patch, and each one has gotten a bit closer, but I still don't get
consistently clean runs. Progress is slow, because you effectively need to do
full test runs in parallel mode to try to get a local repro and then fix root
cause.
Of course, we have existing problems with stability of our test runs even in
single-process mode. It's hard to tell if parallel execution is really
introducing the problems.
bq. hadoop-common module seems stable with parallel tests. But hadoop-hdfs is
little giving some failures.
Maybe a good plan would be:
# Split off a patch covering Hadoop Common only. We're probably close to
stability on that part, and there is still a modest performance improvement
from doing that.
# Split off some of the individual test fixes into their own small separate
patches and get them committed. As was discussed on the dev list, these fixes
have merit on their own, even without enabling parallel tests.
# Track a separate HDFS JIRA for parallel execution of HDFS tests, and keep
working there.
Please let me know your thoughts on the plan. If it sounds good, I'd be happy
to take care of the JIRA red tape and split up the patches. :-)
> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>
> Key: HADOOP-11984
> URL: https://issues.apache.org/jira/browse/HADOOP-11984
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, scripts, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-11984.001.patch, HADOOP-11984.002.patch,
> HADOOP-11984.003.patch, HADOOP-11984.004.patch, HADOOP-11984.005.patch,
> HADOOP-11984.006.patch, HADOOP-11984.007.patch, HADOOP-11984.008.patch,
> HADOOP-11984.009.patch, HADOOP-11984.010.patch, HADOOP-11984.011.patch,
> HADOOP-11984.013.patch, HADOOP-11984.014.patch, HADOOP-11984.015.patch,
> HADOOP-11984.016.patch, HADOOP-11984.017.patch, HADOOP-11984.019.patch,
> HADOOP-11984.020.patch, HADOOP-11984.021.patch,
> HADOOP-11984.HADOOP-12111.018.patch
>
>
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile
> for running JUnit tests in multiple concurrent processes. This issue
> proposes to activate that profile during pre-commit to speed up execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
