[
https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551436#comment-14551436
]
Chris Nauroth commented on HADOOP-11984:
----------------------------------------
I finally was able to repro some of these {{OutOfMemoryError}} conditions
locally.
I figured out that HADOOP-11363 introduced configurability of the max heap size
for forked Surefire processes. As part of that, it also increased the default
setting from 1 GB to 4 GB, but only in the case where the build was not called
with {{-Pparallel-tests}}. 4 GB is pretty high, and the comments in
HADOOP-11363 suggest that it was going well beyond what was really needed. I'm
going to try downtuning this to 2 GB and making sure the same value gets used
for {{-Pparallel-tests}}.
We also have an extremely naughty test,
{{TestConfiguration#testConcurrentAccesses}}, that was introduced in
HADOOP-11209. It works by spinning up 100 threads, each of which tries to put
100000 key-value pairs into a {{Configuration}}. When I run this beast with
anything less than 4 GB, it goes into heavy GC churn before the threads can
complete. That's why it's been reported as a timeout in the last several runs.
(BTW, that's good validation that my change to the timed out test detection
awk script actually works.) I'm temporarily removing this test to unblock the
current testing effort while I think about a permanent fix.
I'm uploading patch v006.
> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>
> Key: HADOOP-11984
> URL: https://issues.apache.org/jira/browse/HADOOP-11984
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-11984.001.patch, HADOOP-11984.002.patch,
> HADOOP-11984.003.patch, HADOOP-11984.004.patch, HADOOP-11984.005.patch,
> HADOOP-11984.006.patch
>
>
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile
> for running JUnit tests in multiple concurrent processes. This issue
> proposes to activate that profile during pre-commit to speed up execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
