[
https://issues.apache.org/jira/browse/LUCENE-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783436#action_12783436
]
Erick Erickson commented on LUCENE-2096:
----------------------------------------
Parallelizing tests is proving trickier than I'd hoped. Part of the problem is
my not-wonderful ant skills...
But what I've found so far with trying to use ForEach is that stuff gets in the
way. In particular, the <sequential> tag in the test-macro body I'm pretty sure
defeats any parallelizing attempts by ForEach. Taking it out isn't
straightforward.
In some of my experiments, I got tests to fire off in parallel, but then
started running into wonky errors that were so strange now I can't remember
them, but some having to do with what looked like file contention for some
temporary test files.
Googling around I think I remember posts by Jason Ruthgren trying to so
something similar in SOLR (?). Jason: if I'm remembering right did you find any
joy?
Then we'd have to rework how success and failure are handled because there's
contention for that file as well.
Now I'm wondering if the "scary python script" gets us more bang for the buck.
I wrote a Groovy script the probably is a near-cousin for experiments and I'm
wondering what would happen if we wrote a special testcase-type target that did
NOT depend upon compile-test or, really, much of anything else and counted on
the user to make sure to build the system first before using whatever script
wecame up with. We don't really lose functionality by recursively looking for
Test*.java files because that's what's done internally in the build files
anyway. So doing that outside or inside the ant files doesn't seem like a loss.
I'm putting this in the JIRA issue to preserve it for posterity. Meanwhile,
I'll appeal to Ant gurus if they want to try whacking the Ant build files, and
see what the script notion brings...
> Investigate parallelizing Ant junit tests
> -----------------------------------------
>
> Key: LUCENE-2096
> URL: https://issues.apache.org/jira/browse/LUCENE-2096
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Build
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Minor
>
> Ant Contrib has a "ForEach" construct that may speed up running all of the
> Junit tests by parallelizing them with a configurable number of threads. I
> envision this in several stages. First, see if ForEach works for us with
> hard-coded lists, distribute this for testing then make the changes "for
> real". I intend to hard-code the list for the first pass, ordered by the time
> they take. This won't do for check-in, but will give us a fast
> proof-of-concept.
> This approach will be most useful for multi-core machines.
> In particular, we need to see whether the parallel tasks are isolated enough
> from each other to prevent mutual interference.
> All this assumes the fragmentary reference I found is still available...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
