I was looking at how to parallelize the tests, seems like this ANT command
would work, is there an open issue to do this?
http://ant.apache.org/manual/CoreTasks/parallel.html
We've touched on this before:
http://www.gossamer-threads.com/lists/lucene/java-dev/69669
I'd love to see a clean solution here (the tests are embarrassingly
parallelizable, and we all have machines with good concurrency these
days)... I have a rather hacked up solution now, that uses
"-Dtest
I was looking at the ant parallelize stuff too - I think that only the
very latest release has the built in parallelize tests functionality
though. Just came out a bit ago.
- Mark
Michael McCandless wrote:
We've touched on this before:
http://www.gossamer-threads.com/lists/lucene
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Rutherglen updated LUCENE-1709:
-
Summary: Parallelize Tests (was: Parallize Tests)
> Parallelize Te
p the chain
to Parallel, that might work?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> C
create a ParallelJUnitTask that extends JUnitTask and
accepts a threadCount property. We can reuse code from the threading code from
Parallel.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org
(contrib crawl can't be parallelized in the general case because some
contribs have dependencies on other contribs and build them if they aren't
already built)
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL:
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723867#action_12723867
]
Jason Rutherglen commented on LUCENE-1709:
--
Thanks for the links!
> Para
e most obvious solution belongs in the the ant world. However
I still hold out some hope for a custom interim solution.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
>
ins/maven-surefire-plugin/examples/testng.html
Look at running tests in parallel.
I havent tried this yet, but why spend time reinventing the wheel?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCE
time left when running with five threads.
Is it worth the development time and the complexity increase of the build?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
>
laris (2009.06) machine, with a Core i7 920 CPU
(= 8 cores presented to the OS) and an Intel X25M SSD, 12 GB RAM. The
hardware has quite a bit of concurrency.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apach
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-1709:
--
Fix Version/s: (was: 3.0)
3.1
> Parallelize Te
2.007975]: run "ant test-backwards -Dtestpackage=document"...
5 [0:02:38.017968]: run "ant test-core -Dtestpackage=queryParser"...
3 [0:02:40.097929]: run "ant test-backwards -Dtestpackage=queryParser"...
4 [0:02:40.151166]: DONE
0 [0:02:41.927786]: DONE
1 [0:02:43.07
tests in parallel (2 jvms per cpu)
On my computer, 'test-core' is 1:02 and 'test' is 3:29 with the patch.
Not ready for committing yet, and needs some improvements and fixes.
> Parallelize Tests
> -
>
> Key: LUCENE-1
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843385#action_12843385
]
Jason Rutherglen commented on LUCENE-1709:
--
Robert, very nice!
> Para
a month by the way.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>
find them unnecessary - and they
complicate the build.
And I would love to Lucene parallel like Solr now.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
>
and committing parallel tests for Lucene too.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
>
ensure it is really correct.
The patch does not do backwards in parallel (only core/contrib)
Also, there is the TEMP_DIR problem i mentioned, which i haven't addressed here.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https
matter too, as they are now
also run in parallel.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvem
n just use java.io.tmpdir for its lock file.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
>
issue in
the backwards tests.
I think we are close, there are only two issues I want to address first:
* Solr tests should do some uptodate check on lucene's test code
* Benchmark's work directory for tests should be under tempDir
> Parallelize Tests
> -
>
&g
,
as solr tests use this code... really outside the scope of this issue, but it
will cause confusion if
people have to 'ant clean' after the commit.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.or
make sure that solr tests detect when lucene
core test code is out of date, as solr tests use this code... really outside
the scope of this issue, but it will cause confusion if people have to 'ant
clean' after the commit.
The ANT specialist will look into this!
>
this!
Ok, thanks uwe.
I will commit this shortly, and send a note to both dev lists mentioning to run
ant clean for the time being for any old checkouts.
I think most people will prefer shaving minutes off their test time for a one
time-clean... unless anyone objects!
> Paralleliz
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir reassigned LUCENE-1709:
---
Assignee: Robert Muir
> Parallelize Tests
> -
>
>
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir resolved LUCENE-1709.
-
Resolution: Fixed
Committed revision 928069.
> Parallelize Te
he default threads-per-cpu to 1 by default, and allow
it to be configurable via sysprop.
i think these apply to solr too, so i'm proposing fixing both the build.xml's
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: ht
run sequentially (by adding the
property).
Robert, are you going to tackle that soon?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
st to generate a
patch out of them, so they don't get lost. The patch doesn't cover the ant
.jars, only the changes to common-build.xml as well as benchmark/build.xml
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https:/
exceed, wordpad open. The tests are
taking 98-99.9% of my cpu while hanging. I suspect there is some kind of
resource issue when running the tests in parallel.
Tom Burton-West
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: htt
;t gotten to fix this yet.
Shai, would you mind committing your patch? we can keep the issue open to add
the sysprop and fix the ant jar thing, and apply the same fixes to Solr's
build.xml
> Parallelize Tests
> -
>
> Key: LUCENE-1709
&g
.java:428)
[junit] at java.lang.Thread.run(Thread.java:619)
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Imp
d to Shai.
Can you try his patch and see if it fixed the problem for you?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
>
rces junit to use a
separate temp folder per thread, it still hung ...
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
>
1708 and ran "ant clean test-contribute" and the tests ran
just fine. The patch seems to have solved the problem.
Tom
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
sion 932857... will handle these one
at a time.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
>
wing:
# benchmark tests force sequential run
# threadsPerProcessor defaults to 1 and can be overridden by
-DthreadsPerProcessor=
# A CHANGES entry
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira
nt and src/test.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Aff
don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
Parallelize Tests
-
Key: LUCENE-1709
URL: https://issues.apache.org/jira/browse/LUCENE-1709
Project: Lucene - Java
Issue Typ
t to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
Parallelize Tests
't get it to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing w
Maven I couldn't get it to work (was
>> running into a bug) so my knowledge isn't very good. We're using
>> ant and ivy for our webapp dev. I don't know what Lucene's
>> position is on Maven, but am interested pursuing whatever makes
>> sense.
&
45 matches
Mail list logo
