[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855713#action_12855713
]
Shai Erera commented on LUCENE-1709:
Committed revision 932878 with the following:
# benchmark tests force sequential run
# threadsPerProcessor defaults to 1 and can be overridden by
-DthreadsPerProcessor=
# A CHANGES entry
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855700#action_12855700
]
Robert Muir commented on LUCENE-1709:
-
I committed the junit tempdir fix in revision 932857... will handle these one
at a time.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855022#action_12855022
]
Tom Burton-West commented on LUCENE-1709:
-
Hi Robert,
I patched Revision 931708 and ran "ant clean test-contribute" and the tests ran
just fine. The patch seems to have solved the problem.
Tom
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855020#action_12855020
]
Shai Erera commented on LUCENE-1709:
Robert, I will commit the patch, seems good to do anyway. We can handle the ant
jars separately later.
And ths hang behavior is exactly what I experience, including the
FileInputStream thing. Only on my machine, when I took a thread dump, it showed
that Ant waits on FIS.read() ...
Robert - to remind you that even with the patch which forces junit to use a
separate temp folder per thread, it still hung ...
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854967#action_12854967
]
Robert Muir commented on LUCENE-1709:
-
Thanks Tom, this is exactly what happened to Shai.
Can you try his patch and see if it fixed the problem for you?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854960#action_12854960
]
Tom Burton-West commented on LUCENE-1709:
-
This may or may not be a clue to the problem in benchmark. When I control-C'd
the hung test, I got the error reported below.
Tom.
[junit] directory = RAMDirectory
[junit] doc.maker =
org.apache.lucene.benchmark.byTask.tasks.WriteLineDocTaskTest$JustDateDocMaker
[junit] line.file.out =
C:\cygwin\home\tburtonw\lucene\april07_good\build\contrib\benchmark\test\W\one-line
[junit] ---
[junit] - ---
[junit] java.io.FileNotFoundException:
C:\cygwin\home\tburtonw\lucene\april07_good\contrib\benchmark\junitvmwatcher203463231158436475.properties
(The process cannot access the file because it is being used by another
process)
[junit] at java.io.FileInputStream.open(Native Method)
[junit] at java.io.FileInputStream.(FileInputStream.java:106)
[junit] at java.io.FileReader.(FileReader.java:55)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeAsForked(JUnitTask.java:1025)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:876)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:803)
[junit] at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[junit] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
[junit] at org.apache.tools.ant.Task.perform(Task.java:348)
[junit] at
org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62)
[junit] at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[junit] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
[junit] at org.apache.tools.ant.Task.perform(Task.java:348)
[junit] at
org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:394)
[junit] at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[junit] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
[junit] at org.apache.tools.ant.Task.perform(Task.java:348)
[junit] at
org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run(Parallel.java:428)
[junit] at java.lang.Thread.run(Thread.java:619)
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854959#action_12854959
]
Robert Muir commented on LUCENE-1709:
-
Thanks Tom and Shai... sorry I haven't gotten to fix this yet.
Shai, would you mind committing your patch? we can keep the issue open to add
the sysprop and fix the ant jar thing, and apply the same fixes to Solr's
build.xml
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854957#action_12854957
]
Tom Burton-West commented on LUCENE-1709:
-
I am having the same issue Shai reported in LUCENE-2353 with the parallel tests
apparently causing the tests to hang on my Windows box with both Revision
931573 and Revision 931304 when running the tests from root.
Tests hang in WriteLineDocTaskTest, on this line:
[junit] > config properties:
[junit] directory = RAMDirectory
[junit] doc.maker =
org.apache.lucene.benchmark.byTask.tasks.WriteLineDocTaskTest$JustDateDocMaker
[junit] line.file.out =
D:\dev\lucene\lucene-trunk\build\contrib\benchmark\test\W\one-line
[junit] ---
I just ran the test last night with Revision 931708 and had no problem. Ran
it again this morning and got the hanging behavior. The difference is that
last night the only thing running on my computer besides a couple of ssh
terminal windows was the tests. Today when I ran the tests and got the
hanging behavior, I have firefox, outlook, exceed, wordpad open. The tests are
taking 98-99.9% of my cpu while hanging. I suspect there is some kind of
resource issue when running the tests in parallel.
Tom Burton-West
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709-2.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854348#action_12854348
]
Shai Erera commented on LUCENE-1709:
One more thing - change benchmark tests to run sequentially (by adding the
property).
Robert, are you going to tackle that soon?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
>Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850363#action_12850363
]
Robert Muir commented on LUCENE-1709:
-
bq. The ANT specialist will look into this!
Ok, thanks uwe.
I will commit this shortly, and send a note to both dev lists mentioning to run
ant clean for the time being for any old checkouts.
I think most people will prefer shaving minutes off their test time for a one
time-clean... unless anyone objects!
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850351#action_12850351
]
Uwe Schindler commented on LUCENE-1709:
---
bq. The only thing left now, is to make sure that solr tests detect when lucene
core test code is out of date, as solr tests use this code... really outside
the scope of this issue, but it will cause confusion if people have to 'ant
clean' after the commit.
The ANT specialist will look into this!
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch,
> LUCENE-1709.patch, LUCENE-1709.patch, LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848718#action_12848718
]
Michael McCandless commented on LUCENE-1709:
+1 for removing the flags and committing parallel tests for Lucene too.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848712#action_12848712
]
Mark Miller commented on LUCENE-1709:
-
+1 on removing those flags - personally I find them unnecessary - and they
complicate the build.
And I would love to Lucene parallel like Solr now.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848707#action_12848707
]
Robert Muir commented on LUCENE-1709:
-
Thanks Jason.
So for newtrunk I applied a similar patch to speed up Solr's tests.
You can see it here: http://svn.apache.org/viewvc?rev=926470&view=rev
In this case the output is not interleaved because it uses a special formatter.
So it basically looks just like you are not using parallel at all.
Additionally -Dtestpackage, -Dtestpackageroot, -Dtestcase all work, the former
two are also parallelized.
So, I propose we do the same thing for Lucene tests.
Solr was simple because it does not have these junit failed flag files.
I propose we just remove these, like how Solr does contrib.
Hudson hasn't failed in over a month by the way.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843385#action_12843385
]
Jason Rutherglen commented on LUCENE-1709:
--
Robert, very nice!
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: LUCENE-1709.patch, runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724815#action_12724815
]
Michael McCandless commented on LUCENE-1709:
Actually I see decent gains from concurrency: when I run tests with 6
threads my tests run a little over 3X faster (12:59 with 1 thread and
4:15 with 6 threads).
I'm using a Python script that launches the threads, each specifying
-Dtestpackage to run a certain subset of Lucene's tests.
This is on an OpenSolaris (2009.06) machine, with a Core i7 920 CPU
(= 8 cores presented to the OS) and an Intel X25M SSD, 12 GB RAM. The
hardware has quite a bit of concurrency.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724720#action_12724720
]
Johan Kindgren commented on LUCENE-1709:
I did a quick maven setup just to see the effects of running tests in parallel,
and the result was a bit surprising. When running in parallel, it took about
30-40 seconds less (~7.40minutes) than running in sequence (~8.10minutes).
Don't know if there would be a significant boost on a quad-core, my dual core
had some idle time left when running with five threads.
Is it worth the development time and the complexity increase of the build?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
Re: [jira] Commented: (LUCENE-1709) Parallelize Tests
P.S Looking forward to this issue. Just ordered the cheapest quad core
laptop I could find :) So long dual core.
- Mark
Mark Miller wrote:
I found a couple external solutions last I looked. They appeared to
have various limitations though. I really think, longer term, the
built in support is the way to go. I'd hate to require a certain
version of ANT (its only the very latest 4.6 release), but perhaps we
could set it up to work easily in the case that you do have the right
version, or we could package it up as a contrib somehow?
Thats where the plug-ability of an external is nice - but for long
term maintainability and correctness - I'd lean towards built in
support myself.
### Test scheduling strategies ###
`JUnitCore` now includes an experimental method that allows you to
specify a model of the `Computer` that runs your tests. Currently,
the only built-in Computers are the default, serial runner, and two
runners provided in the `ParallelRunner` class:
`ParallelRunner.classes()`, which runs classes in parallel, and
`ParallelRunner.methods()`, which runs classes and methods in parallel.
This feature is currently less stable than MaxCore, and may be
merged with MaxCore in some way in the future.
Example:
public static class Example {
@Test public void one() throws InterruptedException {
Thread.sleep(1000);
}
@Test public void two() throws InterruptedException {
Thread.sleep(1000);
}
}
@Test public void testsRunInParallel() {
long start= System.currentTimeMillis();
Result result= JUnitCore.runClasses(ParallelComputer.methods(),
Example.class);
assertTrue(result.wasSuccessful());
long end= System.currentTimeMillis();
assertThat(end - start, betweenInclusive(1000, 1500));
}
- Mark
Jason Rutherglen wrote:
There's this foreach parallel ant-contrib task. Perhaps we have a
winner!
http://www.jadeliquid.com/liquidtest/docs/doku.php?id=developers:cookbooks:scaling_liquidtest_with_junit_parallel:home
http://ant-contrib.sourceforge.net/tasks/tasks/foreach.html
On Thu, Jun 25, 2009 at 7:14 PM, Mark Miller > wrote:
JUnit also supports parallelizing tasks, but its only in the very
latest release. I'd check out that. There are generally more
issues than just firing off multiple tests at once.
-- - Mark
http://www.lucidimagination.com
Jason Rutherglen (JIRA) wrote:
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724325#action_12724325
]
Jason Rutherglen commented on LUCENE-1709:
--
{quote} Do you mean that you would implement your own task? In
which module would you put this custom ant task? {quote}
Yes, seems creating a custom task should work? In contrib/ant?
Last time I tried to use Maven I couldn't get it to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
Parallelize Tests
-
Key: LUCENE-1709
URL:
https://issues.apache.org/jira/browse/LUCENE-1709
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Fix For: 3.0
Original Estimate: 48h
Remaining Estimate: 48h
The Lucene tests can be parallelized to make for a faster
testing system. This task from ANT can be used:
http://ant.apache.org/manual/CoreTasks/parallel.html
Previous discussion:
http://www.gossamer-threads.com/lists/lucene/java-dev/69669
Notes from Mike M.:
{quote}
I'd love to see a clean solution here (the tests are
embarrassingly
parallelizable, and we all have machines with good
concurrency these
days)... I have a rather hacked up solution now, that uses
"-Dtestpackage=XXX" to split the tests up.
Ideally I would be able to say "use N threads" and it'd do
the right
thing... like the -j flag to make.
{quote}
-
To unsubscribe, e-mail: ja
Re: [jira] Commented: (LUCENE-1709) Parallelize Tests
I found a couple external solutions last I looked. They appeared to have
various limitations though. I really think, longer term, the built in
support is the way to go. I'd hate to require a certain version of ANT
(its only the very latest 4.6 release), but perhaps we could set it up
to work easily in the case that you do have the right version, or we
could package it up as a contrib somehow?
Thats where the plug-ability of an external is nice - but for long term
maintainability and correctness - I'd lean towards built in support myself.
### Test scheduling strategies ###
`JUnitCore` now includes an experimental method that allows you to
specify a model of the `Computer` that runs your tests. Currently,
the only built-in Computers are the default, serial runner, and two
runners provided in the `ParallelRunner` class:
`ParallelRunner.classes()`, which runs classes in parallel, and
`ParallelRunner.methods()`, which runs classes and methods in parallel.
This feature is currently less stable than MaxCore, and may be
merged with MaxCore in some way in the future.
Example:
public static class Example {
@Test public void one() throws InterruptedException {
Thread.sleep(1000);
}
@Test public void two() throws InterruptedException {
Thread.sleep(1000);
}
}
@Test public void testsRunInParallel() {
long start= System.currentTimeMillis();
Result result= JUnitCore.runClasses(ParallelComputer.methods(),
Example.class);
assertTrue(result.wasSuccessful());
long end= System.currentTimeMillis();
assertThat(end - start, betweenInclusive(1000, 1500));
}
- Mark
Jason Rutherglen wrote:
There's this foreach parallel ant-contrib task. Perhaps we have a winner!
http://www.jadeliquid.com/liquidtest/docs/doku.php?id=developers:cookbooks:scaling_liquidtest_with_junit_parallel:home
http://ant-contrib.sourceforge.net/tasks/tasks/foreach.html
On Thu, Jun 25, 2009 at 7:14 PM, Mark Miller > wrote:
JUnit also supports parallelizing tasks, but its only in the very
latest release. I'd check out that. There are generally more
issues than just firing off multiple tests at once.
--
- Mark
http://www.lucidimagination.com
Jason Rutherglen (JIRA) wrote:
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724325#action_12724325
]
Jason Rutherglen commented on LUCENE-1709:
--
{quote} Do you mean that you would implement your own task? In
which module would you put this custom ant task? {quote}
Yes, seems creating a custom task should work? In contrib/ant?
Last time I tried to use Maven I couldn't get it to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
Parallelize Tests
-
Key: LUCENE-1709
URL:
https://issues.apache.org/jira/browse/LUCENE-1709
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Fix For: 3.0
Original Estimate: 48h
Remaining Estimate: 48h
The Lucene tests can be parallelized to make for a faster
testing system. This task from ANT can be used:
http://ant.apache.org/manual/CoreTasks/parallel.html
Previous discussion:
http://www.gossamer-threads.com/lists/lucene/java-dev/69669
Notes from Mike M.:
{quote}
I'd love to see a clean solution here (the tests are
embarrassingly
parallelizable, and we all have machines with good
concurrency these
days)... I have a rather hacked up solution now, that uses
"-Dtestpackage=XXX" to split the tests up.
Ideally I would be able to say "use N threads" and it'd do
the right
thing... like the -j flag to make.
{quote}
-
To unsubscribe, e-mail: java-dev-unsubscr
Re: [jira] Commented: (LUCENE-1709) Parallelize Tests
There's this foreach parallel ant-contrib task. Perhaps we have a winner!
http://www.jadeliquid.com/liquidtest/docs/doku.php?id=developers:cookbooks:scaling_liquidtest_with_junit_parallel:home
http://ant-contrib.sourceforge.net/tasks/tasks/foreach.html
On Thu, Jun 25, 2009 at 7:14 PM, Mark Miller wrote:
> JUnit also supports parallelizing tasks, but its only in the very latest
> release. I'd check out that. There are generally more issues than just
> firing off multiple tests at once.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> Jason Rutherglen (JIRA) wrote:
>
>>[
>> https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724325#action_12724325]
>> Jason Rutherglen commented on LUCENE-1709:
>> --
>>
>> {quote} Do you mean that you would implement your own task? In
>> which module would you put this custom ant task? {quote}
>>
>> Yes, seems creating a custom task should work? In contrib/ant?
>> Last time I tried to use Maven I couldn't get it to work (was
>> running into a bug) so my knowledge isn't very good. We're using
>> ant and ivy for our webapp dev. I don't know what Lucene's
>> position is on Maven, but am interested pursuing whatever makes
>> sense.
>>
>>
>>
>>> Parallelize Tests
>>> -
>>>
>>>Key: LUCENE-1709
>>>URL: https://issues.apache.org/jira/browse/LUCENE-1709
>>>Project: Lucene - Java
>>> Issue Type: Improvement
>>> Components: Index
>>> Affects Versions: 2.4.1
>>> Reporter: Jason Rutherglen
>>>Fix For: 3.0
>>>
>>> Original Estimate: 48h
>>> Remaining Estimate: 48h
>>>
>>> The Lucene tests can be parallelized to make for a faster testing system.
>>> This task from ANT can be used:
>>> http://ant.apache.org/manual/CoreTasks/parallel.html
>>> Previous discussion:
>>> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
>>> Notes from Mike M.:
>>> {quote}
>>> I'd love to see a clean solution here (the tests are embarrassingly
>>> parallelizable, and we all have machines with good concurrency these
>>> days)... I have a rather hacked up solution now, that uses
>>> "-Dtestpackage=XXX" to split the tests up.
>>> Ideally I would be able to say "use N threads" and it'd do the right
>>> thing... like the -j flag to make.
>>> {quote}
>>>
>>>
>>
>>
>>
>
>
>
> -
> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-dev-h...@lucene.apache.org
>
>
Re: [jira] Commented: (LUCENE-1709) Parallelize Tests
JUnit also supports parallelizing tasks, but its only in the very latest
release. I'd check out that. There are generally more issues than just
firing off multiple tests at once.
--
- Mark
http://www.lucidimagination.com
Jason Rutherglen (JIRA) wrote:
[ https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724325#action_12724325 ]
Jason Rutherglen commented on LUCENE-1709:
--
{quote} Do you mean that you would implement your own task? In
which module would you put this custom ant task? {quote}
Yes, seems creating a custom task should work? In contrib/ant?
Last time I tried to use Maven I couldn't get it to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
Parallelize Tests
-
Key: LUCENE-1709
URL: https://issues.apache.org/jira/browse/LUCENE-1709
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Fix For: 3.0
Original Estimate: 48h
Remaining Estimate: 48h
The Lucene tests can be parallelized to make for a faster testing system.
This task from ANT can be used: http://ant.apache.org/manual/CoreTasks/parallel.html
Previous discussion: http://www.gossamer-threads.com/lists/lucene/java-dev/69669
Notes from Mike M.:
{quote}
I'd love to see a clean solution here (the tests are embarrassingly
parallelizable, and we all have machines with good concurrency these
days)... I have a rather hacked up solution now, that uses
"-Dtestpackage=XXX" to split the tests up.
Ideally I would be able to say "use N threads" and it'd do the right
thing... like the -j flag to make.
{quote}
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724325#action_12724325
]
Jason Rutherglen commented on LUCENE-1709:
--
{quote} Do you mean that you would implement your own task? In
which module would you put this custom ant task? {quote}
Yes, seems creating a custom task should work? In contrib/ant?
Last time I tried to use Maven I couldn't get it to work (was
running into a bug) so my knowledge isn't very good. We're using
ant and ivy for our webapp dev. I don't know what Lucene's
position is on Maven, but am interested pursuing whatever makes
sense.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724317#action_12724317
]
Johan Kindgren commented on LUCENE-1709:
Just a thought from an outsider...
Do you mean that you would implement your own task? In which module would you
put this custom ant task? This would mean that you need a module in Lucene just
to be able to build Lucene?
Can't help keep on pointing to maven which has already solved this problem...
http://maven.apache.org/plugins/maven-surefire-plugin/examples/testng.html
Look at running tests in parallel.
I havent tried this yet, but why spend time reinventing the wheel?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724277#action_12724277
]
Jason Rutherglen commented on LUCENE-1709:
--
{quote} As mentioned in the "Previous discussion" link: a simple
way for lucene to get some paralllelization using existing ant
functionality would be to extend (or make a variant) of our
contrib-crawl so it could run the contrib tests in parallel
(contrib crawl can't be parallelized in the general case because
some contribs have dependencies on other contribs and build them
if they aren't already built) {quote}
I'd like to get test-core multithreaded. In looking at the
ant-junit code, it's unfortunately not easy (as noted in the
links) to execute the batchtest(s) in parallel. So agreed that
the most obvious solution belongs in the the ant world. However
I still hold out some hope for a custom interim solution.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723867#action_12723867
]
Jason Rutherglen commented on LUCENE-1709:
--
Thanks for the links!
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723850#action_12723850
]
Hoss Man commented on LUCENE-1709:
--
if you're looking to extend ant it might be better to tackle that on the
d...@ant list ... there's only two or three ant hackers i can think of in the
lucene community (and a lot in the ant community)
I would also suggest becoming familiar with these two links...
http://www.nabble.com/1.7.1---Beta-Vote-to16148645.html#a15698118
https://parallel-junit.dev.java.net/
As mentioned in the "Previous discussion" link: a simple way for lucene to get
*some* paralllelization using existing ant functionality would be to extend (or
make a variant) of our contrib-crawl so it could run the contrib tests in
parallel (contrib crawl can't be parallelized in the general case because some
contribs have dependencies on other contribs and build them if they aren't
already built)
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723815#action_12723815
]
Jason Rutherglen commented on LUCENE-1709:
--
So am thinking we'd create a ParallelJUnitTask that extends JUnitTask and
accepts a threadCount property. We can reuse code from the threading code from
Parallel.
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1709) Parallelize Tests
[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723810#action_12723810
]
Jason Rutherglen commented on LUCENE-1709:
--
Given my limited understanding of ANT, it seems like it would be
ideal long term to be able to set a the numThreads on JUnitTask
which underneath executes the tests in BatchTest in parallel.
I'm not sure how to hack Parallel and JUnitTask together in the
ANT XML.
If it's possible for JUnit to somehow return Tasks up the chain
to Parallel, that might work?
> Parallelize Tests
> -
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
>Affects Versions: 2.4.1
>Reporter: Jason Rutherglen
> Fix For: 3.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
