ParallelMultiSearcher reimplementation

2006-11-02 Thread Gus Holcomb
Hello everyone, We are currently using Lucene 1.9.1 at work. Using a profiler, I discovered that searching with a HitCollector in a ParallelMultiSearcher is single threaded. By extending ParallelMultiSearcher I was able to parallelize it without a problem (and without requiring a new lucene jar f

Re: ParallelMultiSearcher reimplementation

2006-11-02 Thread Chris Hostetter
Hi Gus, : Is this development already taking place in the trunk? I was unable to : uncover any progress in this area. I haven't contributed to lucene (or : any open source project) before, but I would be willing to clean up a : number of things in this area if there was interest. I'm not aware o

RE: ParallelMultiSearcher reimplementation

2006-11-03 Thread Gus Holcomb
Is there any timeline for when Java 1.5 packages will be allowed? Thanks, Gus Holcomb -Original Message- From: Chris Hostetter [mailto:[EMAIL PROTECTED] Sent: Thursday, November 02, 2006 8:08 PM To: java-dev@lucene.apache.org Subject: Re: ParallelMultiSearcher reimplementation Hi Gus

Re: ParallelMultiSearcher reimplementation

2006-11-03 Thread Doug Cutting
Gus Holcomb wrote: We are currently using Lucene 1.9.1 at work. Using a profiler, I discovered that searching with a HitCollector in a ParallelMultiSearcher is single threaded. By extending ParallelMultiSearcher I was able to parallelize it without a problem (and without requiring a new lucene

RE: ParallelMultiSearcher reimplementation

2006-11-03 Thread Chris Hostetter
e rioting by saying "my guess is 1.5 packages will be supported when the patches requiring them become highly desired. : -Original Message- : From: Chris Hostetter [mailto:[EMAIL PROTECTED] : Sent: Thursday, November 02, 2006 8:08 PM : To: java-dev@lucene.apache.org : Subject: Re: P

Re: ParallelMultiSearcher reimplementation

2006-11-03 Thread Chuck Williams
Chris Hostetter wrote on 11/03/2006 09:40 AM: > : Is there any timeline for when Java 1.5 packages will be allowed? > > I don't think i'll incite too much rioting to say "no there is no > timeline" > .. I may incite some rioting by saying "my guess is 1.5 packages will be > supported when the patch

Re: ParallelMultiSearcher reimplementation

2006-11-03 Thread Doug Cutting
Chuck Williams wrote: Why would a thread pool be more controversial? Dynamically creating and garbaging threads has many downsides. The JVM already pools native threads, so mostly what's saved by thread pools is the allocation & initialization of new Thread instances. There are also downsid

Re: ParallelMultiSearcher reimplementation

2006-11-03 Thread Otis Gospodnetic
.apache.org Sent: Friday, November 3, 2006 11:54:56 AM Subject: RE: ParallelMultiSearcher reimplementation Is there any timeline for when Java 1.5 packages will be allowed? Thanks, Gus Holcomb -Original Message- From: Chris Hostetter [mailto:[EMAIL PROTECTED] Sent: Thursday, November 02, 2

Re: ParallelMultiSearcher reimplementation

2006-11-05 Thread Chuck Williams
Doug Cutting wrote on 11/03/2006 12:18 PM: > Chuck Williams wrote: >> Why would a thread pool be more controversial? Dynamically creating and >> garbaging threads has many downsides. > > The JVM already pools native threads, so mostly what's saved by thread > pools is the allocation & initializati

Re: ParallelMultiSearcher reimplementation

2006-11-13 Thread Doug Cutting
Chuck Williams wrote: I followed this same logic in ParallelWriter and got burned. My first implementation (still the version submitted as a patch in jira) used dynamic threads to add the subdocuments to the parallel subindexes simultaneously. This hit a problem with abnormal native heap OOM's

Re: ParallelMultiSearcher reimplementation

2006-11-13 Thread eks dev
TED]> To: java-dev@lucene.apache.org Sent: Monday, 13 November, 2006 9:50:28 PM Subject: Re: ParallelMultiSearcher reimplementation Chuck Williams wrote: > I followed this same logic in ParallelWriter and got burned. My first > implementation (still the version submitted as a patch in jira) us

Re: ParallelMultiSearcher reimplementation

2006-11-13 Thread Chuck Williams
Doug Cutting wrote on 11/13/2006 10:50 AM: > Chuck Williams wrote: >> I followed this same logic in ParallelWriter and got burned. My first >> implementation (still the version submitted as a patch in jira) used >> dynamic threads to add the subdocuments to the parallel subindexes >> simultaneou