Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-16 Thread Shaun Senecal
Thanks Mike. The queries are now running faster than they ever were before, and are returning the expected results! On Fri, Oct 16, 2009 at 7:39 AM, Shaun Senecal ssenecal.w...@gmail.comwrote: Ah! I thought that the ConstantScoreQuery would also be rewritten into a BooleanQuery, resulting

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-16 Thread Michael McCandless
Super! Mike On Fri, Oct 16, 2009 at 4:06 AM, Shaun Senecal ssenecal.w...@gmail.com wrote: Thanks Mike.  The queries are now running faster than they ever were before, and are returning the expected results! On Fri, Oct 16, 2009 at 7:39 AM, Shaun Senecal ssenecal.w...@gmail.comwrote: Ah!  

PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Shaun Senecal
I know this has been discussed to great length, but I still have not found a satisfactory solution and I am hoping someone on the list has some ideas... We have a large index (4M+ Documents) with a handful of Fields. We need to perform PrefixQueries on multiple fields. The problem is that when

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Shaun Senecal
Sorry for the double post, but I think I can clarify the problem a little more. We want to execute: query: A | B | C | D filter: null However, C and D cause TooManyClauses, so instead we execute: query: A | B filter: C | D My understanding is that Lucene will apply the Filter (C

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Michael McCandless
On Thu, Oct 15, 2009 at 4:57 AM, Shaun Senecal ssenecal.w...@gmail.com wrote: Up to Lucene 2.4, this has been working out for us. However, in Lucene 2.9 this breaks since rewrite() now returns a ConstantScoreQuery. You can get back to the 2.4 behavior by calling

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Shaun Senecal
Thanks for the explanation Mike. It looks like I have no choice but to move any queries which throw TooManyClauses to be Filters. Sadly, this means a max query time of 6s under load unless I can find a way to rewrite the query to span a Query and a Filter. Thanks again On Thu, Oct 15, 2009

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Michael McCandless
You should be able to do exactly what you were doing on 2.4, right? (By setting the rewrite method). Mike On Thu, Oct 15, 2009 at 8:30 AM, Shaun Senecal ssenecal.w...@gmail.com wrote: Thanks for the explanation Mike.  It looks like I have no choice but to move any queries which throw

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Shaun Senecal
At first I thought so, yes, but then I realised that the query I wanted to execute was A | B | C | D and in reality I was executing (A | B) (C | D). I guess my unit tests were missing some cases and don't currently catch this. On Thu, Oct 15, 2009 at 11:59 PM, Michael McCandless

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Michael McCandless
Well, you could wrap the C | D filter as a Query (using ConstantScoreQuery), and then add that as a SHOULD clause on your toplevel BooleanQuery? Mike On Thu, Oct 15, 2009 at 5:42 PM, Shaun Senecal ssenecal.w...@gmail.com wrote: At first I thought so, yes, but then I realised that the query I

Re: PrefixQueries on large indexes (4M+ Documents) using a partial Query partial Filter solution

2009-10-15 Thread Shaun Senecal
Ah! I thought that the ConstantScoreQuery would also be rewritten into a BooleanQuery, resulting in the same exception. If that's the case, then this should work. I'll give that a try when I get into the office this morning. On Fri, Oct 16, 2009 at 6:46 AM, Michael McCandless