Re: on-the-fly filters from docID lists

2010-07-23 Thread Mark Harwood
of thousands of docIDs? Tom Burton-West http://www.hathitrust.org/blogs/large-scale-search -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Thursday, July 22, 2010 5:20 AM To: java-user@lucene.apache.org Subject: Re: on-the-fly filters from

RE: on-the-fly filters from docID lists

2010-07-23 Thread Burton-West, Tom
...@yahoo.co.uk] Sent: Friday, July 23, 2010 2:56 AM To: java-user@lucene.apache.org Subject: Re: on-the-fly filters from docID lists Re scalability of filter construction - the database is likely to hold stable primary keys not lucene doc ids which are unstable in the face of updates. You therefore

Re: on-the-fly filters from docID lists

2010-07-23 Thread Mark Harwood
To: java-user@lucene.apache.org Subject: Re: on-the-fly filters from docID lists Re scalability of filter construction - the database is likely to hold stable primary keys not lucene doc ids which are unstable in the face of updates. You therefore need a quick way of converting stable database

Re: on-the-fly filters from docID lists

2010-07-22 Thread Michael McCandless
It sounds like you should implement a custom Filter? Its getDocIdSet would consult your foreign key-value store and iterate through the allowed docIDs, per segment. Mike On Wed, Jul 21, 2010 at 8:37 AM, Martin J martinj.eng...@gmail.com wrote: Hello, we are trying to implement a query type for

RE: on-the-fly filters from docID lists

2010-07-22 Thread Burton-West, Tom
[mailto:luc...@mikemccandless.com] Sent: Thursday, July 22, 2010 5:20 AM To: java-user@lucene.apache.org Subject: Re: on-the-fly filters from docID lists It sounds like you should implement a custom Filter? Its getDocIdSet would consult your foreign key-value store and iterate through the allowed

Re: on-the-fly filters from docID lists

2010-07-22 Thread Michael McCandless
...@mikemccandless.com] Sent: Thursday, July 22, 2010 5:20 AM To: java-user@lucene.apache.org Subject: Re: on-the-fly filters from docID lists It sounds like you should implement a custom Filter? Its getDocIdSet would consult your foreign key-value store and iterate through the allowed docIDs

on-the-fly filters from docID lists

2010-07-21 Thread Martin J
Hello, we are trying to implement a query type for Lucene (with eventual target being Solr) where the query string passed in needs to be filtered through a large list of document IDs per user. We can't store the user ID information in the lucene index per document so we were planning to pull the