Almost parallel indexes

2007-09-27 Thread Tim Sturge
Hi, I have an index which contains two very distinct types of fields: - Some fields are large (many term documents) and change fairly slowly. - Some fields are small (mostly titles, names, anchor text) and change fairly rapidly. Right now I keep around the large fields in raw form and when the

Re: Almost parallel indexes

2007-09-27 Thread Erick Erickson
OK, this isn't well thought out, more the first thing that pops to mind... You're right, Lucene doesn't do joins. But would it serve to keep two indexes? One the slow-changing stuff and one the fast-changing stuff. They are related by some *external* (as in "not the Lucene doc id) field. You'd h

Re: Almost parallel indexes

2007-09-28 Thread Nixon
- Original Message - From: "Erick Erickson" <[EMAIL PROTECTED]> To: Sent: Friday, September 28, 2007 5:43 AM Subject: Re: Almost parallel indexes OK, this isn't well thought out, more the first thing that pops to mind... You're right, Lucene doesn't

Re: Almost parallel indexes

2007-09-28 Thread Chris Hostetter
: I can't really use ParallelReader to keep the indexes the same; it : requires me to add documents to both indexes which means I have to : retokenize the large fields anyway. I would want to do a "join" on an : external id, and as far as I can tell, Lucene doesn't support that. correction: it