Re: IndexUpdateListener

2006-05-14 Thread Erik Hatcher
Lucene itself doesn't provide this, but Solr does. And with Solr you don't need your own custom Hits cache - it already provides caches in several areas. Erik On May 13, 2006, at 9:12 AM, karl wettin wrote: This might exist? How about a list of index listeners that is called

Re: Recommendations please

2006-05-14 Thread Erik Hatcher
On May 13, 2006, at 12:10 PM, Malcolm Clark wrote: Hi everyone, I am about to index the INEX collection (22 files with 3 files in each-ish) using Java Lucene. I am undecided with the approach to indexing and have left my LIA book at uni :-/ Would you recommend: 1.. indexing all

Re: IndexUpdateListener

2006-05-14 Thread karl wettin
On Sun, 2006-05-14 at 07:05 -0400, Erik Hatcher wrote: Lucene itself doesn't provide this, but Solr does. I'm not using Solr :) But what you suggest is that I should try to fit in in my own layer around Lucene rather than in Lucene? May I ask why? Not sure if I would do it that way. To me it

Re: IndexUpdateListener

2006-05-14 Thread Yonik Seeley
On 5/14/06, karl wettin [EMAIL PROTECTED] wrote: To me it feels as the index is the only thing that knows for sure if it has been updated. I guess that would be whenever an IndexReader that had new deletions is closed, or an IndexWriter that changed the segments file? An app using Lucene

out-of-memory when searching, paging does not work.

2006-05-14 Thread Beady Geraghty
I have an out-of-memroy error when returning many hits. I am still on Lucene 1.4.3 I have a simple term query. It returned 899810 documents. I try to retrieve the name of each document and nothing else and I ran out of memory. Instead of getting the names all at once, I tried to query again

Re: out-of-memory when searching, paging does not work.

2006-05-14 Thread Erik Hatcher
Could you share at least some pseudo-code of what you're doing in the loop of retrieving the name of each document? Are you storing all of those names as you iterate? Have you profiled your application to see exactly where the memory is going? It is surely being eaten by your own code

Re: IndexUpdateListener

2006-05-14 Thread Marc Dauncey
I'm planning to write a listener which will recieve messages from our indexing application via SOAP - the message will indicate which indexes have been updated as we will be using quite a few separate lucene indexes. No point dropping all the searchers (and any caches, filters etc) if only one

Re: Backing up indexes, reliability and robustness

2006-05-14 Thread Marc Dauncey
Thank you for all your thoughts on this - I think we will go the route of having two copies for each index, a working copy and the one the searchers use. If we start off indexing like this it shouldn't be too hard to keep them in sync. Thanks again, Marc - Original Message From:

Re: IndexUpdateListener

2006-05-14 Thread karl wettin
On Sun, 2006-05-14 at 10:50 -0400, Yonik Seeley wrote: On 5/14/06, karl wettin [EMAIL PROTECTED] wrote: To me it feels as the index is the only thing that knows for sure if it has been updated. I guess that would be whenever an IndexReader that had new deletions is closed, or an

Re: IndexUpdateListener

2006-05-14 Thread karl wettin
On Sun, 2006-05-14 at 22:32 -0400, Erik Hatcher wrote: On May 14, 2006, at 1:48 PM, karl wettin wrote: An app using Lucene still needs to coordinate all the activity surrounding IndexReaders and IndexWriters, including explicit closure, so the app will know anyway when the index has

Re: IndexUpdateListener

2006-05-14 Thread karl wettin
On Sun, 2006-05-14 at 22:27 -0400, Erik Hatcher wrote: What are the boundaries of what you call an index? Is it the current Lucene API, or could it be a service-like layer such as Solr on top of it? The persistence mechanism. So that would be a part of the the current Lucene API.

Re: IndexUpdateListener

2006-05-14 Thread Yonik Seeley
On 5/14/06, karl wettin [EMAIL PROTECTED] wrote: On Sun, 2006-05-14 at 22:27 -0400, Erik Hatcher wrote: What are the boundaries of what you call an index? Is it the current Lucene API, or could it be a service-like layer such as Solr on top of it? The persistence mechanism. So that would be

Re: IndexUpdateListener

2006-05-14 Thread Chris Hostetter
: As deep as possible. : : All the user would need to know is Directory.getListerners() I don't really get what the point of such a low level callback mechanism would be ... do you have uses cases where you're really going to want to know every time someone uses a Directory object to do

Re: IndexUpdateListener

2006-05-14 Thread karl wettin
On Sun, 2006-05-14 at 21:09 -0700, Chris Hostetter wrote: Personally: I think the best way to be notified that the index is changed at a really fine level of granularity is to just poll on IndexReader.getCurrentVersion (and compare with IndexReader.getVersion) ... that way you don't have to