Indexing - scheduled batch process or server?

2006-04-17 Thread Marc Dauncey
Hi everyone, I'm currently designing a Lucene search system and i'm considering the indexing side of things. Just wondered what kind of architecture people have adopted for indexing - are CHRON jobs sufficient for high volume drip feed indexing or has anyone implemented a more sophisticated solut

Re: Indexing - scheduled batch process or server?

2006-04-17 Thread Jeremy Hanna
I'm pretty new with this, but with my index for a database, I'm using a Quartz scheduler. Also at the end of the index update, I set my singleton of IndexSearcher to null. That way the index searcher will be using the latest information. That bit as well as setting it to null and not clo

Re: Indexing - scheduled batch process or server?

2006-04-18 Thread Marc Dauncey
Thanks for the response, Jeremy. Quartz seems like a great solution - are you running it within the app server? I think the benefits of doing this would be convenience of messaging the search server to pick up fresh indexes. Previously I considered a CRON job and was thinking of making a web serv

Re: Indexing - scheduled batch process or server?

2006-04-18 Thread Yonik Seeley
On 4/17/06, Marc Dauncey <[EMAIL PROTECTED]> wrote: > or has anyone > implemented a more sophisticated solution with web > services to index on demand? In Solr, documents (XML versions of Lucene Documents) are POSTed to the server. There are explicit commands that cause an new IndexReader to be o

Re: Indexing - scheduled batch process or server?

2006-04-18 Thread Jeremy Hanna
Marc, I am using it within the web app. I use Spring and there are ways to throttle a call down to one thread with Spring, if you're worried about overloading the server when you update the index. I'm not sure about Quartz and its ability to set a priority or limit the number of threads