What about issuing separate commits to the index on a regularly scheduled basis? For example, you add documents to the index every 2 seconds, or however often, but these operations don't commit. Instead, you have a cron'd script or something that just issues a commit every 5 or 10 minutes or whatever interval you'd like.
I had to do something similar when I was running a re-index of my entire dataset. My program wasn't issuing commits, so I just cron'd a commit for every half hour so it didn't overload the server. Thanks, Charlie -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Thursday, May 10, 2007 9:07 AM To: solr-user@lucene.apache.org Subject: Re: fast update handlers On 5/10/07, Will Johnson <[EMAIL PROTECTED]> wrote: > I guess I was more concerned with doing the frequent commits and how > that would affect the caches. Say I have 2M docs in my main index but I > want to add docs every 2 seconds all while doing queries. if I do > commits every 2 seconds I basically loose any caching advantage and my > faceting performance goes down the tube. If however, I were to add > things to a smaller index and then roll it into the larger one every ~30 > minutes then I only take the hit on computing the larger filters caches > on that interval. Further, if my smaller index were based on a > RAMDirectory instead of a FSDirectory I assume computing the filter sets > for the smaller index should be fast enough even every 2 seconds. There isn't currently any support for incrementally updating filters. -Yonik
