Adding segment warming to IW is the only way to insure newly merged segments are quickly searchable without the impact brought up by John W regarding queries on new segments being slow when they load field caches.
On Tue, Sep 22, 2009 at 8:37 AM, Michael McCandless <luc...@mikemccandless.com> wrote: > On Tue, Sep 22, 2009 at 11:08 AM, Yonik Seeley > <yo...@lucidimagination.com> wrote: > >> I'm still not sure I see the reason for complicating the IndexWriter >> with warming... can't this be done just as efficiently (if not more >> efficiently) in user/application space? > > It will be less efficient when you warm outside of IndexWriter, ie, > you will necessarily delay the app's net turnaround time on being able > to search newly added/deleted docs. > > The whole point of putting optional warming into IndexWriter was so > the segment could be warmed *before* the merge commits the change to > the writer's SegmentInfos. Any newly opened near-real-timer readers > continue to search the old (merged away) segments, until the warming > completes. > > This way the warming of merged segments is independent of making any > newly flushed segments searchable (as long as you use CMS, or any > merge scheduler that uses separate threads for merging). New segments > can be flushed and then become searchable (with getReader()) even > while the warming is happening. > > So... if your merge policy allows large merges, setting a warmer in > the IndexWriter is crucial for minimizing turnaround time. But, even > once you do that, merging is still IO & CPU intensive, plus IO caches > are unnecessarily flushed (since we can't easily madvise/posix_fadvise > from java), and we have no IO scheduler control to have merging run at > very lower priority, etc., so while the merge & warming are taking > place, search performance will be impacted. > > Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
