> meaning in Bobo you'd like to manage your own memory resident field caches, and merge them whenever IW has merged a segment? Seems like you don't need genealogy for that.
Agreed, there is no need for full genealogy. > CSF isn't really designed yet. How come it can't be used with Bobo's field caches? I guess CSF should be able to support it, makes sense. As long as the container is flexible with the encoding (I need to look into this more on the Bobo side). > Lucene's internal field cache usage is now entirely at the segment level (ie, Lucene core should never request full field cache array at the MultiSegmentReader level). I think Bobo must have to do the same, if it handles near realtime updates, to get adequate performance. Bobo needs to migrate to this model, I don't think we've done that yet. > EG how come Bobo made its own field cache impl? Just because uninversion is too slow? It could be integrated once LUCENE-831 is completed. I think the current model of a weak reference and the inability to unload if needed is a concern. I don't think it's because of uninversion. On Fri, Apr 3, 2009 at 3:35 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Thu, Apr 2, 2009 at 5:56 PM, Jason Rutherglen > <jason.rutherg...@gmail.com> wrote: > >> I think I need to understand better why delete by Query isn't > > viable in your situation... > > > > The delete by query is a separate problem which I haven't fully > > explored yet. > > Oh, I had thought we were tugging on this thread in order to explore > delete-by-docID in the writer. OK. > > > Tracking the segment genealogy is really an > > interim step for merging field caches before column stride > > fields gets implemented. > > I see -- meaning in Bobo you'd like to manage your own memory resident > field caches, and merge them whenever IW has merged a segment? Seems > like you don't need genealogy for that. > > > Actually CSF cannot be used with Bobo's > > field caches anyways which means we'd need a way to find out > > about the segment parents. > > CSF isn't really designed yet. How come it can't be used with Bobo's > field caches? We can try to accommodate Bobo's field cache needs when > designing CSF. > > >> Does it operate at the segment level? Seems like that'd give > > you good enough realtime performance (though merging in RAM will > > definitely be faster). > > > > We need to see how Bobo integrates with LUCENE-1483. > > Lucene's internal field cache usage is now entirely at the segment > level (ie, Lucene core should never request full field cache array at > the MultiSegmentReader level). I think Bobo must have to do the same, > if it handles near realtime updates, to get adequate performance. > > Though... since we have LUCENE-831 (rework API Lucene exposes for > accessing arrays-of-atomic-types-per-segment) and LUCENE-1231 (CSF = a > more efficient impl (than uninversion) of the API we expose in > LUCENE-831) on deck, we should try to understand Bobo's needs. > > EG how come Bobo made its own field cache impl? Just because > uninversion is too slow? > > > It seems like we've been talking about CSF for 2 years and there > > isn't a patch for it? If I had more time I'd take a look. What > > is the status of it? > > I think Michael is looking into it? I'd really like to get it into > 2.9. We should do it in conjunction with 831 since they are so tied. > > > I'll write a patch that implements a callback for the segment > > merging such that the user can decide what information they want > > to record about the merged SRs (I'm pretty sure there isn't a > > way to do this with MergePolicy?) > > Actually I think you can do this w/ a simple MergeScheduler wrapper or > by subclassing CMS. I'll put a comment on the issue. > > Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
