Benson, I like your idea. I think your idea can be achieved as a codec, one that wraps another codec that establishes the on-disk format. By default the wrapped codec can be Lucene's default codec. I think, if implemented, this would be a change to DPF instead of an additional DPF-variant codec.
~ David On Mon, Apr 7, 2014 at 9:22 AM, Benson Margulies <bimargul...@gmail.com>wrote: > On Mon, Apr 7, 2014 at 9:14 AM, Robert Muir <rcm...@gmail.com> wrote: > > On Thu, Apr 3, 2014 at 12:27 PM, Benson Margulies <bimargul...@gmail.com> > wrote: > > > >> > >> My takeaway from the prior conversation was that various people didn't > >> entirely believe that I'd seen a dramatic improvement in query perfo > >> using D-P-F, and so would not smile upon a patch intended to liberate > >> D-P-F from codecs. It could be that the effect I saw has to do with > >> the fact that our system depends on hitting and scoring 50% of the > >> documents in an index with a lot of documents. > >> > > > > I dont understand the word "liberate" here. why is it such a problem > > that this is a codec? > > I don't want to have to declare my intentions at the time I create > the index. I don't want to have to use D-P-F for all readers all the > time. Because I want to be able to decide to open up an index with an > arbitrary on-disk format and get the in-memory cache behavior of > D-P-F. Thus 'liberate' -- split the question of 'keep a copy in > memory' from the choice of the on-disk format. > > > > > > i do not think we should give it any more status than that, it wastes > > too much ram. > > It didn't seem like 'waste' when it solved a big practical for us. We > had an application that was too slow, and had plenty of RAM available, > and we were able to trade space for time by applying D-P-F. > > Maybe I'm going about this backwards; if I can come up with a small, > inconspicuous proposed change that does what I want, there won't be > any disagreement. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
