Wow, those gains are substantial: PrefixQuery http://people.apache.org/~mikemccand/lucenebench/Prefix3.html
WildcardQuery http://people.apache.org/~mikemccand/lucenebench/Wildcard.html I will add an annotation for this change :) The motivation for FBS was to store docIDs, and since those are limited to a java int, we used an int index back then. You're right that OpenBitSet has "elasticity", but one could also achieve that by growing a FBS. Maybe it should be renamed to ElasticBitSet. It does also allow long bit sets ... probably if we added get/set(long) to FBS that wouldn't hurt performance? If we replaced get/set(int) with get/set(long) I'm not sure... Mike McCandless http://blog.mikemccandless.com On Thu, Feb 6, 2014 at 3:39 AM, Shai Erera <[email protected]> wrote: > Hi > > I recently modified FixedBitSet.iterator() to return its own iterator, > instead of OpenBitSetIterator, and while I haven't done a thorough > performance test, the results on LUCENE-5425 suggested this helped in > general (even though we only tested facets). Also, last night's benchmark > for WildcardQuery and PrefixQuery indicate a 20% jump in QPS. > > This got me thinking (again) why do we keep both bitsets, and whether we > couldn't live with just one. So clearly FBS is faster than OBS (perhaps > unless you use fastSet/Get) since it doesn't need to do bounds checking. > Also, FBS lets your grow itself by offering a convenient copy constructor > which allows to expand/shrink the set. > > OpenBigSet however lets you manage 64 * 2^32 bits, while FixedBitSet only > allows managing 2^32 bits. Is there any reason for this? Besides the fact > that neither can return a DISI over more than 2^32-1 bits/docs, I don't see > any real reason for that. We'll need to change FBS API to take/return long, > but besides that? > > I did a quick search on the usage of OpenBitSet and eclipse found 272 > mentions of it. I believe some, if not all, of these uses can be replaced by > FixedBitSet. Definitely when the number of bits is known in advance, but I > think also when it's not known, by growing the bitset. > > One place I found is DocValuesConsumer, which clearly knows the number of > bits in advance (dv.getValueCount()), however this returns a long, so this > prevents cutting over to FBS immediately, but if we allow FBS to handle > longs, I think that we can? > > Anyway, wanted to get your thoughts before I open an issue and start this > work. > > Shai --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
