Yeah, have a look at gen_ForUtil.py On Mon, Mar 1, 2021 at 1:05 PM Greg Miller <gsmil...@gmail.com> wrote:
> Thanks for the feedback Robert; makes sense to me. I'll tinker with a > forked codec and see if the experimentation produces anything interesting. > > When you mention "autogenerated decompression code", do you mean that > some of this code is actually being generated? > > Cheers, > -Greg > > On Sun, Feb 28, 2021 at 5:05 AM Robert Muir <rcm...@gmail.com> wrote: > >> If you want to test a different block size (say 64 or 256), I really >> recommend to just fork a different codec for the experiment. >> >> There will likely be higher level changes you need to make, not just >> changing a number. For example if you just increased this number to 256 >> without doing anything else, I wouldn't be surprised if you see worse >> performance. More of the postings would be vint-encoded than before with >> 128, which might have some consequences. skipdata layout might be >> inappropriate, these things are optimized for blocks of 128. >> >> Just in general, I recommend making a codec for the benchmarking >> experiments, tools like luceneutil support comparing codecs against each >> other anyway so you can easily compare fairly against the existing codec. >> Also, it should be much easier/faster to just make a new codec and adapt it >> to test what you want! >> >> I think it is an antipattern to make stuff within the codec "flexible", >> it is autogenerated decompression code :) I am concerned such "flexibility" >> would create barriers in the future to optimizations. For example we should >> be able to experiment with converting this compression code over to >> explicit vector API in java. >> >> On Sat, Feb 27, 2021 at 4:29 PM Greg Miller <gsmil...@gmail.com> wrote: >> >>> Hi folks! >>> >>> I've been a bit curious to test out different block size configurations >>> in the Lucene postings list format, but thought I'd reach out to the >>> community here first to see what work may have gone into this previously. >>> I'm essentially interested in benchmarking different block size >>> configurations on the real-world application of Lucene I'm working on. >>> >>> If my understanding of the code is correct, I know we're currently >>> encoding compressed runs of 128 docs per block, relying on ForUtil for >>> encoding/decoding purposes. It looks like we define this in >>> ForUtil#BLOCK_SIZE (and reference it in a few external classes), but also >>> know that it's not as simple as just changing that one definition. It >>> appears much of the logic in ForUtil relies on the assumption of 128 >>> docs-per-block. >>> >>> I'm toying with the idea of making ForUtil a bit more flexible to allow >>> for different block sizes to be tested in order to run the benchmarking I'd >>> like to run, but the class looks heavily optimized to generate SIMD >>> instructions (I think?), so that might be folly. Before I start hacking on >>> a local branch to see what I can learn, is there any prior work that might >>> be useful to be aware of? Anyone gone down this path and have some >>> learnings to share? Any thoughts would be much appreciated! >>> >>> Cheers, >>> -Greg >>> >>
