[
https://issues.apache.org/jira/browse/LUCENE-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308845#comment-17308845
]
Greg Miller commented on LUCENE-9850:
-------------------------------------
[~jpountz] interesting thought! I'm using *wikimedium.10M.nostopwords.tasks* in
the benchmark. Note that in my benchmark run, I did incorporate your earlier
feedback for special-casing the 0 bpv case (thanks!). So I think my branch
should be handling the 0 bpv cases in a reasonably optimal way, but here's that
[diff|https://github.com/apache/lucene/compare/main...gsmiller:LUCENE-9850/pfordocids#diff-9f4cb4a664b2a8f0594b221368085548a58ecb1cc1290f18160b613d400fcc29].
Maybe you see something I overlooked. Appreciate the thoughts!
> Explore PFOR for Doc ID delta encoding (instead of FOR)
> -------------------------------------------------------
>
> Key: LUCENE-9850
> URL: https://issues.apache.org/jira/browse/LUCENE-9850
> Project: Lucene - Core
> Issue Type: Task
> Components: core/codecs
> Affects Versions: main (9.0)
> Reporter: Greg Miller
> Priority: Minor
>
> It'd be interesting to explore using PFOR instead of FOR for doc ID encoding.
> Right now PFOR is used for positions, frequencies and payloads, but FOR is
> used for doc ID deltas. From a recent
> [conversation|http://mail-archives.apache.org/mod_mbox/lucene-dev/202103.mbox/%3CCAPsWd%2BOp7d_GxNosB5r%3DQMPA-v0SteHWjXUmG3gwQot4gkubWw%40mail.gmail.com%3E]
> on the dev mailing list, it sounds like this decision was made based on the
> optimization possible when expanding the deltas.
> I'd be interesting in measuring the index size reduction possible with
> switching to PFOR compared to the performance reduction we might see by no
> longer being able to apply the deltas in as optimal a way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
