[
https://issues.apache.org/jira/browse/LUCENE-10033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17405826#comment-17405826
]
Greg Miller commented on LUCENE-10033:
--------------------------------------
[~jpountz] I realized yesterday that the benchmarks I was running for your
change were an unfair comparison. They were comparing your patch against the
Lucene80 doc values codec instead of Lucene90 without your change. I discovered
yesterday that moving from Lucene80 to Lucene90 (as it exists currently on
{{main}}) is actually a regression for our application in a few dimensions. So
I'll need to spend more time figuring that out, but that's a separate topic :)
To create a more fair comparison, I created a baseline branch internally that
moves us to Lucene90 and then compared your change on top of that. The
regressions are still there (and still significant), but not quite as large.
Note that I have not grabbed your latest code yet, so this is running the same
version of your change that I ran back on 8/17 (with one iteration of
speedups/improvements):
# red-line qpg regressed by ~9%
# latency overall increased on average by 13% (9% p50 / 8% p99.9)
# our facet counting phase increased in latency on average by 6% (0% p50 / 14%
p99.9)
> Encode doc values in smaller blocks of values, like postings
> ------------------------------------------------------------
>
> Key: LUCENE-10033
> URL: https://issues.apache.org/jira/browse/LUCENE-10033
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: benchmark, benchmark-10m
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This is a follow-up to the discussion on this thread:
> https://lists.apache.org/thread.html/r7b757074d5f02874ce3a295b0007dff486bc10d08fb0b5e5a4ba72c5%40%3Cdev.lucene.apache.org%3E.
> Our current approach for doc values uses large blocks of 16k values where
> values can be decompressed independently, using DirectWriter/DirectReader.
> This is a bit inefficient in some cases, e.g. a single outlier can grow the
> number of bits per value for the entire block, we can't easily use run-length
> compression, etc. Plus, it encourages using a different sub-class for every
> compression technique, which puts pressure on the JVM.
> We'd like to move to an approach that would be more similar to postings with
> smaller blocks (e.g. 128 values) whose values get all decompressed at once
> (using SIMD instructions), with skip data within blocks in order to
> efficiently skip to arbitrary doc IDs (or maybe still use jump tables as
> today's doc values, and as discussed here for postings:
> https://lists.apache.org/thread.html/r7c3cb7ab143fd4ecbc05c04064d10ef9fb50c5b4d6479b0f35732677%40%3Cdev.lucene.apache.org%3E).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
