[
https://issues.apache.org/jira/browse/SOLR-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097600#comment-18097600
]
Sanjay Dutt commented on SOLR-18251:
------------------------------------
Thank you [~vkirilchuk] for continue to work on it and I can discard mine PR.
I ran the benchmark on my PR (should be same for yours) just to further
understand the benefits of enabling skip list.
*Result:* a doc-values field with skip list matches BKD-tree range-query
throughput, with no BKD tree on disk. At 5M docs (long field, random values,
narrow/medium/wide selectivity), confirmed again at 100M docs:
||Heading 1||narrow (ops/s)||medium (ops/s)||wide (ops/s)||
|BKD (indexed=true)|12,227|12,518|12,198|
|skip-list + index sort|12,005|11,333|11,438|
|plain doc values|41|40|31|
*Condition:* this throughput parity holds only when the field's value is
monotonically correlated with docId order - via natural insertion order, or
forced by an explicit Lucene index sort. Remove the correlation the throughput
drops down to the same as plain doc values with no skip list.
*Practical implication:* today, BKD-speed range queries plus doc values support
(sort/facet/function queries) means maintaining both structures (indexed=true,
docValues=true). This feature lets you drop the BKD tree (indexed=false) and
keep BKD-equivalent range-query speed, provided the field is correlated with
docId order - avoiding a duplicate on-disk structure, not just adding a faster
query path.
> Add Solr schema support for DocValues skip lists
> ------------------------------------------------
>
> Key: SOLR-18251
> URL: https://issues.apache.org/jira/browse/SOLR-18251
> Project: Solr
> Issue Type: Task
> Components: Schema and Analysis
> Affects Versions: 10.1
> Reporter: Sanjay Dutt
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Lucene introduced optional skip lists on top of DocValues in
> apache/lucene#13449. The feature is exposed through the {{DocValuesSkipper}}
> abstraction and is controlled by a Lucene {{FieldType}} flag.
> Solr currently exposes DocValues through schema configuration, but does not
> provide a schema-level option to enable this DocValues skip list support.
> This issue proposes adding an opt-in Solr schema option, such as:
> {code:java}
> docValuesSkipList="true"{code}
> Example:
> {code:java}
> <field name="price"
> type="plong"
> indexed="false"
> docValues="true"
> docValuesSkipList="true" />{code}
> The Lucene PR notes that this can be efficient when the index is sorted and
> the field belongs to the index sorting.
> The actual benefit depends on data layout and query pattern, so this should
> be exposed as an opt-in feature.
> Only compatible DocValues field types should allow this option. Based on the
> Lucene PR, this applies to DocValues types such as {{{}NUMERIC{}}},
> {{{}SORTED_NUMERIC{}}}, {{{}SORTED{}}}, and {{{}SORTED_SET{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
