[
https://issues.apache.org/jira/browse/LUCENE-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787683#comment-13787683
]
Shai Erera commented on LUCENE-5189:
------------------------------------
Let me explain: SCR is still used for all the shared objects between _all_ SRs,
objects that are closed when the last of the SRs using this 'core' is closed.
With gen'd DVPs, they no longer belong in SCR, because there isn't a single DVP
that _all_ SRs share. For example, suppose that you start with an index without
updates, then the DV fields gen is -1. So a DVP for field 'f' will be created.
Next you update a field and reopen, the gen for 'f' is incremented to 1 and a
new DVP is needed. The DVP for gen=-1 is no longer needed.
Rob, if we did what you propose - store the initial DVPs in SCR and the updated
ones in SRs (as they are updated and reopened), Soon the DVPs in SCR will not
be used by any SR, and just hang there (possibly consuming expensive RAM, e.g.
MemoryDVF) until the last SR is closed.
Rather, DVPs *are shared* between SRs. RefCount is a simple object which keeps
ref-counting for an object. When an SR is opened anew (e.g. initializes a new
SCR), the DVPs it initializes all have RefCount(1). When an SR is opened by
sharing another SR (e.g. NRT, DirReader.openIfChanged), it lists all the fields
with DV. If the other SR has a DVP for their dvGen, it reuses it and inc-ref
it, otherwise it opens a new DVP with RC(1).
SR itself doesn't need any ref-counting, it's the DVPs that need them. Putting
them in SCR I think only complicates things (or at least doesn't simplify). For
example, currently when SR.doClose is called, it dec-ref all DVPs that it uses.
And DocValuesRefCount.release() closes the DVP when its ref-count reaches 0. If
we moved all the DVPs to SCR, then SR.doClose would need to go an dec-ref all
DVPs that it uses in SCR .. but how does it know which DVP it uses if all DVPs
just sit there in SCR - even ones that it doesn't use? I think that that they
are in SR actually simplifies and keeps the code clear.
Rob, if you don't use DV updates, all DV fields have dvGen=-1 and they are
shared between all SRs.
> Numeric DocValues Updates
> -------------------------
>
> Key: LUCENE-5189
> URL: https://issues.apache.org/jira/browse/LUCENE-5189
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Attachments: LUCENE-5189-4x.patch, LUCENE-5189.patch,
> LUCENE-5189.patch, LUCENE-5189.patch, LUCENE-5189.patch, LUCENE-5189.patch,
> LUCENE-5189.patch, LUCENE-5189.patch, LUCENE-5189.patch, LUCENE-5189.patch,
> LUCENE-5189.patch, LUCENE-5189.patch, LUCENE-5189_process_events.patch,
> LUCENE-5189_process_events.patch, LUCENE-5189-updates-order.patch,
> LUCENE-5189-updates-order.patch
>
>
> In LUCENE-4258 we started to work on incremental field updates, however the
> amount of changes are immense and hard to follow/consume. The reason is that
> we targeted postings, stored fields, DV etc., all from the get go.
> I'd like to start afresh here, with numeric-dv-field updates only. There are
> a couple of reasons to that:
> * NumericDV fields should be easier to update, if e.g. we write all the
> values of all the documents in a segment for the updated field (similar to
> how livedocs work, and previously norms).
> * It's a fairly contained issue, attempting to handle just one data type to
> update, yet requires many changes to core code which will also be useful for
> updating other data types.
> * It has value in and on itself, and we don't need to allow updating all the
> data types in Lucene at once ... we can do that gradually.
> I have some working patch already which I'll upload next, explaining the
> changes.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
