[
https://issues.apache.org/jira/browse/LUCENE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688331#comment-16688331
]
Adrien Grand commented on LUCENE-8551:
--------------------------------------
Field infos are tracked at IndexWriter#globalFieldNumberMap and written with
every segment. My understanding of this issue is that we would like to
garbage-collect unused fields, so I was thinking that if a point-in-time view
is open, then a field is removed, then a field with the same name is added back
with a different configuration, then the point-in-time view that got open
before the field got removed would have different field infos from IndexWriter.
The fact that field infos are currently (almost) append-only makes things
easier to reason about. It's a different issue but related: for a long time
index options could be downgraded. Only until we figured out that this
prevented a relevancy bug (LUCENE-8031) from being addressed, so we removed
this feature (LUCENE-8134).
> Purge unused FieldInfo on segment merge
> ---------------------------------------
>
> Key: LUCENE-8551
> URL: https://issues.apache.org/jira/browse/LUCENE-8551
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: David Smiley
> Priority: Major
>
> If a field is effectively unused (no norms, terms index, term vectors,
> docValues, stored value, points index), it will nonetheless hang around in
> FieldInfos indefinitely. It would be nice to be able to recognize an unused
> FieldInfo and allow it to disappear after a merge (or two).
> SegmentMerger merges FieldInfo (from each segment) as nearly the first thing
> it does. After that, the different index parts, before it's known what's
> "used" or not. After writing, we theoretically know which fields are used or
> not, though we're not doing any bookkeeping to track it. Maybe we should
> track the fields used during writing so we write a filtered merged fieldInfo
> at the end instead of unfiltered up front? Or perhaps upon reading a
> segment, we make it cheap/easy for each index type (e.g. terms index, stored
> fields, ...) to know which fields have data for the corresponding type.
> Then, on a subsequent merge, we know up front to filter the FieldInfos.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
