Re: Lucene 9.0.0 inconsistent index options
Thanks for the response. https://issues.apache.org/jira/browse/LUCENE-10314
Will we still be able to decide, maybe years down the line, that we do want
to search on fieldX after all, and be able to change the code and reindex
the maybe small proportion of documents that have a value for fieldX
without having to create a new index from scratch? That happens.
--
Ian.
On Tue, Dec 14, 2021 at 1:54 PM Adrien Grand wrote:
> This looks related to the new changes around schema validation. Lucene
> now requires a field to either be absent from a document or be indexed
> with the exact same options (index options, points dimensions, norms,
> doc values type, etc.) as already indexed documents that also have
> this field.
>
> However it's a bug that Lucene fails to open an index that was legal
> in Lucene 8. Can you file a JIRA issue?
>
> On Mon, Dec 13, 2021 at 4:23 PM Ian Lea wrote:
> >
> > Hi
> >
> >
> > We have a long-standing index with some mandatory fields and some
> optional
> > fields that has been through multiple lucene upgrades without a full
> > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> > open an IndexWriter we are hitting the exception
> >
> > Exception in thread "main" java.lang.IllegalArgumentException: cannot
> > change field "language" from index options=NONE to inconsistent index
> > options=DOCS
> > at
> >
> org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> > at
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> > at
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> > at
> >
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> > at
> org.apache.lucene.index.IndexWriter.(IndexWriter.java:1116)
> >
> > Where language is one of our optional fields.
> >
> > Presumably this is at least somewhat related to "Index options can no
> > longer be changed dynamically" as mentioned at
> > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails
> before
> > our code attempts to update the index, and we are not trying to change
> any
> > index options.
> >
> > Adding some displays to IndexWriter and FieldInfos and logging rather
> than
> > throwing the exception I see
> >
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> >
> > where there is one line per segment. It logs the exception whenever
> > other=DOCS. Subset with segment info:
> >
> > segment _x8(8.2.0):c31753/-1:[diagnostics={timestamp=1565623850605,
> > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=NONE
> >
> > segment _y9(8.7.0):c43531/-1:[diagnostics={timestamp=1604597581562,
> > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=DOCS
> >
> > NOT throwing java.lang.IllegalArgumentException: cannot change field
> > "language" from index options=NONE to inconsistent index options=DOCS
> >
> >
> > Some variation on an old-fashioned not set versus not present bug
> perhaps?
> >
> >
> > --
> > Ian.
>
>
>
> --
> Adrien
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
Re: Lucene 9.0.0 inconsistent index options
Strictly speaking, we could have opened an older index using Lucene 8
(say one that was created using Lucene 7, or 6) that would no longer
be valid in Lucene 9, at least according to the policy? I agree we
should try to fix this, just want to clarify the policy
On Tue, Dec 14, 2021 at 8:54 AM Adrien Grand wrote:
>
> This looks related to the new changes around schema validation. Lucene
> now requires a field to either be absent from a document or be indexed
> with the exact same options (index options, points dimensions, norms,
> doc values type, etc.) as already indexed documents that also have
> this field.
>
> However it's a bug that Lucene fails to open an index that was legal
> in Lucene 8. Can you file a JIRA issue?
>
> On Mon, Dec 13, 2021 at 4:23 PM Ian Lea wrote:
> >
> > Hi
> >
> >
> > We have a long-standing index with some mandatory fields and some optional
> > fields that has been through multiple lucene upgrades without a full
> > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> > open an IndexWriter we are hitting the exception
> >
> > Exception in thread "main" java.lang.IllegalArgumentException: cannot
> > change field "language" from index options=NONE to inconsistent index
> > options=DOCS
> > at
> > org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> > at
> > org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> > at
> > org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> > at
> > org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> > at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1116)
> >
> > Where language is one of our optional fields.
> >
> > Presumably this is at least somewhat related to "Index options can no
> > longer be changed dynamically" as mentioned at
> > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
> > our code attempts to update the index, and we are not trying to change any
> > index options.
> >
> > Adding some displays to IndexWriter and FieldInfos and logging rather than
> > throwing the exception I see
> >
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> >
> > where there is one line per segment. It logs the exception whenever
> > other=DOCS. Subset with segment info:
> >
> > segment _x8(8.2.0):c31753/-1:[diagnostics={timestamp=1565623850605,
> > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=NONE
> >
> > segment _y9(8.7.0):c43531/-1:[diagnostics={timestamp=1604597581562,
> > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=DOCS
> >
> > NOT throwing java.lang.IllegalArgumentException: cannot change field
> > "language" from index options=NONE to inconsistent index options=DOCS
> >
> >
> > Some variation on an old-fashioned not set versus not present bug perhaps?
> >
> >
> > --
> > Ian.
>
>
>
> --
> Adrien
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Lucene 9.0.0 inconsistent index options
This looks related to the new changes around schema validation. Lucene
now requires a field to either be absent from a document or be indexed
with the exact same options (index options, points dimensions, norms,
doc values type, etc.) as already indexed documents that also have
this field.
However it's a bug that Lucene fails to open an index that was legal
in Lucene 8. Can you file a JIRA issue?
On Mon, Dec 13, 2021 at 4:23 PM Ian Lea wrote:
>
> Hi
>
>
> We have a long-standing index with some mandatory fields and some optional
> fields that has been through multiple lucene upgrades without a full
> rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> open an IndexWriter we are hitting the exception
>
> Exception in thread "main" java.lang.IllegalArgumentException: cannot
> change field "language" from index options=NONE to inconsistent index
> options=DOCS
> at
> org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> at
> org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> at
> org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> at
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1116)
>
> Where language is one of our optional fields.
>
> Presumably this is at least somewhat related to "Index options can no
> longer be changed dynamically" as mentioned at
> https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
> our code attempts to update the index, and we are not trying to change any
> index options.
>
> Adding some displays to IndexWriter and FieldInfos and logging rather than
> throwing the exception I see
>
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=DOCS
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
>
> where there is one line per segment. It logs the exception whenever
> other=DOCS. Subset with segment info:
>
> segment _x8(8.2.0):c31753/-1:[diagnostics={timestamp=1565623850605,
> lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> java.runtime.version=11.0.3+7,
> os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
>
> language curr=NONE, other=NONE
>
> segment _y9(8.7.0):c43531/-1:[diagnostics={timestamp=1604597581562,
> lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> java.runtime.version=11.0.3+7,
> os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
>
> language curr=NONE, other=DOCS
>
> NOT throwing java.lang.IllegalArgumentException: cannot change field
> "language" from index options=NONE to inconsistent index options=DOCS
>
>
> Some variation on an old-fashioned not set versus not present bug perhaps?
>
>
> --
> Ian.
--
Adrien
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
