Enabling/disabling docValues

2019-06-09 Thread John Davis
Hi there, We recently changed a field from TextField + no docValues to SortableTextField which has docValues enabled by default. Once I did this I do not see any facet values for the field. I know that once all the docs are re-indexed facets should work again, however can someone clarify the curren

Re: Enabling/disabling docValues

2019-06-09 Thread Erick Erickson
It’s basically undefined. When segments are merged that have dissimilar definitions like this what can Lucene do? Consider: Faceting on a text (not sortable) means that each individual token in the index is uninverted on the Java heap and the facets are computed for each individual term. Facet

Re: Enabling/disabling docValues

2019-06-09 Thread John Davis
Understood, however code is rarely random/undefined. Does lucene look at % docs in each state, or the first doc or something else? On Sun, Jun 9, 2019 at 1:58 PM Erick Erickson wrote: > It’s basically undefined. When segments are merged that have dissimilar > definitions like this what can Lucen

Re: Enabling/disabling docValues

2019-06-10 Thread Erick Erickson
bq. Does lucene look at %docs in each state, or the first doc or something else? Frankly I don’t care since no matter what, the results of faceting mixed definitions is not useful. tl;dr; “When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean

Re: Enabling/disabling docValues

2019-06-10 Thread John Davis
You have made many assumptions which might not always be realistic a) TextField is always tokenized b) Users care about precise counts and c) Users have the luxury or ability to do a full re-index anytime. These are real issues and there is no black/white solution. I will ask Lucene folks on the ac

Re: Enabling/disabling docValues

2019-06-11 Thread Gus Heck
On Mon, Jun 10, 2019 at 10:53 PM John Davis wrote: > You have made many assumptions which might not always be realistic a) > TextField is always tokenized Well, you could of course change configuration or code to do something else but this would be a very odd and misleading thing to do and we w

Re: Enabling/disabling docValues

2019-06-11 Thread John Davis
There is no way to match case insensitive without TextFields + no tokenization. Its a long standing limitation of not being able to apply any analyzers with str fields. Thanks for pointing out the re-index page I've seen it. However sometimes it is hard to re-index in a reasonable amount of time &