Re: Change field to DocValues

2021-02-17 Thread Mahmoud Almokadem
That's right, I want to avoid a complete reindexing process. But should I create another field with the docValues property or change the current field directly? Can I use streaming expressions to update the whole index or should I select and update using batches? Thanks, Mahmoud On Wed, Feb

Re:Change field to DocValues

2021-02-17 Thread xiefengchang
r field on an index with billions of documents and need to do >facets on this field, unfortunately the field doesn't have the docValues >property, so the FieldCache will be fired and use much memory. > >What is the best way to change the field to be docValues supported? > >Regards, >Mahmoud

Change field to DocValues

2021-02-17 Thread Mahmoud Almokadem
Hello, I've an integer field on an index with billions of documents and need to do facets on this field, unfortunately the field doesn't have the docValues property, so the FieldCache will be fired and use much memory. What is the best way to change the field to be docValues supported? Regards

Re: docValues usage

2020-11-04 Thread Wei
And in the case of both stored=true and docValues=true, Solr 8.x shall be choosing the optimal approach by itself? On Wed, Nov 4, 2020 at 9:15 AM Wei wrote: > Thanks Erick. As indexed is not necessary, and docValues is more > efficient than stored fields for function queries, so we sh

Re: docValues usage

2020-11-04 Thread Wei
Thanks Erick. As indexed is not necessary, and docValues is more efficient than stored fields for function queries, so we shall go with the following: 3) indexed=false, stored=false, docValues=true. Is my understanding correct? Best, Wei On Wed, Nov 4, 2020 at 5:24 AM Erick Erickson

Re: docValues usage

2020-11-04 Thread Erick Erickson
the question “for doc X, what is the value of field Y” to compute the score. DocValues are still more efficient I think, although I haven’t measured explicitly... For sorting, faceting and grouping, it’s a much different story. Take sorting. You have to ask “for field Y, what’s the value in docX

Re: when to use stored over docValues and useDocValuesAsStored

2020-11-04 Thread Erick Erickson
> On Nov 4, 2020, at 6:43 AM, uyilmaz wrote: > > Hi, > > I heavily use streaming expressions and facets, or export large amounts of > data from Solr to Spark to make analyses. > > Please correct me if I know wrong: > > + requesting a non-docValues field in a

when to use stored over docValues and useDocValuesAsStored

2020-11-04 Thread uyilmaz
Hi, I heavily use streaming expressions and facets, or export large amounts of data from Solr to Spark to make analyses. Please correct me if I know wrong: + requesting a non-docValues field in a response causes whole document to be decompressed and read from disk + streaming expressions

Re: docValues usage

2020-11-04 Thread uyilmaz
-Thebf_BoostFunctions_Parameter). It's like boosting a regular field by putting ^2 next to it in a query. Storing or enabling docValues will unnecesarily consume space/memory. On Tue, 3 Nov 2020 16:10:50 -0800 Wei wrote: > Hi, > > I have a couple of primitive single value numeric type fields, their

docValues usage

2020-11-03 Thread Wei
Hi, I have a couple of primitive single value numeric type fields, their values are used in boosting functions, but not used in sort/facet. or in returned response. Should I use docValues for them in the schema? I can think of the following options: 1) indexed=true, stored=true

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread uyilmaz
nalysis is too complex to > do in Solr, we export large query results from Solr to a more capable > analysis tool. > > So I guess all fields need to be docValues="true", because export handler and > streaming both require fields to have docValues, and even if I won't

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread uyilmaz
all fields need to be docValues="true", because export handler and streaming both require fields to have docValues, and even if I won't use a field in queries or facets, it should be in available to read in result set. Fields that won't be searched or faceted can be (indexed=false st

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread Walter Underwood
Hmm. Fields used for faceting will also be used for filtering, which is a kind of search. Are docValues OK for filtering? I expect they might be slow the first time, then cached. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 19, 2020, at 11

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread Erick Erickson
uyilmaz: Hmm, that _is_ confusing. And inaccurate. In this context, it should read something like The Text field should have indexed="true" docValues=“false" if used for searching but not faceting and the String field should have indexed="false" docValues=“t

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread Michael Gibney
As you've observed, it is indeed possible to facet on fields with docValues=true, indexed=false; but in almost all cases you should probably set indexed=true. 1. for distributed facet count refinement, the "indexed" approach is used to look up counts by value; 2. assuming you're wan

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread uyilmaz
e two versions of the field: one Text and one String. Make sure both are indexed="true"." On Mon, 19 Oct 2020 13:08:00 -0400 Alexandre Rafalovitch wrote: > I think this is all explained quite well in the Ref Guide: > https://lucene.apache.org/solr/guide/8_6/docvalues.html >

Re: Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread Alexandre Rafalovitch
I think this is all explained quite well in the Ref Guide: https://lucene.apache.org/solr/guide/8_6/docvalues.html DocValues is a different way to index/store values. Faceting is a primary use case where docValues are better than what 'indexed=true' gives you. Regards, Alex. On Mon, 19 Oct

Faceting on indexed=false stored=false docValues=true fields

2020-10-19 Thread uyilmaz
Hey all, >From my little experiments, I see that (if I didn't make a stupid mistake) we >can facet on fields marked as both indexed and stored being false: I'm suprised by this, I thought I would need to index it. Can you confirm this? Regards -- uyilmaz

Re: facets & docValues

2020-05-13 Thread ART GALLERY
nstein > > > http://joelsolr.blogspot.com/ > > > > > > > > > On Mon, May 4, 2020 at 10:27 AM Revas wrote: > > > > > > > Hi Erick, Thanks for the explanation and advise. With facet queries, > > does > > > > doc Values help at

Re: facets & docValues

2020-05-07 Thread Joel Bernstein
blogspot.com/ > > > > > > On Mon, May 4, 2020 at 10:27 AM Revas wrote: > > > > > Hi Erick, Thanks for the explanation and advise. With facet queries, > does > > > doc Values help at all ? > > > > > > 1) indexed=true, docValues=true =>

Re: facets & docValues

2020-05-05 Thread Revas
t queries, does > > doc Values help at all ? > > > > 1) indexed=true, docValues=true => all facets > > > > 2) > > > > - indexed=true , docValues=true => only for subfacets > >- inexed=true, docValues=false=> facet query > >

Re: facets & docValues

2020-05-05 Thread Joel Bernstein
/ On Mon, May 4, 2020 at 10:27 AM Revas wrote: > Hi Erick, Thanks for the explanation and advise. With facet queries, does > doc Values help at all ? > > 1) indexed=true, docValues=true => all facets > > 2) > >- indexed=true , docValues=true => only for subfacets

Re: facets & docValues

2020-05-04 Thread Revas
Hi Erick, Thanks for the explanation and advise. With facet queries, does doc Values help at all ? 1) indexed=true, docValues=true => all facets 2) - indexed=true , docValues=true => only for subfacets - inexed=true, docValues=false=> facet query - docValues=true, inde

Re: facets & docValues

2020-04-16 Thread Erick Erickson
DocValues should help when faceting over fields, i.e. facet.field=blah. I would expect docValues to help with sub facets and, but don’t know the code well enough to say definitely one way or the other. The empirical approach would be to set “uninvertible=true” (Solr 7.6) and turn docValues off

Re: facets & docValues

2020-04-16 Thread Revas
Hi Erick, You are correct, we have only about 1.8M documents so far and turning on the indexing on the facet fields helped improve the timings of the facet query a lot which has (sub facets and facet queries). So does docValues help at all for sub facets and facet query, our tests revealed further

Re: facets & docValues

2020-04-15 Thread Erick Erickson
In a word, “yes”. I also suspect your corpus isn’t very big. I think the key is the facet queries. Now, I’m talking from theory rather than diving into the code, but querying on a docValues=true, indexed=false field is really doing a search. And searching on a field like that is effectively

facets & docValues

2020-04-14 Thread Revas
We have faceting fields that have been defined as indexed=false, stored=false and docValues=true However we use a lot of subfacets using json facets and facet ranges using facet.queries. We see that after every soft-commit our performance worsens and performs ideal between commits how

Re: Does it make sense docValues="true" for _root_ field for uniqueBlock()

2020-01-22 Thread Mikhail Khludnev
It's hard to predict will it be faster read docValues files or uninvert field ad-hoc and read them from heap. Only test might judge it. On Wed, Jan 22, 2020 at 11:08 PM kumar gaurav wrote: > HI Mikhail > > for example :- 6GB index size (Parent-child documents) > indexing in 12 ho

Re: Does it make sense docValues="true" for _root_ field for uniqueBlock()

2020-01-22 Thread kumar gaurav
HI Mikhail for example :- 6GB index size (Parent-child documents) indexing in 12 hours interval . need to use uniqueBlock for json facet for child faceting . Should i use docValues="true" for _root_ field ? Thanks . regards Kumar Gaurav On Thu, Jan 23, 2020 at 1:28 AM Mikhai

Re: Does it make sense docValues="true" for _root_ field for uniqueBlock()

2020-01-22 Thread Mikhail Khludnev
It depends from env. On Wed, Jan 22, 2020 at 9:31 PM kumar gaurav wrote: > Hi Everyone > > Should i use docValues="true" for _root_ field to improve nested child > json.facet performance ? i am using uniqueBlock() . > > > Thanks in advance . > > regards &

Does it make sense docValues="true" for _root_ field for uniqueBlock()

2020-01-22 Thread kumar gaurav
Hi Everyone Should i use docValues="true" for _root_ field to improve nested child json.facet performance ? i am using uniqueBlock() . Thanks in advance . regards Kumar Gaurav

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-11 Thread Erick Erickson
GCEasy works fine. GCViewer is something you can have on your local machine, sometimes if you have very large GC logs uploading them can take quite a while. The next step, if you can’t find anything satisfactory is to put a profiler on the running Solr instance, which will tell you where the time

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-11 Thread Paras Lehana
Hi Erick, You're right - IO was extraordinarily high. But something odd happened. To actually build a relation, I tried different heap sizes with default solrconfig.xml values as you recommended. 1. Increased RAM to 4G, speed 8500k. 2. Decreased to 2G, back to old 65k. 3. Increased back

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-11 Thread Erick Erickson
I doubt GC alone would make nearly that difference. More likely it’s I/O interacting with MMapDirectory. Lucene uses OS memory space for much of its index, i.e. the RAM left over after that used for the running Solr process (and any other processes of course). See:

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-11 Thread Paras Lehana
Just to update, I kept the defaults. The indexing got only a little boost though I have decided to continue with the defaults and do incremental experiments only. To my surprise, our development server had only 12GB RAM, of which 8G was allocated to Java. Because I could not increase the RAM, I

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-09 Thread Erick Erickson
Note that that article is from 2011. That was in the Solr 3x days when many, many, many things were different. There was no SolrCloud for instance. Plus Tom’s problem space is indexing _books_. Whole, complete, books. Which is, actually, not “normal” indexing at all as most Solr indexes are

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-08 Thread Paras Lehana
Hi Erick, I have reverted back to original values and yes, I did see improvement. I will collect more stats. *Thank you for helping. :)* Also, here is the reference article that I had referred for changing values:

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-06 Thread Paras Lehana
Hey Erick, We have just upgraded to 8.3 before starting the indexing. We were on 6.6 before that. Thank you for your continued support and resources. Again, I have already taken your suggestion to start afresh and that's what I'm going to do. Don't get me wrong but I have been just asking

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-06 Thread Erick Erickson
Nothing implicitly handles optimization, you must continue to do that externally. Until you get to the bottom of your indexing slowdown, I wouldn’t bother with it at all, trying to do all these things at once is what lead to your problem in the first place, please change one thing at a time. You

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-05 Thread Paras Lehana
Hi Erick, I believed optimizing explicitly merges segments and that's why I was expecting it to give performance boost. I know that optimizations should not be done very frequently. For a full indexing, optimizations occurred 30 times between batches. I take your suggestion to undo all the

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-05 Thread Erick Erickson
> I think I should have also done optimize between batches, no? No, no, no, no. Absolutely not. Never. Never, never, never between batches. I don’t recommend optimizing at _all_ unless there are demonstrable improvements. Please don’t take this the wrong way, the whole merge process is really

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-04 Thread Paras Lehana
Hey Erick, This is a huge red flag to me: "(but I could only test for the first few > thousand documents”. Yup, that's probably where the culprit lies. I could only test for the starting batch because I had to wait for a day to actually compare. I tweaked the merge values and kept whatever gave

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-04 Thread Erick Erickson
This is a huge red flag to me: "(but I could only test for the first few thousand documents” You’re probably right that that would speed things up, but pretty soon when you’re indexing your entire corpus there are lots of other considerations. The indexing rate you’re seeing is abysmal unless

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-04 Thread Paras Lehana
; uniqueKey and then does the normal index using the information in the > > fetched document. Is this actually taking time? As the number of > documents > > increases, Solr might be taking time to fetch the stored document. > > > > But shouldn't the fetch by unique

Re: [Q] Faster Atomic Updates - use docValues?

2019-12-03 Thread Erick Erickson
ocument. Is this actually taking time? As the number of documents > increases, Solr might be taking time to fetch the stored document. > > But shouldn't the fetch by uniqueKey take O(1) time? If this really impacts > the fetch, can we use docValues for the field id (uniqueKe

[Q] Faster Atomic Updates - use docValues?

2019-12-03 Thread Paras Lehana
might be taking time to fetch the stored document. But shouldn't the fetch by uniqueKey take O(1) time? If this really impacts the fetch, can we use docValues for the field id (uniqueKey)? Our field is of type string. I'm pasting my config l

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-11 Thread Antony Alphonse
update request processor > chain and have all the logic in Solr. Here is blog post explaining how it > could work: > https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html < > https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html> > > HTH, &g

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-11 Thread Emir Arnautović
Hi Antony, Like Erick explained, you still have to preprocess your field in order to be able to use doc values. What you can do is use update request processor chain and have all the logic in Solr. Here is blog post explaining how it could work: https://www.od-bits.com/2018/02/solr-docvalues

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-10 Thread Erick Erickson
So “lowercase” is, indeed, a solr.TextField, which is ineligible for docValues. Given that definition, the difference will be that a “string” type is totally un-analyzed, so the values that go into the index and the query itself will be case-sensitive. You’ll have to pre-process both to do

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-09 Thread Antony Alphonse
our use case. > 2> we don’t know what the field type “lowercase” does. It’s composed of an > analysis chain that you may have changed. And whatever config you are using > may have changed with different releases of Solr. > > Grouping is generally done on a docValues-eligible field typ

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-09 Thread Erick Erickson
ferent releases of Solr. Grouping is generally done on a docValues-eligible field type. AFAIK, “lowercase” is a solr-text based field so is ineligible for docValues. I’ve got to guess here, but I’d suggest you start with a fieldType of “string”, and enable docValues on it. Best, Erick > On No

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-08 Thread Antony Alphonse
> > Hi Shawn, > I will try that solution. Also I had to mention that the queries that fail with this error has the "group.field":"lowercase". Should I change the field type? Thanks, Antony

Re: Solr 7.2.1 - unexpected docvalues type

2019-11-08 Thread Shawn Heisey
java.lang.IllegalStateException: unexpected docvalues type SORTED_SET for field 'lowercase' (expected=SORTED). Re-index with correct docvalues type. This error means that part of the index was created with one definition for the field in question, then the schema was changed in an incompatible way

Solr 7.2.1 - unexpected docvalues type

2019-11-08 Thread Antony Alphonse
docvalues type SORTED_SET for field 'lowercase' (expected=SORTED). Re-index with correct docvalues type. at org.apache.lucene.index.DocValues.checkField(DocValues.java:340) at org.apache.lucene.index.DocValues.getSorted(DocValues.java:392

Re: In Place Updates: Can we filter on fields with only docValues="true"

2019-09-15 Thread Erick Erickson
an ordered list of terms, each term has the list of documents it appears in. DocValues is the exact opposite. It answers “For doc X, what is the value of field Y?”. When _searching_ on a DV only field, think “table scan” in DB terms. Pick a field with high cardinality. Worst-case, every doc has a unique

Re: In Place Updates: Can we filter on fields with only docValues="true"

2019-09-14 Thread Shawn Heisey
On 9/14/2019 4:29 PM, Mikhail Khludnev wrote: Shawn, would you mind to provide some numbers? I'm experimenting with lucene 8.0.0. I have 100 shard index of 100M docs with 2000 docVals only updateable fields. Searching for such field turns to be blazingly fast $ curl

Re: In Place Updates: Can we filter on fields with only docValues="true"

2019-09-14 Thread Mikhail Khludnev
"boost" : 1.0, "index" : false, "store" : false, "doc_values" : true, "term_vector" : "no", "norms" : false, "eager_global_ordinals" : fa

Re: In Place Updates: Can we filter on fields with only docValues="true"

2019-09-10 Thread Shawn Heisey
On 9/10/2019 7:15 AM, Doss wrote: 4 to 5 million documents. For an NTR index, we need a field to be updated very frequently and filter results based on it. Will In-Place updates help us? Although you CAN search on docValues-only fields, the performance is terrible. So the answer I have

Re: In Place Updates: Can we filter on fields with only docValues="true"

2019-09-10 Thread Mikhail Khludnev
orted ones. These are updateable inplace. On Tue, Sep 10, 2019 at 4:15 PM Doss wrote: > Hi, > > 4 to 5 million documents. > > For an NTR index, we need a field to be updated very frequently and filter > results based on it. Will In-Place updates help us? > > docValues="true"

In Place Updates: Can we filter on fields with only docValues="true"

2019-09-10 Thread Doss
Hi, 4 to 5 million documents. For an NTR index, we need a field to be updated very frequently and filter results based on it. Will In-Place updates help us? Thanks, Doss.

Re: Are docValues useful for FilterQueries?

2019-07-08 Thread Erick Erickson
DocValues are irrelevant for scoring. Here’s the way I think of it. When querying (and thus scoring), you have a term X. I need to know > what docs does it appear in? > how many docs does it appear in? > how often does the term appear in the entire corpus? These are questions the inver

Are docValues useful for FilterQueries?

2019-07-08 Thread Ashwin Ramesh
Hi everybody, I can't find concrete evidence whether docValues are indeed useful for filter queries. One example of a field: This field will have a value between 0-1 The only usecase for this field is to filter on a range / subset of values. There will be no scoring / querying

Re: Enabling/disabling docValues

2019-06-11 Thread John Davis
; > has (1) > > > fleas (1) > > > > > > But for SortableText will be: > > > > > > my dog has fleas (1) > > > > > > Consider doc1 with “my dog has fleas” and doc2 with “my cat has fleas”. > > > doc1 was indexed before switching

Re: Enabling/disabling docValues

2019-06-11 Thread Gus Heck
t; has (1) > > fleas (1) > > > > But for SortableText will be: > > > > my dog has fleas (1) > > > > Consider doc1 with “my dog has fleas” and doc2 with “my cat has fleas”. > > doc1 was indexed before switching to SortableText and doc2 after. > &g

Re: Enabling/disabling docValues

2019-06-10 Thread John Davis
t; my cat has fleas (1) > > But you can’t get that output. There are three cases: > > 1> Lucene treats all documents as SortableText, faceting on the docValues > parts. No facets on doc1 > > my cat has fleas (1) > > 2> Lucene treats all documents as tokenized, faceting on e

Re: Enabling/disabling docValues

2019-06-10 Thread Erick Erickson
after. Presumably the output you want is: my dog has fleas (1) my cat has fleas (1) But you can’t get that output. There are three cases: 1> Lucene treats all documents as SortableText, faceting on the docValues parts. No facets on doc1 my cat has fleas (1) 2> Lucene treats al

Re: Enabling/disabling docValues

2019-06-09 Thread John Davis
at can Lucene do? Consider: > > Faceting on a text (not sortable) means that each individual token in the > index is uninverted on the Java heap and the facets are computed for each > individual term. > > Faceting on a SortableText field just has a single term per document, and

Re: Enabling/disabling docValues

2019-06-09 Thread Erick Erickson
. Faceting on a SortableText field just has a single term per document, and that in the docValues structures as opposed to the inverted index. Now you change the value and start indexing. At some point a segment containing no docValues is merged with a segment containing docValues for the field

Enabling/disabling docValues

2019-06-09 Thread John Davis
Hi there, We recently changed a field from TextField + no docValues to SortableTextField which has docValues enabled by default. Once I did this I do not see any facet values for the field. I know that once all the docs are re-indexed facets should work again, however can someone clarify

Re: Slow faceting performance on a docValues field

2019-05-10 Thread gulats
maybe quite late to the party but for the benefit of future readers, experimentation with facet.range.method might be helpful (for solr versions 6 and above) as it allows us to use docValues as well for range faceting -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr exception: java.lang.IllegalStateException: unexpected docvalues type NUMERIC for field 'weight' (expected one of [BINARY, NUMERIC, SORTED, SORTED_NUMERIC, SORTED_SET]). Re-index with correct

2019-04-10 Thread Erick Erickson
"Re-index with correct docvalues”. I.e. define weight to have docValues=true in your schema. WARNING: you have to totally get rid of your current data, I’d recommend starting with a new collection. > On Apr 10, 2019, at 12:21 AM, Alex Broitman > wrote: > > We got the So

Solr exception: java.lang.IllegalStateException: unexpected docvalues type NUMERIC for field 'weight' (expected one of [BINARY, NUMERIC, SORTED, SORTED_NUMERIC, SORTED_SET]). Re-index with correct doc

2019-04-10 Thread Alex Broitman
t; gid="21" group="34" pcid="6" ecid="174"}202.2+(Dashboard Dashboard*)edismaxDashboardname nls_NAME___en-ustrueproduct(sum(1,product(norm(acl_i),termfreq(acl_i,5164077))),if(exists(weight),weight,1))sysid1trueunexpected docvalues type NUMERIC for field '

Re: DocValues or stored fields to enable atomic updates

2019-04-05 Thread Emir Arnautović
not for sorting, faceting or highlighting - > should I use docValues=true or stored=true to enable atomic updates? Or even > both? I understand that either docValues or stored fields are needed for > atomic updates but which of the two would perform better / consume less > resources in thi

DocValues or stored fields to enable atomic updates

2019-04-05 Thread Andreas Hubold
Hi, I have a question on schema design: If a single-valued StrField is just used for filtering results by exact value (indexed=true) and its value isn't needed in the search result and not for sorting, faceting or highlighting - should I use docValues=true or stored=true to enable atomic

Re: Unexpected docvalues type SORTED_NUMERIC Exception when grouping by a PointField facet

2019-04-03 Thread Erick Erickson
Looks like: https://issues.apache.org/jira/browse/SOLR-11728 > On Apr 3, 2019, at 1:09 AM, JiaJun Zhu wrote: > > Hello, > > > I got an "Unexpected docvalues type SORTED_NUMERIC" exception when I perform > group facet on an IntPointField. Debugging

Unexpected docvalues type SORTED_NUMERIC Exception when grouping by a PointField facet

2019-04-03 Thread JiaJun Zhu
Hello, I got an "Unexpected docvalues type SORTED_NUMERIC" exception when I perform group facet on an IntPointField. Debugging into the source code, the cause is that internally the docvalue type for PointField is "NUMERIC" (single value) or "SORTED_NUME

Re: uniqueKey and docValues?

2018-11-23 Thread Mikhail Khludnev
It make sense to have docValues=true for _root_ for uniqueBlock() On Thu, Nov 22, 2018 at 6:44 PM Vincenzo D'Amore wrote: > Hi guys, this is an interesting thread. > > Looking at schema.xml I found having uniqueKey (type="string") configured > as docValues="true&quo

Re: uniqueKey and docValues?

2018-11-22 Thread Walter Underwood
/export needs all fields to be docValues. If you are going to export, including the id seems like a good idea. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 22, 2018, at 6:51 PM, Erick Erickson wrote: > > I doubt it mattes. The o

Re: uniqueKey and docValues?

2018-11-22 Thread Erick Erickson
I doubt it mattes. The only point for docValues is to speed up situations where you want to answer the question "for docX, what is the value of fieldY"? Unless you're doing something interesting with the_route_ field, it's only used to, well, route documents at index time. By "in

Re: uniqueKey and docValues?

2018-11-22 Thread Vincenzo D'Amore
Hi guys, this is an interesting thread. Looking at schema.xml I found having uniqueKey (type="string") configured as docValues="true" but, I also found that _root_ is configured docValues="false" Is there any drawbacks in having _root_ with docValues="false&qu

Re: uniqueKey and docValues?

2018-11-21 Thread Erick Erickson
On Wed, Nov 21, 2018 at 12:04 PM Walter Underwood wrote: > > Is it a good idea to store the uniqueKey as docValues? A great idea? A maybe > or maybe not idea? > > It looks like it will speed up export and streaming. Otherwise, I can’t find > anything the docs pro or con. > > w

uniqueKey and docValues?

2018-11-21 Thread Walter Underwood
Is it a good idea to store the uniqueKey as docValues? A great idea? A maybe or maybe not idea? It looks like it will speed up export and streaming. Otherwise, I can’t find anything the docs pro or con. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog)

Re: Able to search with indexed=false and docvalues=true

2018-11-21 Thread Toke Eskildsen
On Tue, 2018-11-20 at 21:17 -0700, Shawn Heisey wrote: > Maybe the error condition should be related to a new schema > property, something like allowQueryOnDocValues. This would default > to true with current schema versions and false in the next schema > version, which I think is 1.7. Then a

Re: Able to search with indexed=false and docvalues=true

2018-11-20 Thread Shawn Heisey
On 11/20/2018 8:18 PM, Rahul Goswami wrote: Erick and Toke, Thank you for the replies. I am surprised there already isn’t a JIRA for this. In my opinion, this should be an error condition on search or alternatively should simply be giving zero results. That would be a defined behavior as

Re: Able to search with indexed=false and docvalues=true

2018-11-20 Thread Rahul Goswami
are not particularly functional for any industry size load anyway. Thanks, Rahul On Tue, Nov 20, 2018 at 3:37 AM Toke Eskildsen wrote: > On Mon, 2018-11-19 at 22:19 -0500, Rahul Goswami wrote: > > I am using SolrCloud 7.2.1. My understanding is that setting > > docvalues=true would optimize fac

Re: Able to search with indexed=false and docvalues=true

2018-11-20 Thread Toke Eskildsen
On Mon, 2018-11-19 at 22:19 -0500, Rahul Goswami wrote: > I am using SolrCloud 7.2.1. My understanding is that setting > docvalues=true would optimize faceting, grouping and sorting; but for > a field to be searchable it needs to be indexed=true. Erick explained the search thing, so

Re: Able to search with indexed=false and docvalues=true

2018-11-19 Thread Erick Erickson
I've noticed this too, but I think it's more a side effect than something usable for the reasons you outlined. Searching a docValues field is akin to a "table scan", the uninverted structure is totally unsuited for searching. It works, but as you've found out it's unusably slow for

Able to search with indexed=false and docvalues=true

2018-11-19 Thread Rahul Goswami
I am using SolrCloud 7.2.1. My understanding is that setting docvalues=true would optimize faceting, grouping and sorting; but for a field to be searchable it needs to be indexed=true. However I was dumbfounded today when I executed a successful search on a field with below configuration: However

Re: Retrieve field from docValues

2018-11-06 Thread Erick Erickson
t;> "Does this still hold if whole index is loaded into memory?" > >> The decompression part yes, the disk seek part no. And it's also > >> sensitive to whether the documentCache already has the document. > >> > >> I'd also make uniqueKey ant the _v

Re: Retrieve field from docValues

2018-11-06 Thread Wei
t;> >> "Does this still hold if whole index is loaded into memory?" >> The decompression part yes, the disk seek part no. And it's also >> sensitive to whether the documentCache already has the document. >> >> I'd also make uniqueKey ant the _version_ fields docVal

Re: Retrieve field from docValues

2018-11-06 Thread Erick Erickson
ot; > > The decompression part yes, the disk seek part no. And it's also > > sensitive to whether the documentCache already has the document. > > > > I'd also make uniqueKey ant the _version_ fields docValues. > > > > Best, > > Erick > > On Tue, Nov

Re: Retrieve field from docValues

2018-11-06 Thread Wei
t JIRA, so only > in 7x. > > "Does this still hold if whole index is loaded into memory?" > The decompression part yes, the disk seek part no. And it's also > sensitive to whether the documentCache already has the document. > > I'd also make uniqueKey ant the _versio

Re: Retrieve field from docValues

2018-11-06 Thread Erick Erickson
also make uniqueKey ant the _version_ fields docValues. Best, Erick On Tue, Nov 6, 2018 at 10:44 AM Wei wrote: > > Thanks Yasufumi and Erick. > > ---. 2. "it depends". Solr will try to do the most efficient thing > possible. If _all_ the fields are docValues, it will

Re: Retrieve field from docValues

2018-11-06 Thread Wei
Thanks Yasufumi and Erick. ---. 2. "it depends". Solr will try to do the most efficient thing possible. If _all_ the fields are docValues, it will return the stored values from the docValues structure. I find this jira: https://issues.apache.org/jira/browse/SOLR-8344Does this

Re: Retrieve field from docValues

2018-11-06 Thread Erick Erickson
2. "it depends". Solr will try to do the most efficient thing possible. If _all_ the fields are docValues, it will return the stored values from the docValues structure. This prevents a disk seek and decompress cycle. However, if even one field is docValues=false Solr will by defa

Re: Retrieve field from docValues

2018-11-06 Thread Yasufumi Mizoguchi
Hi, > 1. For schema version 1.6, useDocValuesAsStored=true is default, so there > is no need to explicitly set it in schema.xml? Yes. > 2. With useDocValuesAsStored=true and the following definition, will Solr > retrieve id from docValues instead of stored field? No. AFAIK, if you

Retrieve field from docValues

2018-11-05 Thread Wei
Hi, I have a few questions about using the useDocValuesAsStored option to retrieve field from docValues: 1. For schema version 1.6, useDocValuesAsStored=true is default, so there is no need to explicitly set it in schema.xml? 2. With useDocValuesAsStored=true and the following definition

Re: Highlighting is not working with docValues only String field

2018-08-13 Thread Karthik Ramachandran
re isn't one already > > On Fri, Aug 10, 2018, 19:49 Karthik Ramachandran < > kramachand...@commvault.com> wrote: > > > We are using Solr 7.2.1, highlighting is not working with docValues only > > String field. > > > > Should I open a JIRA for this?

Re: Highlighting is not working with docValues only String field

2018-08-11 Thread Erick Erickson
olr 7.2.1, highlighting is not working with docValues only > String field. > > Should I open a JIRA for this? > > Schema: > > id > >required="true"/> >stored="true"/> >stored="false"/> > > > > Data: &

Highlighting is not working with docValues only String field

2018-08-10 Thread Karthik Ramachandran
We are using Solr 7.2.1, highlighting is not working with docValues only String field. Should I open a JIRA for this? Schema: id Data: [{"id":1,"name":"Testing line 1"},{"id":2,"name":"Testing line 2"},{"i

  1   2   3   4   5   6   >