Re: docvalues set to true, and indexed is false and stored is set to false

[Emir Arnautović]

Hi Ganesh,
I cannot confirm for sure, but I would assume that it will not get reindexed, 
but just segments doc values file rewritten. It is best if you test this and 
see for yourself.

Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 14 Feb 2018, at 13:09, mganeshs  wrote:
> 
> Hi Emir,
> 
> Thanks for confirming that strField is not considered / available for in
> place updates. 
> 
> As per documentation, it says...
> 
> *An atomic update operation is performed using this approach only when the
> fields to be updated meet these three conditions:
> 
> are non-indexed (indexed="false"), non-stored (stored="false"), single
> valued (multiValued="false") numeric docValues (docValues="true") fields;
> 
> the _version_ field is also a non-indexed, non-stored single valued
> docValues field; and,
> 
> copy targets of updated fields, if any, are also non-indexed, non-stored
> single valued numeric docValues fields.*
> 
> Let's consider I have declared following three fields in the schema
> 
> id
> 
>  docValues="false"/>
>  docValues="false"/>
>  docValues="true"/>
> 
> With this I am trying to create couple of solr document ( id =1) with only
> Field1 and Field2 and it's also indexed. And I could search the documents
> based on Field1 and Field2
> 
> Now after a while, I am adding a new field called Field3 by passing the id
> field ( id=1) and Field3 ( Field3=100 ( which is docvalues field in our case
> ).
> 
> What will happen now ? Will the complete document gets re indexed or only
> Field3 get added under docValues ?
> 
> Pls confirm.
> 
> Regards,
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[mganeshs]

Hi Emir,

Thanks for confirming that strField is not considered / available for in
place updates. 

As per documentation, it says...

*An atomic update operation is performed using this approach only when the
fields to be updated meet these three conditions:

are non-indexed (indexed="false"), non-stored (stored="false"), single
valued (multiValued="false") numeric docValues (docValues="true") fields;

the _version_ field is also a non-indexed, non-stored single valued
docValues field; and,

copy targets of updated fields, if any, are also non-indexed, non-stored
single valued numeric docValues fields.*

Let's consider I have declared following three fields in the schema

id





With this I am trying to create couple of solr document ( id =1) with only
Field1 and Field2 and it's also indexed. And I could search the documents
based on Field1 and Field2

Now after a while, I am adding a new field called Field3 by passing the id
field ( id=1) and Field3 ( Field3=100 ( which is docvalues field in our case
).

What will happen now ? Will the complete document gets re indexed or only
Field3 get added under docValues ?

Pls confirm.

Regards,



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[Emir Arnautović]

Hi Ganesh,
Doc values are enabled for strField and UUID but in place updates are not.

It is not free = according to some discussions on mailing list (did not check 
the code) in place updates are not update of some value in doc values file but 
rewrite of doc values file for the segment that it is holding doc that is 
updated. In case of updating docs that are in larger segment, larger doc values 
file will be rewritten.

Regards,
Emir

--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 14 Feb 2018, at 04:38, mganeshs  wrote:
> 
> Hi,
> 
> Thanks for clearing.
> 
> But as per this  link
> 
>  
> (Enabling DocValues) it says that it supports strField and UUID field also. 
> 
> Again, what you mean by it's not free for large segments. Can you point me
> to some documentation on that ?
> 
> Regards,
> Ganesh
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[mganeshs]

Hi,

Thanks for clearing.

But as per this  link

 
(Enabling DocValues) it says that it supports strField and UUID field also. 

Again, what you mean by it's not free for large segments. Can you point me
to some documentation on that ?

Regards,
Ganesh



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[Emir Arnautović]

Hi,
It is clearer now, but you mentioned strings in your first mail and in place 
updates only work for numeric fields. If you meet all conditions, document will 
not be reindexed, but only doc values rewritten for the segment where in place 
update happened. Note that this is not free for large segments.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 13 Feb 2018, at 13:43, mganeshs <mgane...@live.in> wrote:
> 
> Hi,
> 
> I guess my point is not conceived correctly. 
> 
> Here I am talking about the field  "In Place Updates
> <https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates>
>  
> "
> 
> As per above link, it says that complete document will not be re-indexed
> during updates, if the field is set as docValues="true" and indexed and
> stored is set as false.
> 
> But I want to know whether complete document will re-index, when I delete a
> field of type "docvalue" is set as true, but indexed and stored is set as
> false. Also when I add new field of type "docvalue"is set as true, but
> indexed and stored is set as false. 
> 
> Hope my question is clear now. 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[mganeshs]

Hi,

I guess my point is not conceived correctly. 

Here I am talking about the field  "In Place Updates
<https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates>
 
"

As per above link, it says that complete document will not be re-indexed
during updates, if the field is set as docValues="true" and indexed and
stored is set as false.

But I want to know whether complete document will re-index, when I delete a
field of type "docvalue" is set as true, but indexed and stored is set as
false. Also when I add new field of type "docvalue"is set as true, but
indexed and stored is set as false. 

Hope my question is clear now. 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[Emir Arnautović]

Whenever you send doc to indexing, it is indexed completely and old document 
with the same id (if one exists) is just flagged as deleted and will be removed 
from index when segment that it is stored is merged. In case of large segments, 
it might be never.

The safest option is to do full reindexing on a fresh collection once you 
change schema.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 13 Feb 2018, at 11:34, mganeshs  wrote:
> 
> Hi,
> 
> Thanks for quick response. 
> 
> I forgot to mention that after adding it, I have re-indexed all the data
> with dynamic fields Field_one, Field_two etc. 
> 
> In that case, by adding new field ( docvalue field ) or removing existing
> docvalue field, Will the whole document will re-indexed again, or only this
> field alone will be deleted and added correspondingly.
> 
> Regards,
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[mganeshs]

Hi,

Thanks for quick response. 

I forgot to mention that after adding it, I have re-indexed all the data
with dynamic fields Field_one, Field_two etc. 

In that case, by adding new field ( docvalue field ) or removing existing
docvalue field, Will the whole document will re-indexed again, or only this
field alone will be deleted and added correspondingly.

Regards,



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: docvalues set to true, and indexed is false and stored is set to false

[Emir Arnautović]

Hi,
Changing schema will not do anything by itself. After changes are applied (core 
reloaded if not used API to update schema) it will use new schema to index new 
documents. What matters is what you had in index before schema updates. So if 
you had defined Field_one as string or you had it as number but never used that 
field, deleting it and letting dynamic field handle from that moment should be 
fine. Similar goes with adding new field - if you are adding new definition for 
Field_100 before you used it, it will be ok.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 13 Feb 2018, at 11:03, mganeshs  wrote:
> 
> Hi,
> 
> If I have set following in the schema
> 
>  stored="false" docValues="true"/>
> 
> What will be the impact of deleting a single field, "Fields_one" field or
> what's the impact of adding a new field "Fields_100" ?
> 
> Will the whole document will re-indexed again, or only this field alone will
> be deleted and added correspondingly.
> 
> Idea here is we are trying to avoid complete re-indexing of document ( as
> document would be very huge one and number of documents are also in huge,
> and we have a situation, where we may need to add one new dynamic field to
> all the documents or to remove a dynamic from all the documents ).
> 
> Early responses are really appreciated !
> 
> Regards,
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

docvalues set to true, and indexed is false and stored is set to false

[mganeshs]

Hi,

If I have set following in the schema



What will be the impact of deleting a single field, "Fields_one" field or
what's the impact of adding a new field "Fields_100" ?

Will the whole document will re-indexed again, or only this field alone will
be deleted and added correspondingly.

Idea here is we are trying to avoid complete re-indexing of document ( as
document would be very huge one and number of documents are also in huge,
and we have a situation, where we may need to add one new dynamic field to
all the documents or to remove a dynamic from all the documents ).

Early responses are really appreciated !

Regards,



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html