RE: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

[Flowerday, Matthew J]

Hi There

 

Thanks for replying to my query. Yes I had seen the notes saying that on 
upgrading to a new major release the advice is to wipe and re-index. But I did 
see this on Major Changes in Solr 8 | Apache Solr Reference Guide 8.7 
 

 

Upgrade Prerequisites

If using SolrCloud, you must be on Solr 7.3.0 or higher. Solr’s 
LeaderInRecovery (LIR) functionality  

 changed significantly in Solr 7.3. While these changes were back-compatible 
for all subsequent 7.x releases, that compatibility has been removed in 8.0. In 
order to upgrade to Solr 8.x, all nodes of your cluster must be running Solr 
7.3 or higher. If an upgrade is attempted with nodes running versions earlier 
than 7.3, documents could be lost.

If you are not using Solr in SolrCloud mode (you use Standalone Mode instead), 
we expect you can upgrade to Solr 8 from any 7.x version without major issues.

So I was hoping to carry out just an upgrade as our customer’s data would take 
2-3 weeks to re-index from scratch.

 

You mentioned about committing after carrying out an update. Were you referring 
to the running of the IndexUpgrader? This is what carried out

 

*   With solr service stopped
*   Open a CMD window
*   cd c:\solr-8.7.0\server\solr-webapp\webapp\WEB-INF\lib
*   "C:\Program Files\Java\jre11_0_2\bin"\java -cp 
lucene-core-8.7.0.jar;lucene-backward-codecs-8.7.0.jar 
org.apache.lucene.index.IndexUpgrader -verbose 
C:\solr-8.7.0\server\solr\uleaf\data\index

And it generated output of the form

 

MS 0 [2021-01-09T10:16:54.405441300Z; main]: initDynamicDefaults spins=true 
maxThreadCount=1 maxMergeCount=6

IFD 0 [2021-01-09T10:16:54.450123300Z; main]: init: current segments file is 
"segments_1s"; 
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@7d68ef40

IFD 0 [2021-01-09T10:16:54.451123400Z; main]: init: load commit "segments_1s"

IFD 0 [2021-01-09T10:16:54.456124300Z; main]: init: seg=_1l set 
nextWriteFieldInfosGen=2 vs current=1

IFD 0 [2021-01-09T10:16:54.456727400Z; main]: init: seg=_1l set 
nextWriteDocValuesGen=2 vs current=1

IFD 0 [2021-01-09T10:16:54.462160900Z; main]: now checkpoint 
"_1i(7.7.1):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=7.7.1, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1609332625247}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
 _1j(7.7.1):C16:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=7.7.1, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1609335456070}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
 _1k(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1610016791080}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]

 

Ending with

 

IFD 0 [2021-01-09T10:16:55.448195200Z; main]: now checkpoint 
"_1q(8.7.0):C17:[diagnostics={os=Windows 10, java.version=11.0.2, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=merge, os.version=10.0, 
java.vendor=AdoptOpenJDK, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
mergeMaxNumSegments=1, mergeFactor=2, 
timestamp=1610187414535}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
 :id=i2dct7nwpvi35aycws4u8gwu 
_1k(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1610016791080}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
 :id=dop9d815p1kjrk1phi955lbg8 
_1l(8.7.0):C2/1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1610097123034}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
 :id=dop9d815p1kjrk1phi955lbgh 
_1m(8.7.0):C2:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1610097143494}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
 :id=dop9d815p1kjrk1phi955lbgg 
_1n(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, 
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, 
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, 
timestamp=1610097253797}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
 :id=dop9d815p

Re: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

[matthew sporleder]

I think the general advice is to do a full re-index on a major version
upgrade.  Also - did you ever commit?

On Sun, Jan 10, 2021 at 11:13 AM Flowerday, Matthew J <
matthew.flower...@gb.unisys.com> wrote:

> Hi There
>
>
>
> Thanks for contacting me.
>
>
>
> I carried out this analysis of the solr log from the updates I carried out
> at the time:
>
>
>
> Looking at the update requests sent to Solr. The first update of an
> existing record generated
>
>
>
> 2021-01-07 06:04:18.958 INFO  (qtp1458091526-17) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={wt=javabin&version=2}{add=[9901020319M01-X11
> (1688206792619720704)]} 0 59
>
> 2021-01-07 06:04:19.186 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.QuerySenderListener QuerySenderListener done.
>
> 2021-01-07 06:04:19.196 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 1 ms
>
> 2021-01-07 06:04:19.198 INFO  (qtp1458091526-23) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
> 0 228
>
>
>
> And the record was duplicated:
>
>
>
>
>
> The next update generated
>
>
>
> 2021-01-07 06:10:59.786 INFO  (qtp1458091526-17) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={wt=javabin&version=2}{add=[9901020319M01-X11
> (1688207212953993216)]} 0 20
>
> 2021-01-07 06:10:59.974 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.QuerySenderListener QuerySenderListener done.
>
> 2021-01-07 06:10:59.982 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms
>
> 2021-01-07 06:10:59.998 INFO  (qtp1458091526-26) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
> 0 208
>
>
>
> Which looks the same as the previous command – so no real difference here.
>
>
>
> And then the records looked like
>
>
>
>
>
> And this shows that the original (7.7.1) item is untouched and only the
> 8.6.3 item is updated on subsequent updates.
>
>
>
> A brand new record being sent to solr generate this dialog
>
>
>
> 2021-01-07 06:20:10.645 INFO  (qtp1458091526-25) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={wt=javabin&version=2}{add=[9901020319M01-X15 (1688207790576762880),
> 9901020319M01-DI21 (1688207790587248640)]} 0 15
>
> 2021-01-07 06:20:10.798 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.QuerySenderListener QuerySenderListener done.
>
> 2021-01-07 06:20:10.802 INFO
> (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf]
> o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms
>
> 2021-01-07 06:20:10.803 INFO  (qtp1458091526-23) [   x:uleaf]
> o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update
> params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
> 0 153
>
>
>
> And this has a similar update request line as the others – so no
> differences here. Solr just seems to leave the migrated records as is and
> just creates a duplicate when they are updated for some reason.
>
>
>
> I hope this is what you are after.
>
>
>
> Many Thanks
>
>
>
> Matthew
>
>
>
> *Matthew Flowerday* | Consultant | ULEAF
>
> Unisys | 01908 774830| matthew.flower...@unisys.com
>
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
> 8LX
>
>
>
> [image: unisys_logo] 
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
> [image: Grey_LI]   [image:
> Grey_TW]  [image: Grey_YT]
> [image: Grey_FB]
> [image: Grey_Vimeo]
> [image: Grey_UB] 
>
>
>
> *From:* xiefengchang 
> *Sent:* 10 January 2021 08:44
> *To:* solr-user@lucene.apache.org
> *Subject:* Re:Query over migrating a solr database from 7.7.1 to 8.7.0
>
>
>
> *EXTERNAL EMAIL - Be cautious of all links and attachments.*
>
> can you show the update request?
>
>
>
>
>
>
>
>
>
>
>
> At 2021-01-07 20:25:13, "Flowerday, Matthew J" <
> matthew.flower...@gb.unisys.com> wrote:
>
> Hi There
>
>
>
> I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped
> the database and re-indexed (as this would take too long to run on site).
>
>
>
> On my local windows machine I have a single solr s

[solr8.7] not relevant results for chinese query

[Bruno Mannina]

Hello,

 

I try to use chinese language with my index.

 

My definition is:



 







  

   

   

   

   

   

  



 

But, I get too much not relevant results.

 

i.e. : With the query (phone case):

tizh:(手機殼)

 

my query is translate to:

tizh:(手 OR 機 OR 殼)

 

But:

tizh:(手 AND 機 AND 殼)

returns 0 result.

 

And:

tizh:”手機殼”

returns also 0 result.

 

Is it possible to improve my fieldType ? or must I add something else ?

 

Thanks,

Bruno

 



-- 
L'absence de virus dans ce courrier electronique a ete verifiee par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

RE: [Solr8.7] Chinese ZH language ?

[Bruno Mannina]

Yes, it was that, I re-index and it works fine.

Thanks !

-Message d'origine-
De : Alexandre Rafalovitch [mailto:arafa...@gmail.com]
Envoyé : dimanche 10 janvier 2021 16:44
À : solr-user
Objet : Re: [Solr8.7] Chinese ZH language ?

>possible analysis error: cannot change field "tizh" from

You have content indexed against old incompatible definition. Deleted but not 
purged records count.

Delete your index data or change field name during testing.

Regards,
Alex
On Sun., Jan. 10, 2021, 9:19 a.m. Bruno Mannina,  wrote:

> Hello,
>
>
>
> I would like to index simplified chinese ZH language (i.e. 一种新型太阳能坪
> 床增温系统),
>
> I added in my solrconfig the lib:
>
>  dir="${solr.install.dir:../../..}/contrib/analysis-extras/lucene-libs/"
> regex="lucene-analyzers-smartcn-8\.7\.0\.jar" />
>
>
>
> First question: Is it enough ?
>
>
>
> But now I need your help to define the fieldtype “text_zh” in my
> schema.xml to use with:
>
> (PS: As other fields, I need highlight)
>
>
>
>  stored="true" termVectors="true" termPositions="true"
> termOffsets="true"/>
>
>
>
> And
>
>
>
> 
>
> 
>
>  positionIncrementGap="100">
>
>   
>
>
>
>
>
>
>   words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
>
>
>
>
>
>   
>
> 
>
>
>
> No error, when I reload my core.
>
>
>
> But I can’t index Chinese data, I get this error:
>
>
>
> POSTing file CN-0005.xml (application/xml) to [base]
>
> SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url:
> http:///solr/yyy/update
>
> SimplePostTool: WARNING: Response:  encoding="UTF-8"?>
>
> 
>
>
>
> 
>
>   400
>
>   1
>
> 
>
> 
>
>   
>
> org.apache.solr.common.SolrException
>
>  name="root-error-class">java.lang.IllegalArgumentException
>
>   
>
>   Exception writing document id CN112091782A to the
> index; possible analysis error: cannot change field "tizh" from index
> options=DOCS to inconsistent index
> options=DOCS_AND_FREQS_AND_POSITIONS
>
>   400
>
> 
>
> 
>
> SimplePostTool: WARNING: IOException while reading response:
> java.io.IOException: Server returned HTTP response code: 400 for URL:
> http:///solr/yyy/update
>
>
>
> Thanks a lot for your help,
>
> Bruno
>
>
>
>
>
> --
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>


--
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re: [Solr8.7] Chinese ZH language ?

[Alexandre Rafalovitch]

>possible analysis error: cannot change field "tizh" from

You have content indexed against old incompatible definition. Deleted but
not purged records count.

Delete your index data or change field name during testing.

Regards,
Alex
On Sun., Jan. 10, 2021, 9:19 a.m. Bruno Mannina,  wrote:

> Hello,
>
>
>
> I would like to index simplified chinese ZH language (i.e. 一种新型太阳能坪
> 床增温系统),
>
> I added in my solrconfig the lib:
>
>  dir="${solr.install.dir:../../..}/contrib/analysis-extras/lucene-libs/"
> regex="lucene-analyzers-smartcn-8\.7\.0\.jar" />
>
>
>
> First question: Is it enough ?
>
>
>
> But now I need your help to define the fieldtype “text_zh” in my
> schema.xml to use with:
>
> (PS: As other fields, I need highlight)
>
>
>
>  stored="true" termVectors="true" termPositions="true" termOffsets="true"/>
>
>
>
> And
>
>
>
> 
>
> 
>
>  positionIncrementGap="100">
>
>   
>
>
>
>
>
>
>   words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
>
>
>
>
>
>   
>
> 
>
>
>
> No error, when I reload my core.
>
>
>
> But I can’t index Chinese data, I get this error:
>
>
>
> POSTing file CN-0005.xml (application/xml) to [base]
>
> SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url:
> http:///solr/yyy/update
>
> SimplePostTool: WARNING: Response: 
>
> 
>
>
>
> 
>
>   400
>
>   1
>
> 
>
> 
>
>   
>
> org.apache.solr.common.SolrException
>
> java.lang.IllegalArgumentException
>
>   
>
>   Exception writing document id CN112091782A to the index;
> possible analysis error: cannot change field "tizh" from index options=DOCS
> to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS
>
>   400
>
> 
>
> 
>
> SimplePostTool: WARNING: IOException while reading response:
> java.io.IOException: Server returned HTTP response code: 400 for URL:
> http:///solr/yyy/update
>
>
>
> Thanks a lot for your help,
>
> Bruno
>
>
>
>
>
> --
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>

[Solr8.7] Chinese ZH language ?

[Bruno Mannina]

Hello,



I would like to index simplified chinese ZH language (i.e. 一种新型太阳能坪
床增温系统),

I added in my solrconfig the lib:





First question: Is it enough ?



But now I need your help to define the fieldtype “text_zh” in my
schema.xml to use with:

(PS: As other fields, I need highlight)







And









  

   

   

   

   

   

  





No error, when I reload my core.



But I can’t index Chinese data, I get this error:



POSTing file CN-0005.xml (application/xml) to [base]

SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url:
http:///solr/yyy/update

SimplePostTool: WARNING: Response: 







  400

  1





  

org.apache.solr.common.SolrException

java.lang.IllegalArgumentException

  

  Exception writing document id CN112091782A to the index;
possible analysis error: cannot change field "tizh" from index options=DOCS
to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS

  400





SimplePostTool: WARNING: IOException while reading response:
java.io.IOException: Server returned HTTP response code: 400 for URL:
http:///solr/yyy/update



Thanks a lot for your help,

Bruno





--
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

RE: [Solr8.7] Indexing only some language ?

[Bruno Mannina]

PErfect ! Thanks !

-Message d'origine-
De : xiefengchang [mailto:fengchang_fi...@163.com]
Envoyé : dimanche 10 janvier 2021 04:50
À : solr-user@lucene.apache.org
Objet : Re:[Solr8.7] Indexing only some language ?

Take a look at the document here:
https://lucene.apache.org/solr/guide/8_7/dynamic-fields.html#dynamic-fields


here's the point: "a field that does not match any explicitly defined fields
can be matched with a dynamic field."


so I guess the priority is quite clear~

















At 2021-01-10 03:38:01, "Bruno Mannina"  wrote:
>Hello,
>
>
>
>I would like to define in my schema.xml some text_xx fields.
>
>I have patent titles in several languages.
>
>Only 6 of them (EN, IT, FR, PT, ES, DE) interest me.
>
>
>
>I know how to define these 6 fields, I use text_en, text_it etc.
>
>
>
>i.e. for English language:
>
>stored="true" termVectors="true" termPositions="true"
>termOffsets="true"/>
>
>
>
>But I have more than 6 languages like: AR, CN, JP, KR etc.
>
>I can't analyze all source files to detect all languages and define
>them in my schema.
>
>
>
>I would like to use a dynamic field to index other languages.
>
>indexed="true" stored="true" omitTermFreqAndPositions="true"
>omitNorms="true"/>
>
>
>
>Is it ok to do that?
>
>Is TIEN field will be indexed twice internally or as tien is already
>defined
>ti* will not process tien?
>
>
>
>Thanks for your kind reply,
>
>
>
>Sincerely
>
>Bruno
>
>
>
>
>
>
>
>
>
>--
>L'absence de virus dans ce courrier électronique a été vérifiée par le
logiciel antivirus Avast.
>https://www.avast.com/antivirus


--
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re:Query over migrating a solr database from 7.7.1 to 8.7.0

[xiefengchang]

can you show the update request?  
















At 2021-01-07 20:25:13, "Flowerday, Matthew J" 
 wrote:

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the 
database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

Installed windows solr 8.7.0 on my machine in a different folder
Copied the core related folder (holding conf, data, lib, core.properties) from 
7.7.1 to the new 8.7.0 folder
Brought up the solr
Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created 
under 7.7.1. Instead of marking the old record as deleted it effectively 
created a new copy of the record with the change in and left the old image as 
still visible. When I updated the record again it then correctly updated the 
new 8.7.0 version without leaving the old image behind. If I created a new 
record and then updated it the solr record would be updated correctly. The 
issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created 
version and the second record is the 8.7.0 version after carrying out an 
update):

 

{

  "responseHeader":{

"status":0,

"QTime":4,

"params":{

  "q":"id:9901020319M01-N26",

  "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

  {

"id":"9901020319M01-N26",

"groupId":"9901020319M01",

"urn":"N26",

"specification":"nominal",

"owningGroupId":"9901020319M01",

"description":"N26, Yates, Mike, Alan, Richard, MALE",

"group_t":"9901020319M01",

"nominalUrn_t":"N26",

"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

"dateTimeCreated_dt":"2020-12-30T12:00:53Z",

"title_t":"Captain",

"surname_t":"Yates",

"qualifier_t":"Voyager",

"forename1_t":"Mike",

"forename2_t":"Alan",

"forename3_t":"Richard",

"sex_t":"MALE",

"orderedType_t":"Nominal",

"_version_":1687507566832123904},

  {

"id":"9901020319M01-N26",

"groupId":"9901020319M01",

"urn":"N26",

"specification":"nominal",

"owningGroupId":"9901020319M01",

"description":"N26, Yates, Mike, Alan, Richard, MALE",

"group_t":"9901020319M01",

"nominalUrn_t":"N26",

"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

"dateTimeCreated_dt":"2020-12-30T12:00:53Z",

"title_t":"Captain",

"surname_t":"Yates",

"qualifier_t":"Voyager enterprise defiant yorktown xx yy",

"forename1_t":"Mike",

"forename2_t":"Alan",

"forename3_t":"Richard",

"sex_t":"MALE",

"orderedType_t":"Nominal",

"_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

  

 

id

 

I was wondering if this behaviour is expected and if there is a way to make 
sure that records created under a previous version are updated correctly (so 
that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded 
it might be causing this issue (I tried hunting through the solr documentation 
online but struggled to find window upgrade notes and the above steps I worked 
out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| matthew.flower...@unisys.com

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is for use only by the intended recipient. If you received this in 
error, please contact the sender and delete the e-mail and its attachments from 
all devices.

   

 

Re:Interpreting Solr indexing times

[xiefengchang]

it's hard to answer your question without your solrconfig.xml, 
managed-schema(or schema.xml), and good to have some log snippet as well~

















At 2021-01-07 21:28:00, "ufuk yılmaz"  wrote:
>Hello all,
>
>I have been looking at our SolrCloud indexing performance statistics and 
>trying to make sense of the numbers. We are using a custom Flume sink and 
>sending updates to Solr (8.4) using SolrJ.
>
>I know these stuff depend on a lot of things but can you tell me if these 
>statistics are horribly bad (which means something is going obviously wrong), 
>or something expectable from a Solr cluster under right circumstances?
>
>We are sending documents in batches of 1000.
>
>{
>  "UPDATE./update.distrib.requestTimes": {
>"count": 7579,
>"meanRate": 0.044953336300254124,
>"1minRate": 0.2855655259375961,
>"5minRate": 0.29214637836736357,
>"15minRate": 0.29510868125823914,
>"min_ms": 5.854106,
>"max_ms": 56854.784017,
>"mean_ms": 3100.877968690649,
>"median_ms": 1084.258683,
>"stddev_ms": 4643.097311691323,
>"p75_ms": 2407.196867,
>"p95_ms": 15509.748909,
>"p99_ms": 16206.134345,
>"p999_ms": 16206.134345
>  },
>  "UPDATE./update.local.totalTime": 0,
>  "UPDATE./update.requestTimes": {
>"count": 7579,
>"meanRate": 0.044953336230621366,
>"1minRate": 0.2855655259375961,
>"5minRate": 0.29214637836736357,
>"15minRate": 0.29510868125823914,
>"min_ms": 5.857796,
>"max_ms": 56854.792298,
>"mean_ms": 3100.885675292589,
>"median_ms": 1084.264825,
>"stddev_ms": 4643.097457508117,
>"p75_ms": 2407.201642,
>"p95_ms": 15509.755934,
>"p99_ms": 16206.141754,
>"p999_ms": 16206.141754
>  },
>  "UPDATE./update.requests": 7580,
>  "UPDATE./update.totalTime": 33520426747162,
>  "UPDATE.update.totalTime": 0,
>  "UPDATE.updateHandler.adds": 854,
>  "UPDATE.updateHandler.autoCommitMaxTime": "15000ms",
>  "UPDATE.updateHandler.autoCommits": 2428,
>  "UPDATE.updateHandler.softAutoCommitMaxTime":"1ms",
>  "UPDATE.updateHandler.softAutoCommits":3380,
>  "UPDATE.updateHandler.commits": {
>"count": 5777,
>"meanRate": 0.034265134931240636,
>"1minRate": 0.13653886429826526,
>"5minRate": 0.12997330621941325,
>"15minRate": 0.12634106125326003
>  },
>  "UPDATE.updateHandler.cumulativeAdds": {
>"count": 2578492,
>"meanRate": 15.293816240408821,
>"1minRate": 90.7054223213904,
>"5minRate": 99.48315440730897,
>"15minRate": 101.77967003607128
>  },
>}
>
>
>Sent from Mail for Windows 10
>