RE: Time-out errors while indexing (Solr 7.7.1)

[Kommu, Vinodh K.]

Anyone has any thoughts or suggestions on this issue?

Thanks & Regards,
Vinodh

From: Kommu, Vinodh K.
Sent: Thursday, July 2, 2020 4:46 PM
To: solr-user@lucene.apache.org
Subject: Time-out errors while indexing (Solr 7.7.1)

Hi,

We are performing QA performance testing on couple of collections which holds 2 
billion and 3.5 billion docs respectively. Indexing happens from a separate 
client using solrJ which uses 10 thread and batch size 1000. From last 2-3 
weeks we have been noticing either slow indexing or timeout errors while 
indexing. As part of troubleshooting, we did noticed that when peak disk IO 
utilization is reaching higher side, then indexing is happening slowly and when 
disk IO is constantly near 100%, timeout issues are observed.

Few questions here:


  1.  Our performance team noticed that read operations are pretty more than 
write operations like 100:1 ratio, is this expected during indexing or solr 
nodes are doing any other operations like syncing?
  2.  Zookeeper has a latency around (min/avg/max: 0/0/2205), can this latency 
create instabilities issues to ZK or Solr clusters? Or impact indexing or 
searching operations?
  3.  Our client timeout is set to 2mins, can they increase further more? Would 
that help or create any other problems?
  4.  When we created an empty collection and loaded same data file, it loaded 
fine without any issues so having more documents in a collection would create 
such problems?

Any suggestions or feedback would be really appreciated.

Solr version - 7.7.1

Time out error snippet:

ERROR 
(updateExecutor-3-thread-30055-processing-x:TestCollection_shard5_replica_n18 
https:localhost:1122//solr//TestCollection_shard6_replica_n22
 r:core_node21 n:localhost:1122_solr c:TestCollection s:shard5) 
[c:TestCollection s:shard5 r:core_node21 x:TestCollection_shard5_replica_n18] 
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient error
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_212]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
~[?:1.8.0_212]
at java.net.SocketInputStream.read(SocketInputStream.java:171) 
~[?:1.8.0_212]
at java.net.SocketInputStream.read(SocketInputStream.java:141) 
~[?:1.8.0_212]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
~[?:1.8.0_212]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
~[?:1.8.0_212]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) 
~[?:1.8.0_212]
at 
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) 
~[?:1.8.0_212]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
~[?:1.8.0_212]
at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120)
 ~[solr-core-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 
2019-02-23 02:39:07]
at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) 
~[httpclient-4.5.6.jar:4.5.6]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
 ~[httpclient-4.5.6.jar:4.5.6]
 

Re: Adding solr-core via maven fails

[Ali Akhtar]

I had to add an additional repository to get the failing dependency to
resolve:

resolvers += "Spring Plugins Repository" at "
https://repo.spring.io/plugins-release/";

> However, we do not officially support Maven builds,

Um, why? This is a java based project, and maven is the de-facto standard
for Java. What if someone wanted to make use of any of Solr's java
libraries in their own JVM based project? There's no (clean) way to do it
other than adding it as a maven dependency and importing the class into
their code.


On Thu, Jul 2, 2020 at 6:07 PM Mike Drob  wrote:

> Does it fail similarly on 8.5.0 and .1?
>
> On Thu, Jul 2, 2020 at 6:38 AM Erick Erickson 
> wrote:
>
> > There have been some issues with Maven, see:
> > https://issues.apache.org/jira/browse/LUCENE-9170
> >
> > However, we do not officially support Maven builds, they’re there as a
> > convenience, so there may still
> > be issues in future.
> >
> > > On Jul 2, 2020, at 1:27 AM, Ali Akhtar  wrote:
> > >
> > > If I try adding solr-core to an existing project, e.g (SBT):
> > >
> > > libraryDependencies += "org.apache.solr" % "solr-core" % "8.5.2"
> > >
> > > It fails due a 404 on the dependencies:
> > >
> > > Extracting structure failed
> > > stack trace is suppressed; run last update for the full output
> > > stack trace is suppressed; run last ssExtractDependencies for the full
> > > output
> > > (update) sbt.librarymanagement.ResolveException: Error downloading
> > > org.restlet.jee:org.restlet:2.4.0
> > > Not found
> > > Not found
> > > not found:
> > > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > > not found:
> > >
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > > Not found
> > > Not found
> > > not found:
> > >
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > > not found:
> > >
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> > > (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error
> > > downloading org.restlet.jee:org.restlet:2.4.0
> > > Not found
> > > Not found
> > > not found:
> > > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > > not found:
> > >
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > > Not found
> > > Not found
> > > not found:
> > >
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > > not found:
> > >
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> > >
> > >
> > >
> > > Any ideas? Do I need to add a specific repository to get it to compile?
> >
> >
>

Re: Solr 8.5.2 indexing issue

[gnandre]

It seems that the issue is not with reference_url field itself. There is
one copy field which has the reference_url field as source and another
field called url_path as destination.
This destination field url_path has the following field type definition.

  

  
  
 
  
  
  
  
 
  
  


  
  
  
 
  
  
  
  

  

If I remove  SynonymGraphFilterFactory and FlattenGraphFilterFactory in
above field type definition then it works otherwise it throws the
same error (IndexOutOfBoundsException) .

On Sun, Jun 28, 2020 at 9:06 AM Erick Erickson 
wrote:

> How are you sending this to Solr? I just tried 8.5, submitting that doc
> through the admin UI and it works fine.
> I defined “asset_id” with as the same type as your reference_url field.
>
> And does the log on the Solr node that tries to index this give any more
> info?
>
> Best,
> Erick
>
> > On Jun 27, 2020, at 10:45 PM, gnandre  wrote:
> >
> > {
> >"asset_id":"add-ons:576deefef7453a9189aa039b66500eb2",
> >
> >
> "reference_url":"modeling-a-high-speed-backplane-part-3-4-port-s-parameters-to-differential-tdr-and-tdt.html"}
>
>

Re: Solr Float/Double multivalues fields

[Erick Erickson]

This is true _unless_ you fetch from docValues. docValues are SORTED_SETs,
so the results will be both ordered and deduplicated if you return them
as part of the field list.

Don’t really think it needs to go into the ref guide, it’s just inherent in 
storing
any kind of value. You wouldn’t expect multiple text entries in a multiValued
field to be rearranged when returning the stored values either.

Best,
Erick

> On Jul 2, 2020, at 2:21 PM, Vincenzo D'Amore  wrote:
> 
> Thanks, and genuinely asking: is there written somewhere in the
> documentation too? If no, could anyone suggest to me which doc page should
> I try to update?
> 
> On Thu, Jul 2, 2020 at 8:08 PM Colvin Cowie 
> wrote:
> 
>> The order of values within a multivalued field should match the insertion
>> order. -- we certainly rely on that in our product.
>> 
>> Order is guaranteed to be maintained for values in a multi-valued field.
>>> 
>> 
>> https://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4028057.html
>> 
>> On Thu, 2 Jul 2020 at 18:52, Vincenzo D'Amore  wrote:
>> 
>>> Hi all,
>>> 
>>> simple question: Solr float/double multivalue fields preserve the order
>> of
>>> inserted values?
>>> 
>>> Best regards,
>>> Vincenzo
>>> 
>>> --
>>> Vincenzo D'Amore
>>> 
>> 
> 
> 
> -- 
> Vincenzo D'Amore

Re: Solr Float/Double multivalues fields

[Vincenzo D'Amore]

Thanks, and genuinely asking: is there written somewhere in the
documentation too? If no, could anyone suggest to me which doc page should
I try to update?

On Thu, Jul 2, 2020 at 8:08 PM Colvin Cowie 
wrote:

> The order of values within a multivalued field should match the insertion
> order. -- we certainly rely on that in our product.
>
> Order is guaranteed to be maintained for values in a multi-valued field.
> >
>
> https://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4028057.html
>
> On Thu, 2 Jul 2020 at 18:52, Vincenzo D'Amore  wrote:
>
> > Hi all,
> >
> > simple question: Solr float/double multivalue fields preserve the order
> of
> > inserted values?
> >
> > Best regards,
> > Vincenzo
> >
> > --
> > Vincenzo D'Amore
> >
>


-- 
Vincenzo D'Amore

Re: Solr Float/Double multivalues fields

[Colvin Cowie]

The order of values within a multivalued field should match the insertion
order. -- we certainly rely on that in our product.

Order is guaranteed to be maintained for values in a multi-valued field.
>
https://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4028057.html

On Thu, 2 Jul 2020 at 18:52, Vincenzo D'Amore  wrote:

> Hi all,
>
> simple question: Solr float/double multivalue fields preserve the order of
> inserted values?
>
> Best regards,
> Vincenzo
>
> --
> Vincenzo D'Amore
>

Solr Float/Double multivalues fields

[Vincenzo D'Amore]

Hi all,

simple question: Solr float/double multivalue fields preserve the order of
inserted values?

Best regards,
Vincenzo

-- 
Vincenzo D'Amore

Re: How to use two search string in a single solr query

[Walter Underwood]

First, remove the “mm” parameter from the request handler definition. That can 
be added back in and tweaked later, or just left out.

Second, you don’t need any query syntax to search for two words. This query 
should work fine:

  books bags

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jul 1, 2020, at 10:22 PM, Tushar Arora  wrote:
> 
> Hi,
> I have a scenario with following entry in the request handler(handler1) of
> solrconfig.xml.(defType=edismax is used)
> description category  "qf">title^4 demand^0.3
> 2<-1 4<-30%
> 
> When I searched 'bags' as a search string, solr returned 15000 results.
> Query Used :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=bags&qt=handler1&rows=10&wt=json
> 
> And when searched 'books' as a search string, solr returns say 3348 results.
> Query Used :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=books&qt=handler1&rows=10&wt=json
> 
> I want to use both 'bags' and 'books' as a search string in a single query.
> I used the following query:
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=%22bags%22+OR+%22books%22&qt=handler1&rows=10&wt=json
> But OR operator not working. It will only give 7 results.
> 
> 
> I even tried this :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=(bags)+OR+(books)&qt=handler1&rows=10&wt=json
> But it also gives 7 results.
> 
> But my concern is to include the result of both 'bags' OR 'books' in a
> single query.
> Is there any way to use two search strings in a single query?

Re: Questions about Solr Search

[Doug Turnbull]

I think it's better to think of Solr as a piece of infrastructure or
component for you to build these things, rather than a product that has a
lot of capabilities for some specific use case.

So you can find 'lego pieces' to build some of these things, but with Solr
you need to build these things yourself. You're trading off targeted
feature you'll find in a search product vs depth of configurability and
pluggability in open source search. With Solr you should expect a big
engineering investment and getting to know the internals to use it most
effectively.

On topics 2 & 3, you might be interested in AI Powered Search which has a
strong NLP component http://aipoweredsearch.com

-Doug

On Thu, Jul 2, 2020 at 10:26 AM Gautam K  wrote:

> Dear Team,
>
> Hope you all are doing well.
>
> Can you please help with the following question? We are using Solr search
> in our Organisation and now checking whether Solr provides search
> capabilities like Google Enterprise search(Google Knowledge Graph Search).
>
> 1, Does Solr Search provide Voice Search like Google?
> 2. Does Solar Search provide NLP Search(Natural Language Processing)?
> 3. Does Solr have all the capabilities which Google Knowledge Graph
> provides like below?
>
>
>- Getting a ranked list of the most notable entities that match
>certain criteria.
>- Predictively completing entities in a search box.
>- Annotating/organizing content using the Knowledge Graph entities.
>
>
> *Your help will be appreciated highly.*
>
> Many thanks
> Gautam Kanaujia
> India
>


-- 
*Doug Turnbull **| CTO* | OpenSource Connections
, LLC | 240.476.9983
Author: Relevant Search ; Contributor: *AI
Powered Search *
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

Re: solrj - get metrics from all nodes

[ChienHuaWang]

Thanks for Jan's response.

I tried to set this "nodes" parameter by ModifiableSolrParams. But the null
is return from GenericSolrRequest. 
Could anyone advise the best approach to setup this parameter for multiple
nodes? 


Thanks,
Chien



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Adding solr-core via maven fails

[Mike Drob]

Does it fail similarly on 8.5.0 and .1?

On Thu, Jul 2, 2020 at 6:38 AM Erick Erickson 
wrote:

> There have been some issues with Maven, see:
> https://issues.apache.org/jira/browse/LUCENE-9170
>
> However, we do not officially support Maven builds, they’re there as a
> convenience, so there may still
> be issues in future.
>
> > On Jul 2, 2020, at 1:27 AM, Ali Akhtar  wrote:
> >
> > If I try adding solr-core to an existing project, e.g (SBT):
> >
> > libraryDependencies += "org.apache.solr" % "solr-core" % "8.5.2"
> >
> > It fails due a 404 on the dependencies:
> >
> > Extracting structure failed
> > stack trace is suppressed; run last update for the full output
> > stack trace is suppressed; run last ssExtractDependencies for the full
> > output
> > (update) sbt.librarymanagement.ResolveException: Error downloading
> > org.restlet.jee:org.restlet:2.4.0
> > Not found
> > Not found
> > not found:
> > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > Not found
> > Not found
> > not found:
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> > (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error
> > downloading org.restlet.jee:org.restlet:2.4.0
> > Not found
> > Not found
> > not found:
> > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > Not found
> > Not found
> > not found:
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> >
> >
> >
> > Any ideas? Do I need to add a specific repository to get it to compile?
>
>

Re: How to use two search string in a single solr query

[Erick Erickson]

Please let s know what version of Solr you use, otherwise it’s very hard to know
whether you’re running into https://issues.apache.org/jira/browse/SOLR-8812
or similar.

But two things to try:
1> specify q.op
lr
2> specify mm=0%

Best,
Erick

> On Jul 2, 2020, at 1:22 AM, Tushar Arora  wrote:
> 
> Hi,
> I have a scenario with following entry in the request handler(handler1) of
> solrconfig.xml.(defType=edismax is used)
> description category  "qf">title^4 demand^0.3
> 2<-1 4<-30%
> 
> When I searched 'bags' as a search string, solr returned 15000 results.
> Query Used :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=bags&qt=handler1&rows=10&wt=json
> 
> And when searched 'books' as a search string, solr returns say 3348 results.
> Query Used :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=books&qt=handler1&rows=10&wt=json
> 
> I want to use both 'bags' and 'books' as a search string in a single query.
> I used the following query:
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=%22bags%22+OR+%22books%22&qt=handler1&rows=10&wt=json
> But OR operator not working. It will only give 7 results.
> 
> 
> I even tried this :
> http://localhost:8984/solr/core_name/select?fl=title&indent=on&q=(bags)+OR+(books)&qt=handler1&rows=10&wt=json
> But it also gives 7 results.
> 
> But my concern is to include the result of both 'bags' OR 'books' in a
> single query.
> Is there any way to use two search strings in a single query?

Re: Adding solr-core via maven fails

[Erick Erickson]

There have been some issues with Maven, see: 
https://issues.apache.org/jira/browse/LUCENE-9170

However, we do not officially support Maven builds, they’re there as a 
convenience, so there may still
be issues in future.

> On Jul 2, 2020, at 1:27 AM, Ali Akhtar  wrote:
> 
> If I try adding solr-core to an existing project, e.g (SBT):
> 
> libraryDependencies += "org.apache.solr" % "solr-core" % "8.5.2"
> 
> It fails due a 404 on the dependencies:
> 
> Extracting structure failed
> stack trace is suppressed; run last update for the full output
> stack trace is suppressed; run last ssExtractDependencies for the full
> output
> (update) sbt.librarymanagement.ResolveException: Error downloading
> org.restlet.jee:org.restlet:2.4.0
> Not found
> Not found
> not found:
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> not found:
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> Not found
> Not found
> not found:
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> not found:
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error
> downloading org.restlet.jee:org.restlet:2.4.0
> Not found
> Not found
> not found:
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> not found:
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> Not found
> Not found
> not found:
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> not found:
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> 
> 
> 
> Any ideas? Do I need to add a specific repository to get it to compile?

Time-out errors while indexing (Solr 7.7.1)

[Kommu, Vinodh K.]

Hi,

We are performing QA performance testing on couple of collections which holds 2 
billion and 3.5 billion docs respectively. Indexing happens from a separate 
client using solrJ which uses 10 thread and batch size 1000. From last 2-3 
weeks we have been noticing either slow indexing or timeout errors while 
indexing. As part of troubleshooting, we did noticed that when peak disk IO 
utilization is reaching higher side, then indexing is happening slowly and when 
disk IO is constantly near 100%, timeout issues are observed.

Few questions here:


  1.  Our performance team noticed that read operations are pretty more than 
write operations like 100:1 ratio, is this expected during indexing or solr 
nodes are doing any other operations like syncing?
  2.  Zookeeper has a latency around (min/avg/max: 0/0/2205), can this latency 
create instabilities issues to ZK or Solr clusters? Or impact indexing or 
searching operations?
  3.  Our client timeout is set to 2mins, can they increase further more? Would 
that help or create any other problems?
  4.  When we created an empty collection and loaded same data file, it loaded 
fine without any issues so having more documents in a collection would create 
such problems?

Any suggestions or feedback would be really appreciated.

Solr version - 7.7.1

Time out error snippet:

ERROR 
(updateExecutor-3-thread-30055-processing-x:TestCollection_shard5_replica_n18 
https:localhost:1122//solr//TestCollection_shard6_replica_n22 r:core_node21 
n:localhost:1122_solr c:TestCollection s:shard5) [c:TestCollection s:shard5 
r:core_node21 x:TestCollection_shard5_replica_n18] 
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient error
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_212]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
~[?:1.8.0_212]
at java.net.SocketInputStream.read(SocketInputStream.java:171) 
~[?:1.8.0_212]
at java.net.SocketInputStream.read(SocketInputStream.java:141) 
~[?:1.8.0_212]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
~[?:1.8.0_212]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
~[?:1.8.0_212]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) 
~[?:1.8.0_212]
at 
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) 
~[?:1.8.0_212]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
~[?:1.8.0_212]
at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
 ~[httpcore-4.4.10.jar:4.4.10]
at 
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120)
 ~[solr-core-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 
2019-02-23 02:39:07]
at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) 
~[httpclient-4.5.6.jar:4.5.6]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) 
~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
 ~[httpclient-4.5.6.jar:4.5.6]
at 
org.apache.solr.client.solr

Re: Suggestion or recommendation for NRT

[ramyogi]

Thanks a lot for your time to respond for my clarifications.

We are having two environment,
ENV A and ENV B ( Both same capacity of RAM ( r5.2xlarge  and same number of
shards and replicas type (NRT) for the collection)

ENV A -  it is having a collection which is optimized ( segment count 1 and
numdocs = maxdocs ) it is used only for Search request. No delta updates are
being triggerred.


ENV B - It is having same collection copied from "ENV A" and continues DELTA
updates in progress so it is used for Indexing and search request. Indexing
using KAFKA connect plugin that uses SOLRJ with
solr.commit.within=30 ( milli seconds )


We are comparing performance between those environments for search request
using automation test running with bunch of queries.

Regarding search warmup:



1








true

20

200




*:*
true






*:*
true




false

24





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr8.5 CPU usage zero always

[Mangla,Kirti]

Hi,

I have setup Solr 8.5 version and now facing problem with CPU utilization shown 
at Dashboard.
It always says 0%, even though there are processes running on the node, like 
"ab"  for load testing and data import.

I need help to fix this. Please help if someone has seen this issue.
[cid:image001.png@01D65082.5EB29220]

Thanks,
Kirti




If you are not the intended recipient or have received this message in error, 
please notify the sender and permanently delete this message and any 
attachments.