Re: SOLR indexing takes longer time

[Aroop Ganguly]

Adding on to what others have said, indexing speed in general is largely 
affected by the parallelism and isolation you can give to each node.
Is there a reason why you cannot have more than 1 shard?
If you have 5 node cluster, why not have 5 shards, maxshardspernode=1 replica=1 
is ok. You should see dramatic gains.
Solr’s power and speed in doing everything comes from using it as a distributed 
system. By sharing more you will be using the benefit of that distributed 
capability,

HTH

Regards
Aroop

> On Aug 17, 2020, at 11:22 AM, Abhijit Pawar  wrote:
> 
> Hello,
> 
> We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
> replicas and just single core.
> It takes almost 3.5 hours to index that data.
> I am using a data import handler to import data from the mongo database.
> 
> Is there something we can do to reduce the time taken to index?
> Will upgrade to newer version help?
> 
> Appreciate your help!
> 
> Regards,
> Abhijit

Re: SOLR indexing takes longer time

[Shawn Heisey]

On 8/17/2020 12:22 PM, Abhijit Pawar wrote:

We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
replicas and just single core.
It takes almost 3.5 hours to index that data.
I am using a data import handler to import data from the mongo database.

Is there something we can do to reduce the time taken to index?
Will upgrade to newer version help?


There's not enough information here to provide a diagnosis.

Are you running Solr in cloud mode (with zookeeper)?

3.5 hours for 20 documents sounds like slowness with the data 
source, not a problem with Solr, but it's too soon to rule anything out.


Would you be able to write a program that pulls data from your mongo 
database but doesn't send it to Solr?  Ideally it would be a Java 
program using the same JDBC driver you're using with DIH.


Thanks,
Shawn

Re: Solr ping taking 600 seconds

[Susheel Kumar]

yes, Alex. This is reproducible. Will check if we can run Wireshark.

Thank you.

On Mon, Aug 17, 2020 at 8:11 PM Alexandre Rafalovitch 
wrote:

> If this is reproducible, I would run Wireshark on the network and see what
> happens at packet level.
>
> Leaning towards firewall timing out and just starting to drop all packets.
>
> Regards,
>Alex
>
> On Mon., Aug. 17, 2020, 6:22 p.m. Susheel Kumar, 
> wrote:
>
> > Thanks for the all responses.
> >
> > Shawn - to your point both ping or select in between taking 600+ seconds
> to
> > return as you can see below 1st ping attempt was all good and 2nd took
> long
> > time.  Similarly for select couple of select all returned fine and then
> > suddenly taking long time. I'll try to run select with shards.info to
> see
> > if it is a problem with any particular shard but solr.log on many of the
> > shard has QTime>600s entries.
> >
> > Heap doesn't seems to be a problem but will take a look on all the
> shards.
> > I'll share top output as well.
> >
> > Thnx
> >
> >
> > Ping
> >
> > server65:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/admin/ping?distrib=true'
> > 
> > 
> > true > name="status">020 > name="q">{!lucene}*:*true > name="df">wordTokensfalse > name="rows">10all > name="status">OK
> > 
> > server65:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/admin/ping?distrib=true'
> > 
> > 
> > true > name="status">0600123 name="params"> > name="q">{!lucene}*:*true > name="df">wordTokensfalse > name="rows">10all > name="status">OK
> > 
> >
> > select
> >
> >
> > server67:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> > {
> >   "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":13,
> > "params":{
> >   "q":"*:*",
> >   "indent":"on",
> >   "rows":"0",
> >   "wt":"json"}},
> >   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
> >   }}
> > server67:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> > {
> >   "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":10,
> > "params":{
> >   "q":"*:*",
> >   "indent":"on",
> >   "rows":"0",
> >   "wt":"json"}},
> >   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
> >   }}
> > server67:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> > {
> >   "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":18,
> > "params":{
> >   "q":"*:*",
> >   "indent":"on",
> >   "rows":"0",
> >   "wt":"json"}},
> >   "response":{"numFound":63094900,"start":0,"maxScore":1.0,"docs":[]
> >   }}
> > server67:/home/kumar # curl --location --request GET '
> > http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> > {
> >   "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":600093,
> > "params":{
> >   "q":"*:*",
> >   "indent":"on",
> >   "rows":"0",
> >   "wt":"json"}},
> >   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
> >   }}
> >
> > On Sat, Aug 15, 2020 at 1:41 PM Dominique Bejean <
> > dominique.bej...@eolya.fr>
> > wrote:
> >
> > > Hi,
> > >
> > > How long to display the solr console ?
> > > What about CPU and iowait with top ?
> > >
> > > You should start by eliminate network issue between your solr nodes by
> > > testing it with netcat on solr port.
> > > http://deice.daug.net/netcat_speed.html
> > >
> > > Dominique
> > >
> > > Le ven. 14 août 2020 à 23:40, Susheel Kumar  a
> > > écrit :
> > >
> > > > Hello,
> > > >
> > > >
> > > >
> > > > One of our Solr 6.6.2 DR cluster (target CDCR) which even doesn't
> have
> > > any
> > > >
> > > > live search load seems to be taking 60 ms many times for the
> ping /
> > > >
> > > > health check calls. Anyone has seen this before/suggestion what could
> > be
> > > >
> > > > wrong. The collection has 8 shards/3 replicas and 64GB memory and
> index
> > > >
> > > > seems to fit in memory. Below solr log entries.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > solr.log.26:2020-08-13 14:03:20.827 INFO  (qtp1775120226-46486)
> [c:COLL
> > > >
> > > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> > > >
> > > > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> > > >
> > > > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> > > >
> > > > hits=62569458 status=0 QTime=600113
> > > >
> > > > solr.log.26:2020-08-13 14:03:20.827 WARN  (qtp1775120226-46486)
> [c:COLL
> > > >
> > > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> > > >
> > > > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> > > >
> > > > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> > > >
> > > > hits=62569458 status=0 QTime=600113
> > > >
> > > > 

Re: Solr ping taking 600 seconds

[Alexandre Rafalovitch]

If this is reproducible, I would run Wireshark on the network and see what
happens at packet level.

Leaning towards firewall timing out and just starting to drop all packets.

Regards,
   Alex

On Mon., Aug. 17, 2020, 6:22 p.m. Susheel Kumar, 
wrote:

> Thanks for the all responses.
>
> Shawn - to your point both ping or select in between taking 600+ seconds to
> return as you can see below 1st ping attempt was all good and 2nd took long
> time.  Similarly for select couple of select all returned fine and then
> suddenly taking long time. I'll try to run select with shards.info to see
> if it is a problem with any particular shard but solr.log on many of the
> shard has QTime>600s entries.
>
> Heap doesn't seems to be a problem but will take a look on all the shards.
> I'll share top output as well.
>
> Thnx
>
>
> Ping
>
> server65:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/admin/ping?distrib=true'
> 
> 
> true name="status">020 name="q">{!lucene}*:*true name="df">wordTokensfalse name="rows">10all name="status">OK
> 
> server65:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/admin/ping?distrib=true'
> 
> 
> true name="status">0600123 name="q">{!lucene}*:*true name="df">wordTokensfalse name="rows">10all name="status">OK
> 
>
> select
>
>
> server67:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":13,
> "params":{
>   "q":"*:*",
>   "indent":"on",
>   "rows":"0",
>   "wt":"json"}},
>   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
>   }}
> server67:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":10,
> "params":{
>   "q":"*:*",
>   "indent":"on",
>   "rows":"0",
>   "wt":"json"}},
>   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
>   }}
> server67:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":18,
> "params":{
>   "q":"*:*",
>   "indent":"on",
>   "rows":"0",
>   "wt":"json"}},
>   "response":{"numFound":63094900,"start":0,"maxScore":1.0,"docs":[]
>   }}
> server67:/home/kumar # curl --location --request GET '
> http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
> {
>   "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":600093,
> "params":{
>   "q":"*:*",
>   "indent":"on",
>   "rows":"0",
>   "wt":"json"}},
>   "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
>   }}
>
> On Sat, Aug 15, 2020 at 1:41 PM Dominique Bejean <
> dominique.bej...@eolya.fr>
> wrote:
>
> > Hi,
> >
> > How long to display the solr console ?
> > What about CPU and iowait with top ?
> >
> > You should start by eliminate network issue between your solr nodes by
> > testing it with netcat on solr port.
> > http://deice.daug.net/netcat_speed.html
> >
> > Dominique
> >
> > Le ven. 14 août 2020 à 23:40, Susheel Kumar  a
> > écrit :
> >
> > > Hello,
> > >
> > >
> > >
> > > One of our Solr 6.6.2 DR cluster (target CDCR) which even doesn't have
> > any
> > >
> > > live search load seems to be taking 60 ms many times for the ping /
> > >
> > > health check calls. Anyone has seen this before/suggestion what could
> be
> > >
> > > wrong. The collection has 8 shards/3 replicas and 64GB memory and index
> > >
> > > seems to fit in memory. Below solr log entries.
> > >
> > >
> > >
> > >
> > >
> > > solr.log.26:2020-08-13 14:03:20.827 INFO  (qtp1775120226-46486) [c:COLL
> > >
> > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> > >
> > > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> > >
> > > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> > >
> > > hits=62569458 status=0 QTime=600113
> > >
> > > solr.log.26:2020-08-13 14:03:20.827 WARN  (qtp1775120226-46486) [c:COLL
> > >
> > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> > >
> > > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> > >
> > > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> > >
> > > hits=62569458 status=0 QTime=600113
> > >
> > > solr.log.26:2020-08-13 14:03:20.827 INFO  (qtp1775120226-46486) [c:COLL
> > >
> > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> > >
> > > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> > >
> > > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> status=0
> > >
> > > QTime=600113
> > >
> > > solr.log.26:2020-08-13 14:03:20.827 WARN  (qtp1775120226-46486) [c:COLL
> > >
> > > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> > >
> > > [COLL_shard1_replica1]  

Re: Solr ping taking 600 seconds

[Susheel Kumar]

Thanks for the all responses.

Shawn - to your point both ping or select in between taking 600+ seconds to
return as you can see below 1st ping attempt was all good and 2nd took long
time.  Similarly for select couple of select all returned fine and then
suddenly taking long time. I'll try to run select with shards.info to see
if it is a problem with any particular shard but solr.log on many of the
shard has QTime>600s entries.

Heap doesn't seems to be a problem but will take a look on all the shards.
I'll share top output as well.

Thnx


Ping

server65:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/admin/ping?distrib=true'


true020{!lucene}*:*truewordTokensfalse10allOK

server65:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/admin/ping?distrib=true'


true0600123{!lucene}*:*truewordTokensfalse10allOK


select


server67:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":13,
"params":{
  "q":"*:*",
  "indent":"on",
  "rows":"0",
  "wt":"json"}},
  "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
  }}
server67:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":10,
"params":{
  "q":"*:*",
  "indent":"on",
  "rows":"0",
  "wt":"json"}},
  "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
  }}
server67:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":18,
"params":{
  "q":"*:*",
  "indent":"on",
  "rows":"0",
  "wt":"json"}},
  "response":{"numFound":63094900,"start":0,"maxScore":1.0,"docs":[]
  }}
server67:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/select?indent=on=*:*=json=0'
{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":600093,
"params":{
  "q":"*:*",
  "indent":"on",
  "rows":"0",
  "wt":"json"}},
  "response":{"numFound":62221186,"start":0,"maxScore":1.0,"docs":[]
  }}

On Sat, Aug 15, 2020 at 1:41 PM Dominique Bejean 
wrote:

> Hi,
>
> How long to display the solr console ?
> What about CPU and iowait with top ?
>
> You should start by eliminate network issue between your solr nodes by
> testing it with netcat on solr port.
> http://deice.daug.net/netcat_speed.html
>
> Dominique
>
> Le ven. 14 août 2020 à 23:40, Susheel Kumar  a
> écrit :
>
> > Hello,
> >
> >
> >
> > One of our Solr 6.6.2 DR cluster (target CDCR) which even doesn't have
> any
> >
> > live search load seems to be taking 60 ms many times for the ping /
> >
> > health check calls. Anyone has seen this before/suggestion what could be
> >
> > wrong. The collection has 8 shards/3 replicas and 64GB memory and index
> >
> > seems to fit in memory. Below solr log entries.
> >
> >
> >
> >
> >
> > solr.log.26:2020-08-13 14:03:20.827 INFO  (qtp1775120226-46486) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> >
> > hits=62569458 status=0 QTime=600113
> >
> > solr.log.26:2020-08-13 14:03:20.827 WARN  (qtp1775120226-46486) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> >
> > hits=62569458 status=0 QTime=600113
> >
> > solr.log.26:2020-08-13 14:03:20.827 INFO  (qtp1775120226-46486) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2} status=0
> >
> > QTime=600113
> >
> > solr.log.26:2020-08-13 14:03:20.827 WARN  (qtp1775120226-46486) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2} status=0
> >
> > QTime=600113
> >
> > solr.log.38:2020-08-08 15:01:45.640 INFO  (qtp1775120226-46254) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> >
> > hits=62221186 status=0 QTime=600092
> >
> > solr.log.38:2020-08-08 15:01:45.640 WARN  (qtp1775120226-46254) [c:COLL
> >
> > s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
> >
> > [COLL_shard1_replica1]  webapp=/solr path=/admin/ping
> >
> > params={distrib=true&_stateVer_=COLL:3032=javabin=2}
> >
> > 

Re: Manipulating client's query using a Query object

[Erick Erickson]

Ed:

Right, doing this in a custom query parser on the Solr end that subclasses 
edismax
is probably the way to go or similar.

Especially because String->parsed query->String, even without any changes in 
the parsing is _not_ guaranteed to give you the same string back. I’m not clear
on whether it gives back a string that would parse to the same query object, 
but…

Best,
Erick

> On Aug 17, 2020, at 4:51 PM, Edward Turner  wrote:
> 
> Hi Markus,
> 
> Many thanks, I see what you are saying. My question was:
> 
> Question: is it possible to get a Lucene Query representation of the
> client's query, which we can then navigate and manipulate -- before we then
> send the String representation of this Query to Solr for evaluation?
> 
> ... and from your answer, I think the last part of this question, "before
> we then send the String representation of this Query to Solr for
> evaluation" perhaps makes no sense when doing things in the proper Solr
> way. I had originally thought about doing this manipulation in, say, a REST
> application, before sending it to Solr, which I suppose isn't the way to do
> it.
> 
> Many thanks,
> 
> Edd
> 
> Edward Turner
> 
> 
> On Mon, 17 Aug 2020 at 21:23, Markus Jelsma 
> wrote:
> 
>> Hello Edward,
>> 
>> You asked for the 'Lucene Query representation of the client's query'
>> which is already inside Solr and needs no forwarding to anything. Just
>> return in parse() and you are good to go.
>> 
>> The Query object contains the analyzed form of your query string.
>> ExtendedDismax has some variable (i think it was qstr) that contains the
>> original input string. In there you have access to that too.
>> 
>> Regards,
>> Markus
>> 
>> 
>> -Original message-
>>> From:Edward Turner 
>>> Sent: Monday 17th August 2020 21:25
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Manipulating client's query using a Query object
>>> 
>>> Hi Markus,
>>> 
>>> That's really great info. Thank you.
>>> 
>>> Supposing we've now modified the Query object, do you know how we would
>> get
>>> the corresponding query String, which we could then forward to our
>>> Solrcloud via SolrClient?
>>> 
>>> (Or should we be using this extended ExtendedDisMaxQParser class server
>>> side in Solr?)
>>> 
>>> Kind regards,
>>> 
>>> Edd
>>> 
>>> 
>>> Edward Turner
>>> 
>>> 
>>> On Mon, 17 Aug 2020 at 15:06, Markus Jelsma 
>>> wrote:
>>> 
 Hello Edward,
 
 Yes you can by extending ExtendedDismaxQParser [1] and override its
 parse() method. You get the main Query object through super.parse().
 
 If you need even more fine grained control on how Query objects are
 created you can extend ExtendedSolrQueryParser's [2] (inner class)
 newFieldQuery() method.
 
 Regards,
 Markus
 
 [1]
 
>> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.html
 [2]
 
>> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.ExtendedSolrQueryParser.html
 
 -Original message-
> From:Edward Turner 
> Sent: Monday 17th August 2020 15:53
> To: solr-user@lucene.apache.org
> Subject: Manipulating client's query using a Query object
> 
> Hi all,
> 
> Thanks for all your help recently. We're now using the edismax query
 parser
> and are happy with its behaviour. We have another question which
>> maybe
> someone can help with.
> 
> We have one use case where we optimise our query before sending it to
 Solr,
> and we do this by manipulating the client's input query string.
>> However,
> we're slightly uncomfortable using String manipulation to do this as
> there's always the possibility we parse their string wrongly. (We
>> have a
> large test suite to check if we're doing the right thing, but even
>> then,
 we
> String manipulation doesn't feel right ...).
> 
> Question: is it possible to get a Lucene Query representation of the
> client's query, which we can then navigate and manipulate -- before
>> we
 then
> send the String representation of this Query to Solr for evaluation?
> 
> Kind regards and thank you for your help in advance,
> 
> Edd
> 
 
>>> 
>> 

Re: Manipulating client's query using a Query object

[Edward Turner]

Hi Markus,

Many thanks, I see what you are saying. My question was:

Question: is it possible to get a Lucene Query representation of the
client's query, which we can then navigate and manipulate -- before we then
send the String representation of this Query to Solr for evaluation?

... and from your answer, I think the last part of this question, "before
we then send the String representation of this Query to Solr for
evaluation" perhaps makes no sense when doing things in the proper Solr
way. I had originally thought about doing this manipulation in, say, a REST
application, before sending it to Solr, which I suppose isn't the way to do
it.

Many thanks,

Edd

Edward Turner


On Mon, 17 Aug 2020 at 21:23, Markus Jelsma 
wrote:

> Hello Edward,
>
> You asked for the 'Lucene Query representation of the client's query'
> which is already inside Solr and needs no forwarding to anything. Just
> return in parse() and you are good to go.
>
> The Query object contains the analyzed form of your query string.
> ExtendedDismax has some variable (i think it was qstr) that contains the
> original input string. In there you have access to that too.
>
> Regards,
> Markus
>
>
> -Original message-
> > From:Edward Turner 
> > Sent: Monday 17th August 2020 21:25
> > To: solr-user@lucene.apache.org
> > Subject: Re: Manipulating client's query using a Query object
> >
> > Hi Markus,
> >
> > That's really great info. Thank you.
> >
> > Supposing we've now modified the Query object, do you know how we would
> get
> > the corresponding query String, which we could then forward to our
> > Solrcloud via SolrClient?
> >
> > (Or should we be using this extended ExtendedDisMaxQParser class server
> > side in Solr?)
> >
> > Kind regards,
> >
> > Edd
> >
> > 
> > Edward Turner
> >
> >
> > On Mon, 17 Aug 2020 at 15:06, Markus Jelsma 
> > wrote:
> >
> > > Hello Edward,
> > >
> > > Yes you can by extending ExtendedDismaxQParser [1] and override its
> > > parse() method. You get the main Query object through super.parse().
> > >
> > > If you need even more fine grained control on how Query objects are
> > > created you can extend ExtendedSolrQueryParser's [2] (inner class)
> > > newFieldQuery() method.
> > >
> > > Regards,
> > > Markus
> > >
> > > [1]
> > >
> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.html
> > > [2]
> > >
> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.ExtendedSolrQueryParser.html
> > >
> > > -Original message-
> > > > From:Edward Turner 
> > > > Sent: Monday 17th August 2020 15:53
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Manipulating client's query using a Query object
> > > >
> > > > Hi all,
> > > >
> > > > Thanks for all your help recently. We're now using the edismax query
> > > parser
> > > > and are happy with its behaviour. We have another question which
> maybe
> > > > someone can help with.
> > > >
> > > > We have one use case where we optimise our query before sending it to
> > > Solr,
> > > > and we do this by manipulating the client's input query string.
> However,
> > > > we're slightly uncomfortable using String manipulation to do this as
> > > > there's always the possibility we parse their string wrongly. (We
> have a
> > > > large test suite to check if we're doing the right thing, but even
> then,
> > > we
> > > > String manipulation doesn't feel right ...).
> > > >
> > > > Question: is it possible to get a Lucene Query representation of the
> > > > client's query, which we can then navigate and manipulate -- before
> we
> > > then
> > > > send the String representation of this Query to Solr for evaluation?
> > > >
> > > > Kind regards and thank you for your help in advance,
> > > >
> > > > Edd
> > > >
> > >
> >
>

RE: Manipulating client's query using a Query object

[Markus Jelsma]

Hello Edward,

You asked for the 'Lucene Query representation of the client's query' which is 
already inside Solr and needs no forwarding to anything. Just return in parse() 
and you are good to go.

The Query object contains the analyzed form of your query string. 
ExtendedDismax has some variable (i think it was qstr) that contains the 
original input string. In there you have access to that too.

Regards,
Markus


-Original message-
> From:Edward Turner 
> Sent: Monday 17th August 2020 21:25
> To: solr-user@lucene.apache.org
> Subject: Re: Manipulating client's query using a Query object
> 
> Hi Markus,
> 
> That's really great info. Thank you.
> 
> Supposing we've now modified the Query object, do you know how we would get
> the corresponding query String, which we could then forward to our
> Solrcloud via SolrClient?
> 
> (Or should we be using this extended ExtendedDisMaxQParser class server
> side in Solr?)
> 
> Kind regards,
> 
> Edd
> 
> 
> Edward Turner
> 
> 
> On Mon, 17 Aug 2020 at 15:06, Markus Jelsma 
> wrote:
> 
> > Hello Edward,
> >
> > Yes you can by extending ExtendedDismaxQParser [1] and override its
> > parse() method. You get the main Query object through super.parse().
> >
> > If you need even more fine grained control on how Query objects are
> > created you can extend ExtendedSolrQueryParser's [2] (inner class)
> > newFieldQuery() method.
> >
> > Regards,
> > Markus
> >
> > [1]
> > https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.html
> > [2]
> > https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.ExtendedSolrQueryParser.html
> >
> > -Original message-
> > > From:Edward Turner 
> > > Sent: Monday 17th August 2020 15:53
> > > To: solr-user@lucene.apache.org
> > > Subject: Manipulating client's query using a Query object
> > >
> > > Hi all,
> > >
> > > Thanks for all your help recently. We're now using the edismax query
> > parser
> > > and are happy with its behaviour. We have another question which maybe
> > > someone can help with.
> > >
> > > We have one use case where we optimise our query before sending it to
> > Solr,
> > > and we do this by manipulating the client's input query string. However,
> > > we're slightly uncomfortable using String manipulation to do this as
> > > there's always the possibility we parse their string wrongly. (We have a
> > > large test suite to check if we're doing the right thing, but even then,
> > we
> > > String manipulation doesn't feel right ...).
> > >
> > > Question: is it possible to get a Lucene Query representation of the
> > > client's query, which we can then navigate and manipulate -- before we
> > then
> > > send the String representation of this Query to Solr for evaluation?
> > >
> > > Kind regards and thank you for your help in advance,
> > >
> > > Edd
> > >
> >
> 

Re: Manipulating client's query using a Query object

[Edward Turner]

Hi Markus,

That's really great info. Thank you.

Supposing we've now modified the Query object, do you know how we would get
the corresponding query String, which we could then forward to our
Solrcloud via SolrClient?

(Or should we be using this extended ExtendedDisMaxQParser class server
side in Solr?)

Kind regards,

Edd


Edward Turner


On Mon, 17 Aug 2020 at 15:06, Markus Jelsma 
wrote:

> Hello Edward,
>
> Yes you can by extending ExtendedDismaxQParser [1] and override its
> parse() method. You get the main Query object through super.parse().
>
> If you need even more fine grained control on how Query objects are
> created you can extend ExtendedSolrQueryParser's [2] (inner class)
> newFieldQuery() method.
>
> Regards,
> Markus
>
> [1]
> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.html
> [2]
> https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.ExtendedSolrQueryParser.html
>
> -Original message-
> > From:Edward Turner 
> > Sent: Monday 17th August 2020 15:53
> > To: solr-user@lucene.apache.org
> > Subject: Manipulating client's query using a Query object
> >
> > Hi all,
> >
> > Thanks for all your help recently. We're now using the edismax query
> parser
> > and are happy with its behaviour. We have another question which maybe
> > someone can help with.
> >
> > We have one use case where we optimise our query before sending it to
> Solr,
> > and we do this by manipulating the client's input query string. However,
> > we're slightly uncomfortable using String manipulation to do this as
> > there's always the possibility we parse their string wrongly. (We have a
> > large test suite to check if we're doing the right thing, but even then,
> we
> > String manipulation doesn't feel right ...).
> >
> > Question: is it possible to get a Lucene Query representation of the
> > client's query, which we can then navigate and manipulate -- before we
> then
> > send the String representation of this Query to Solr for evaluation?
> >
> > Kind regards and thank you for your help in advance,
> >
> > Edd
> >
>

Re: SOLR indexing takes longer time

[Walter Underwood]

I’m seeing multiple red flags for performance here. The top ones are “DIH”,
“MongoDB”, and “SQL on MongoDB”. MongoDB is not a relational database.

Our multi-threaded extractor using the Mongo API was still three times slower
than the same approach on MySQL.

Check the CPU usage on the Solr hosts while you are indexing. If it is under 
50%, the bottleneck is MongoDB and single-threaded indexing.

For another check, run that same query in a regular database client and time it.
The Solr indexing will never be faster than that.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Aug 17, 2020, at 11:58 AM, Abhijit Pawar  wrote:
> 
> Sure Divye,
> 
> *Here's the config.*
> 
> *conf/solr-config.xml:*
> 
> 
> 
>  class="org.apache.solr.handler.dataimport.DataImportHandler">
> 
>  name="config">/home/ec2-user/solr/solr-5.4.1/server/solr/test_core/conf/dataimport/data-source-config.xml
> 
> 
> 
> 
> 
> *schema.xml:*
> has of all the field definitions
> 
> *conf/dataimport/data-source-config.xml*
> 
> 
>  driver="com.mongodb.jdbc.MongoDriver" url="mongodb://< ADDRESS>>:27017/<>"/>
> 
>  dataSource="mongod"
> transformer="<>,TemplateTransformer"
> onError="continue"
> pk="uuid"
> query="SELECT field1,field2,field3,.. FROM products"
> deltaImportQuery="SELECT field1,field2,field3,.. FROM products WHERE
> orgidStr = '${dataimporter.request.orgid}' AND idStr =
> '${dataimporter.delta.idStr}'"
> deltaQuery="SELECT idStr FROM products WHERE orgidStr =
> '${dataimporter.request.orgid}' AND updatedAt >
> '${dataimporter.last_index_time}'"
>> 
> 
> 
> 
> 
> .
> .
> . 4-5 more nested entities...
> 
> On Mon, Aug 17, 2020 at 1:32 PM Divye Handa 
> wrote:
> 
>> Can you share the dih configuration you are using for same?
>> 
>> On Mon, 17 Aug, 2020, 23:52 Abhijit Pawar,  wrote:
>> 
>>> Hello,
>>> 
>>> We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
>>> replicas and just single core.
>>> It takes almost 3.5 hours to index that data.
>>> I am using a data import handler to import data from the mongo database.
>>> 
>>> Is there something we can do to reduce the time taken to index?
>>> Will upgrade to newer version help?
>>> 
>>> Appreciate your help!
>>> 
>>> Regards,
>>> Abhijit
>>> 
>> 

Re: SOLR indexing takes longer time

[Abhijit Pawar]

Sure Divye,

*Here's the config.*

*conf/solr-config.xml:*





/home/ec2-user/solr/solr-5.4.1/server/solr/test_core/conf/dataimport/data-source-config.xml





*schema.xml:*
has of all the field definitions

*conf/dataimport/data-source-config.xml*









.
.
. 4-5 more nested entities...

On Mon, Aug 17, 2020 at 1:32 PM Divye Handa 
wrote:

> Can you share the dih configuration you are using for same?
>
> On Mon, 17 Aug, 2020, 23:52 Abhijit Pawar,  wrote:
>
> > Hello,
> >
> > We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
> > replicas and just single core.
> > It takes almost 3.5 hours to index that data.
> > I am using a data import handler to import data from the mongo database.
> >
> > Is there something we can do to reduce the time taken to index?
> > Will upgrade to newer version help?
> >
> > Appreciate your help!
> >
> > Regards,
> > Abhijit
> >
>

Re: SOLR indexing takes longer time

[Jörn Franke]

The DIH is single threaded and deprecated. Your best bet is to have a 
script/program extracting data from MongoDB and write them to Solr in Batches 
using multiple threads. You will see a significant higher performance for your 
data.

> Am 17.08.2020 um 20:23 schrieb Abhijit Pawar :
> 
> Hello,
> 
> We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
> replicas and just single core.
> It takes almost 3.5 hours to index that data.
> I am using a data import handler to import data from the mongo database.
> 
> Is there something we can do to reduce the time taken to index?
> Will upgrade to newer version help?
> 
> Appreciate your help!
> 
> Regards,
> Abhijit

Looking for Solr contractor at Chegg

[Walter Underwood]

We plan to upgrade all of our custers to Solr 8.x and are looking for a 
contractor. The Solr Cloud clusters are on 6.6.2 and we have a master/slave 
cluster on 4.10.4 with a customized edismax query parser (eedismax?).

https://jobs.chegg.com/job/CHEGA0056OWKCDFWP/Senior-Software-Engineer-Search-6-month-contract
 


Glad to answer further questions by direct email.

Sorry if this is off-topic for the solr-user list. I checked the Apache page 
and job listings aren’t prohibited, though they are somewhat different than the 
main topic of helping Solr users.

wunder
Walter Underwood
Principal Software Engineer, Chegg
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

Re: Elevation with distributed search causes NPE

[Erick Erickson]

I "enabled patch review”… Thanks!

> On Aug 17, 2020, at 9:43 AM, smk  wrote:
> 
> H Erick,
> 
> I am a colleague of Marc and have attached a patch to his ticket
> https://issues.apache.org/jira/browse/SOLR-14662
> Can you set the ticket to "Patch Available"? For me there is no such option.
> 
> Thank you very much, Thomas
> 
> 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SOLR indexing takes longer time

[Divye Handa]

Can you share the dih configuration you are using for same?

On Mon, 17 Aug, 2020, 23:52 Abhijit Pawar,  wrote:

> Hello,
>
> We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
> replicas and just single core.
> It takes almost 3.5 hours to index that data.
> I am using a data import handler to import data from the mongo database.
>
> Is there something we can do to reduce the time taken to index?
> Will upgrade to newer version help?
>
> Appreciate your help!
>
> Regards,
> Abhijit
>

SOLR indexing takes longer time

[Abhijit Pawar]

Hello,

We are indexing some 200K plus documents in SOLR 5.4.1 with no shards /
replicas and just single core.
It takes almost 3.5 hours to index that data.
I am using a data import handler to import data from the mongo database.

Is there something we can do to reduce the time taken to index?
Will upgrade to newer version help?

Appreciate your help!

Regards,
Abhijit

Re: Slow query response from SOLR 5.4.1

[Abhijit Pawar]

Jason,

Not yet.This issue was on the back burner for a few daysHowever we
still need to figure out what could be a potential solution to it.
The setup is basic one - with one node / no shards or replicas
2 cores

When I run the query adding debug=timing to raw query parameters it just
hangs in there without a response back.
Today the response is much better though it takes 15-20 seconds however we
would like to speed it up more better.

Any suggestion would be appreciated.Thanks!



On Tue, Aug 11, 2020 at 11:01 AM Jason Gerlowski 
wrote:

> Hey Abhijit,
>
> The information you provided isn't really enough for anyone else on
> the mailing list to debug the problem.  If you'd like help, please
> provide some more information.
>
> Good places to start would be: what is the query, what does Solr tell
> you when you add a "debug=timing" parameter to your request, what does
> your Solr setup look like (num nodes, shards, replicas, other
> collections/cores, QPS).  It's hard to say upfront what piece of info
> will be the one that helps you get an answer to your question -
> performance problems have a lot of varied causes.  But providing
> _some_ of these things or other related details might help you get the
> answer you're looking for.
>
> Alternately, if you've figured out the issue already post the answer
> on this thread - help anyone with a similar issue in the future.
> Jason
>
> On Tue, Aug 4, 2020 at 4:11 PM Abhijit Pawar 
> wrote:
> >
> > Hello,
> >
> > I am seeing a performance issue in querying in one of the SOLR servers -
> > instance version 5.4.1.
> > Total number of documents indexed are 20K plus.
> > Data returned for this particular query is just as less as 22 documents
> > however it takes almost 2 minutes to get the results back.
> >
> > Is there a way to improve on performance of query - in general the query
> > response time is slow..
> >
> > I have most of the fields which are stored and indexed both.I can take
> off
> > some fields which are just needed to be indexed however those are not
> many
> > fields.
> >
> > Can I do something solrconfig.xml in terms of cache or something else?
> >
> > Any suggestions?
> >
> > Thanks!!
>

Re: IOException occured when talking to server

[Dominique Bejean]

These links are not providing solutions but may be provide some ideas for
the investigation.
I suggest to try the  -Djavax.net.debug=all JVM parameter for your client
application.

Good luke.

Dominique

Le lun. 17 août 2020 à 19:11, Odysci  a écrit :

> Dominique,
> thanks, but I'm not sure the links you sent point to an actual solution.
> The Nginx logs, sometimes give a 499 return code which is:
> (499 Client Closed Request
> Used when the client has closed the request before the server could send a
> response.
>
> but the timestamps of these log msgs do not coincide with the IOException,
> so I'm not sure they are related.
> Reinaldo
>
> On Mon, Aug 17, 2020 at 12:59 PM Dominique Bejean <
> dominique.bej...@eolya.fr> wrote:
>
>> Hi,
>>
>> It looks like this issues
>> https://github.com/eclipse/jetty.project/issues/4883
>> https://github.com/eclipse/jetty.project/issues/2571
>>
>> The Nginx server closed the connection. Any info in nginx log ?
>>
>> Dominique
>>
>> Le lun. 17 août 2020 à 17:33, Odysci  a écrit :
>>
>>> Hi,
>>> thanks for the reply.
>>> We're using solr 8.3.1, ZK 3.5.6
>>> The stacktrace is below.
>>> The address on the first line "
>>> http://192.168.15.10:888/solr/mycollection; is the "server" address in
>>> my nginx configuration, which points to 2 upstream solr nodes. There were
>>> no other solr or ZK messages in the logs.
>>>
>>> StackTrace:
>>> (Msg = IOException occured when talking to server at:
>>> http://192.168.15.10:888/solr/mycollection)
>>> org.apache.solr.client.solrj.SolrServerException: IOException occured
>>> when talking to server at: http://192.168.15.10:888/solr/mycollection
>>> at
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
>>> at
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
>>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
>>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
>>> ... calls from our code
>>> ... calls from our code
>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at java.base/java.lang.Thread.run(Thread.java:834)
>>> Caused by: java.nio.channels.AsynchronousCloseException
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
>>> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
>>> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
>>> at
>>> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
>>> at
>>> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
>>> at
>>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
>>> at
>>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
>>> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
>>> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>> at
>>> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
>>> at
>>> 

Re: Elevation with distributed search causes NPE

[smk]

H Erick,

I am a colleague of Marc and have attached a patch to his ticket
https://issues.apache.org/jira/browse/SOLR-14662
Can you set the ticket to "Patch Available"? For me there is no such option.

Thank you very much, Thomas





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: IOException occured when talking to server

[Odysci]

Dominique,
thanks, but I'm not sure the links you sent point to an actual solution.
The Nginx logs, sometimes give a 499 return code which is:
(499 Client Closed Request
Used when the client has closed the request before the server could send a
response.

but the timestamps of these log msgs do not coincide with the IOException,
so I'm not sure they are related.
Reinaldo

On Mon, Aug 17, 2020 at 12:59 PM Dominique Bejean 
wrote:

> Hi,
>
> It looks like this issues
> https://github.com/eclipse/jetty.project/issues/4883
> https://github.com/eclipse/jetty.project/issues/2571
>
> The Nginx server closed the connection. Any info in nginx log ?
>
> Dominique
>
> Le lun. 17 août 2020 à 17:33, Odysci  a écrit :
>
>> Hi,
>> thanks for the reply.
>> We're using solr 8.3.1, ZK 3.5.6
>> The stacktrace is below.
>> The address on the first line "http://192.168.15.10:888/solr/mycollection;
>> is the "server" address in my nginx configuration, which points to 2
>> upstream solr nodes. There were no other solr or ZK messages in the logs.
>>
>> StackTrace:
>> (Msg = IOException occured when talking to server at:
>> http://192.168.15.10:888/solr/mycollection)
>> org.apache.solr.client.solrj.SolrServerException: IOException occured
>> when talking to server at: http://192.168.15.10:888/solr/mycollection
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
>> ... calls from our code
>> ... calls from our code
>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> Caused by: java.nio.channels.AsynchronousCloseException
>> at
>> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
>> at
>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
>> at
>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
>> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
>> at
>> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
>> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
>> at
>> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
>> at
>> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
>> at
>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
>> at
>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
>> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
>> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
>> ... 3 more
>>
>> -
>>
>> I did consider using the solrJ cloud or lb 

Re: IOException occured when talking to server

[Dominique Bejean]

I mean add this parameter on your client application JVM :)


Le lun. 17 août 2020 à 18:36, Dominique Bejean 
a écrit :

> If you want a more detailed debug information from your client
> application, you can add this parameter while starting Solr JVM.
> -Djavax.net.debug=all
>
> It is very verbose !
>
> Dominique
>
>
> Le lun. 17 août 2020 à 17:59, Dominique Bejean 
> a écrit :
>
>> Hi,
>>
>> It looks like this issues
>> https://github.com/eclipse/jetty.project/issues/4883
>> https://github.com/eclipse/jetty.project/issues/2571
>>
>> The Nginx server closed the connection. Any info in nginx log ?
>>
>> Dominique
>>
>> Le lun. 17 août 2020 à 17:33, Odysci  a écrit :
>>
>>> Hi,
>>> thanks for the reply.
>>> We're using solr 8.3.1, ZK 3.5.6
>>> The stacktrace is below.
>>> The address on the first line "
>>> http://192.168.15.10:888/solr/mycollection; is the "server" address in
>>> my nginx configuration, which points to 2 upstream solr nodes. There were
>>> no other solr or ZK messages in the logs.
>>>
>>> StackTrace:
>>> (Msg = IOException occured when talking to server at:
>>> http://192.168.15.10:888/solr/mycollection)
>>> org.apache.solr.client.solrj.SolrServerException: IOException occured
>>> when talking to server at: http://192.168.15.10:888/solr/mycollection
>>> at
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
>>> at
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
>>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
>>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
>>> ... calls from our code
>>> ... calls from our code
>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at java.base/java.lang.Thread.run(Thread.java:834)
>>> Caused by: java.nio.channels.AsynchronousCloseException
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
>>> at
>>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
>>> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
>>> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
>>> at
>>> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
>>> at
>>> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
>>> at
>>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
>>> at
>>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
>>> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
>>> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>>> at
>>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>> at
>>> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>> at
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>>> at
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
>>> ... 3 more
>>>
>>> -
>>>

Re: IOException occured when talking to server

[Dominique Bejean]

If you want a more detailed debug information from your client application,
you can add this parameter while starting Solr JVM.
-Djavax.net.debug=all

It is very verbose !

Dominique


Le lun. 17 août 2020 à 17:59, Dominique Bejean 
a écrit :

> Hi,
>
> It looks like this issues
> https://github.com/eclipse/jetty.project/issues/4883
> https://github.com/eclipse/jetty.project/issues/2571
>
> The Nginx server closed the connection. Any info in nginx log ?
>
> Dominique
>
> Le lun. 17 août 2020 à 17:33, Odysci  a écrit :
>
>> Hi,
>> thanks for the reply.
>> We're using solr 8.3.1, ZK 3.5.6
>> The stacktrace is below.
>> The address on the first line "http://192.168.15.10:888/solr/mycollection;
>> is the "server" address in my nginx configuration, which points to 2
>> upstream solr nodes. There were no other solr or ZK messages in the logs.
>>
>> StackTrace:
>> (Msg = IOException occured when talking to server at:
>> http://192.168.15.10:888/solr/mycollection)
>> org.apache.solr.client.solrj.SolrServerException: IOException occured
>> when talking to server at: http://192.168.15.10:888/solr/mycollection
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
>> ... calls from our code
>> ... calls from our code
>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> Caused by: java.nio.channels.AsynchronousCloseException
>> at
>> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
>> at
>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
>> at
>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
>> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
>> at
>> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
>> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
>> at
>> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
>> at
>> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
>> at
>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
>> at
>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
>> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
>> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
>> ... 3 more
>>
>> -
>>
>> I did consider using the solrJ cloud or lb clients, but nginx gives me
>> more flexibility in controlling how the load balancing is done. I'm still
>> running experiments to see which one works best for me.
>> In the meantime, if you have 

Re: IOException occured when talking to server

[Dominique Bejean]

Hi,

It looks like this issues
https://github.com/eclipse/jetty.project/issues/4883
https://github.com/eclipse/jetty.project/issues/2571

The Nginx server closed the connection. Any info in nginx log ?

Dominique

Le lun. 17 août 2020 à 17:33, Odysci  a écrit :

> Hi,
> thanks for the reply.
> We're using solr 8.3.1, ZK 3.5.6
> The stacktrace is below.
> The address on the first line "http://192.168.15.10:888/solr/mycollection;
> is the "server" address in my nginx configuration, which points to 2
> upstream solr nodes. There were no other solr or ZK messages in the logs.
>
> StackTrace:
> (Msg = IOException occured when talking to server at:
> http://192.168.15.10:888/solr/mycollection)
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://192.168.15.10:888/solr/mycollection
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
> ... calls from our code
> ... calls from our code
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.nio.channels.AsynchronousCloseException
> at
> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
> at
> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
> at
> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
> at org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
> at
> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
> at
> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
> at
> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
> at
> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
> at
> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
> at
> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
> at
> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
> at
> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ... 3 more
>
> -
>
> I did consider using the solrJ cloud or lb clients, but nginx gives me
> more flexibility in controlling how the load balancing is done. I'm still
> running experiments to see which one works best for me.
> In the meantime, if you have any clues for why I'm getting this
> IOException, I'd appreciate.
> Thanks
>
> Reinaldo
>
>
>
> On Mon, Aug 17, 2020 at 10:59 AM Dominique Bejean <
> dominique.bej...@eolya.fr> wrote:
>
>> Hi,
>>
>>
>>
>>
>>
>> Can you provide more information ?
>>
>>
>> - Solr and ZK version
>>
>>
>> - full error stacktrace generated by SolrJ
>>
>>
>> - any 

Re: IOException occured when talking to server

[Odysci]

Hi,
thanks for the reply.
We're using solr 8.3.1, ZK 3.5.6
The stacktrace is below.
The address on the first line "http://192.168.15.10:888/solr/mycollection;
is the "server" address in my nginx configuration, which points to 2
upstream solr nodes. There were no other solr or ZK messages in the logs.

StackTrace:
(Msg = IOException occured when talking to server at:
http://192.168.15.10:888/solr/mycollection)
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://192.168.15.10:888/solr/mycollection
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051)
... calls from our code
... calls from our code
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.channels.AsynchronousCloseException
at
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144)
at
org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
at
org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
at org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153)
at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438)
at
org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
at
org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
at
org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
at
org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108)
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
... 3 more

-

I did consider using the solrJ cloud or lb clients, but nginx gives me more
flexibility in controlling how the load balancing is done. I'm still
running experiments to see which one works best for me.
In the meantime, if you have any clues for why I'm getting this
IOException, I'd appreciate.
Thanks

Reinaldo



On Mon, Aug 17, 2020 at 10:59 AM Dominique Bejean 
wrote:

> Hi,
>
> Can you provide more information ?
> - Solr and ZK version
> - full error stacktrace generated by SolrJ
> - any concomitant and relevant information in solr nodes logs or zk logs
>
> Just to know, why not use a load balanced LBHttp... Solr Client ?
>
> Regards.
>
> Dominique
>
>
> Le lun. 17 août 2020 à 00:41, Odysci  a écrit :
>
> > Hi,
> >
> > We have a solrcloud setup with 2 solr nodes and 3 ZK instances. Until
> > recently I had my application server always call one of the solr nodes
> (via
> > solrJ), and it worked just fine.
> >
> > In order to improve reliability I put an Nginx reverse-proxy load
> balancer
> > between my 

RE: Manipulating client's query using a Query object

[Markus Jelsma]

Hello Edward,

Yes you can by extending ExtendedDismaxQParser [1] and override its parse() 
method. You get the main Query object through super.parse().

If you need even more fine grained control on how Query objects are created you 
can extend ExtendedSolrQueryParser's [2] (inner class) newFieldQuery() method.

Regards,
Markus

[1] 
https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.html
[2] 
https://lucene.apache.org/solr/8_6_0/solr-core/org/apache/solr/search/ExtendedDismaxQParser.ExtendedSolrQueryParser.html

-Original message-
> From:Edward Turner 
> Sent: Monday 17th August 2020 15:53
> To: solr-user@lucene.apache.org
> Subject: Manipulating client's query using a Query object
> 
> Hi all,
> 
> Thanks for all your help recently. We're now using the edismax query parser
> and are happy with its behaviour. We have another question which maybe
> someone can help with.
> 
> We have one use case where we optimise our query before sending it to Solr,
> and we do this by manipulating the client's input query string. However,
> we're slightly uncomfortable using String manipulation to do this as
> there's always the possibility we parse their string wrongly. (We have a
> large test suite to check if we're doing the right thing, but even then, we
> String manipulation doesn't feel right ...).
> 
> Question: is it possible to get a Lucene Query representation of the
> client's query, which we can then navigate and manipulate -- before we then
> send the String representation of this Query to Solr for evaluation?
> 
> Kind regards and thank you for your help in advance,
> 
> Edd
> 

Re: IOException occured when talking to server

[Dominique Bejean]

Hi,

Can you provide more information ?
- Solr and ZK version
- full error stacktrace generated by SolrJ
- any concomitant and relevant information in solr nodes logs or zk logs

Just to know, why not use a load balanced LBHttp... Solr Client ?

Regards.

Dominique


Le lun. 17 août 2020 à 00:41, Odysci  a écrit :

> Hi,
>
> We have a solrcloud setup with 2 solr nodes and 3 ZK instances. Until
> recently I had my application server always call one of the solr nodes (via
> solrJ), and it worked just fine.
>
> In order to improve reliability I put an Nginx reverse-proxy load balancer
> between my application server and the solr nodes. The response time
> remained almost the same but we started getting the msg:
>
> IOException occured when talking to server http://myserver
>
> every minute or so (very randomly but consistently). Since our code will
> just try again after a few milliseconds, the overall system continues to
> work fine, despite the delay. I tried increasing all nginx related
> timeout's to no avail.
> I've searched for this msg a lot and most replies seem to be related to
> ssl.
> We are using http2solrclient but no ssl to solr.
> Can anyone shed any light on this?
>
> Thanks!
> Reinaldo
>

Manipulating client's query using a Query object

[Edward Turner]

Hi all,

Thanks for all your help recently. We're now using the edismax query parser
and are happy with its behaviour. We have another question which maybe
someone can help with.

We have one use case where we optimise our query before sending it to Solr,
and we do this by manipulating the client's input query string. However,
we're slightly uncomfortable using String manipulation to do this as
there's always the possibility we parse their string wrongly. (We have a
large test suite to check if we're doing the right thing, but even then, we
String manipulation doesn't feel right ...).

Question: is it possible to get a Lucene Query representation of the
client's query, which we can then navigate and manipulate -- before we then
send the String representation of this Query to Solr for evaluation?

Kind regards and thank you for your help in advance,

Edd

Solr with HDFS

[Prashant Jyoti]

Hi,
I am trying to get Solr running with HDFS but getting the attached
exception in logs when trying to create a collection. I have attached the
relevant portions of solrconfig.xml and solr.in.cmd that I have modified.
Could anybody point me in the right direction? What might I be doing wrong?
Any links would be helpful.

The steps that I am following are as under:

   1. Make reqd hdfs changes in solr.in.cmd and solrconfig.xml
   2. start solr in cloud mode
   3. upload configuration to zookeeper
   4. create collection with solr create on the command line


Thanks!

-- 
Regards,
Prashant.
 Changes in solr.in.cmd 


set SOLR_OPTS=%SOLR_OPTS% -Dsolr.directoryFactory=HdfsDirectoryFactory
set SOLR_OPTS=%SOLR_OPTS% -Dsolr.lock.type=hdfs
set SOLR_OPTS=%SOLR_OPTS% 
-Dsolr.hdfs.home=hdfs://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data



 Changes in solrconfig.xml 


 
  hdfs://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data
  /etc/hadoop/conf
  true
  1
  true
  16384
  true
  true
  16
  192