Re: Potential Slow searching for unified highlighting on Solr 8.8.0/8.8.1
Hi,
Solr uses JIRA for issue tickets. You can find it here:
https://issues.apache.org/jira/browse/SOLR
I'd suggest filing a new bug issue in the SOLR project (note that
several other projects also use this JIRA installation). Here's an
example of an existing highlighter issue for reference:
https://issues.apache.org/jira/browse/SOLR-14019.
See also some brief documentation:
https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-JIRAtips(ourissue/bugtracker)
Regards,
Ere
Flowerday, Matthew J kirjoitti 1.3.2021 klo 14.58:
Hi Ere
Please to be of service!
No I have not filed a JIRA ticket. I am new to interacting with the Solr
Community and only beginning to 'find my legs'. I am not too sure what JIRA
is I am afraid!
Regards
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
-Original Message-
From: Ere Maijala
Sent: 01 March 2021 12:53
To: solr-user@lucene.apache.org
Subject: Re: Potential Slow searching for unified highlighting on Solr
8.8.0/8.8.1
EXTERNAL EMAIL - Be cautious of all links and attachments.
Hi,
Whoa, thanks for the heads-up! You may just have saved me from a whole lot
of trouble. Did you file a JIRA ticket already?
Thanks,
Ere
Flowerday, Matthew J kirjoitti 1.3.2021 klo 14.00:
Hi There
I just came across a situation where a unified highlighting search
under solr 8.8.0/8.8.1 can take over 20 mins to run and eventually times
out.
I resolved it by a config change – but it can catch you out. Hence
this email.
With solr 8.8.0 a new unified highlighting parameter
was implemented which if not set defaults to 0.5.
This attempts to improve the high lighting so that highlighted text
does not appear right at the left. This works well but if you have a
search result with numerous occurrences of the word in question within
the record performance goes right down!
2021-02-27 06:45:03.151 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.c.S.Request [uleaf] webapp=/solr path=/select
params={hl.snippets=2=test=on=100=id,d
escription,specification,score=20=*=10&_=161440511913
4}
hits=57008 status=0 QTime=1414320
2021-02-27 06:45:03.245 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.s.HttpSolrCall Unable to write response, client closed
connection or we are shutting down =>
org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
org.eclipse.jetty.io.EofException: null
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
when I set =0.25 results came back much quicker
2021-02-27 14:59:57.189 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,s
core=1=0.25=100=2=test
axAnalyzedChars=100=*=unified=9&_=
1614430061690}
hits=136939 status=0 QTime=87024
And =0.1
2021-02-27 15:18:45.542 INFO (qtp1291367132-19) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,s
core=1=0.1=100=2=test
xAnalyzedChars=100=*=unified=9&_=1
614430061690}
hits=136939 status=0 QTime=69033
And =0.0
2021-02-27 15:20:38.194 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,s
core=1=0.0=100=2=test
xAnalyzedChars=100=*=unified=9&_=1
614430061690}
hits=136939 status=0 QTime=2841
I left our setting at 0.0 – this presumably how it was in 7.7.1 (fully
left aligned). I am not too sure as to how many time a word has to
occur in a record for performance to go right down – but if too many
it can have a BIG impact.
I also noticed that setting =9 did not break out of
the query until it finished. Perhaps because the query finished
quickly and what took the time was the highlighting. It might be an
idea to get to also cover any highlighting so that the
query does not run until the jetty timeout is hit. The machine 100%
one core for about
20 mins!.
Hope this helps.
Regards
Matthew
*Matthew Flowerday*| Consultant | ULEAF
Unisys | 01908 774830| matthew.flower...@unisys.com
<mailto:matthew.flower...@unisys.com>
Address Enigma | Wavendon Business Park |
RE: Potential Slow searching for unified highlighting on Solr 8.8.0/8.8.1
Hi Ere
Please to be of service!
No I have not filed a JIRA ticket. I am new to interacting with the Solr
Community and only beginning to 'find my legs'. I am not too sure what JIRA
is I am afraid!
Regards
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
-Original Message-
From: Ere Maijala
Sent: 01 March 2021 12:53
To: solr-user@lucene.apache.org
Subject: Re: Potential Slow searching for unified highlighting on Solr
8.8.0/8.8.1
EXTERNAL EMAIL - Be cautious of all links and attachments.
Hi,
Whoa, thanks for the heads-up! You may just have saved me from a whole lot
of trouble. Did you file a JIRA ticket already?
Thanks,
Ere
Flowerday, Matthew J kirjoitti 1.3.2021 klo 14.00:
> Hi There
>
> I just came across a situation where a unified highlighting search
> under solr 8.8.0/8.8.1 can take over 20 mins to run and eventually times
out.
> I resolved it by a config change but it can catch you out. Hence
> this email.
>
> With solr 8.8.0 a new unified highlighting parameter
> was implemented which if not set defaults to 0.5.
> This attempts to improve the high lighting so that highlighted text
> does not appear right at the left. This works well but if you have a
> search result with numerous occurrences of the word in question within
> the record performance goes right down!
>
> 2021-02-27 06:45:03.151 INFO (qtp762476028-20) [ x:uleaf]
> o.a.s.c.S.Request [uleaf] webapp=/solr path=/select
> params={hl.snippets=2=test=on=100=id,d
> escription,specification,score=20=*=10&_=161440511913
> 4}
> hits=57008 status=0 QTime=1414320
>
> 2021-02-27 06:45:03.245 INFO (qtp762476028-20) [ x:uleaf]
> o.a.s.s.HttpSolrCall Unable to write response, client closed
> connection or we are shutting down =>
> org.eclipse.jetty.io.EofException
>
>at
> org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
>
> org.eclipse.jetty.io.EofException: null
>
>at
> org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
>at
> org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
>at
> org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
> when I set =0.25 results came back much quicker
>
> 2021-02-27 14:59:57.189 INFO (qtp1291367132-24) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.25=100=2=test
> axAnalyzedChars=100=*=unified=9&_=
> 1614430061690}
> hits=136939 status=0 QTime=87024
>
> And =0.1
>
> 2021-02-27 15:18:45.542 INFO (qtp1291367132-19) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.1=100=2=test
> xAnalyzedChars=100=*=unified=9&_=1
> 614430061690}
> hits=136939 status=0 QTime=69033
>
> And =0.0
>
> 2021-02-27 15:20:38.194 INFO (qtp1291367132-24) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.0=100=2=test
> xAnalyzedChars=100=*=unified=9&_=1
> 614430061690}
> hits=136939 status=0 QTime=2841
>
> I left our setting at 0.0 this presumably how it was in 7.7.1 (fully
> left aligned). I am not too sure as to how many time a word has to
> occur in a record for performance to go right down but if too many
> it can have a BIG impact.
>
> I also noticed that setting =9 did not break out of
> the query until it finished. Perhaps because the query finished
> quickly and what took the time was the highlighting. It might be an
> idea to get to also cover any highlighting so that the
> query does not run until the jetty timeout is hit. The machine 100%
> one core for about
> 20 mins!.
>
> Hope this helps.
>
> Regards
>
> Matthew
>
> *Matthew Flowerday*| Consultant | ULEAF
>
> Unisys | 01908 774830| matthew.flower...@unisys.com
> <mailto:matthew.flower...@unisys.com>
>
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes |
> MK17 8LX
>
> unisys_logo <http://www.
Re: Potential Slow searching for unified highlighting on Solr 8.8.0/8.8.1
Hi,
Whoa, thanks for the heads-up! You may just have saved me from a whole
lot of trouble. Did you file a JIRA ticket already?
Thanks,
Ere
Flowerday, Matthew J kirjoitti 1.3.2021 klo 14.00:
Hi There
I just came across a situation where a unified highlighting search under
solr 8.8.0/8.8.1 can take over 20 mins to run and eventually times out.
I resolved it by a config change – but it can catch you out. Hence this
email.
With solr 8.8.0 a new unified highlighting parameter
was implemented which if not set defaults to 0.5. This attempts to
improve the high lighting so that highlighted text does not appear right
at the left. This works well but if you have a search result with
numerous occurrences of the word in question within the record
performance goes right down!
2021-02-27 06:45:03.151 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.c.S.Request [uleaf] webapp=/solr path=/select
params={hl.snippets=2=test=on=100=id,description,specification,score=20=*=10&_=1614405119134}
hits=57008 status=0 QTime=1414320
2021-02-27 06:45:03.245 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.s.HttpSolrCall Unable to write response, client closed connection
or we are shutting down => org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
org.eclipse.jetty.io.EofException: null
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
when I set =0.25 results came back much quicker
2021-02-27 14:59:57.189 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score=1=0.25=100=2=test=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=87024
And =0.1
2021-02-27 15:18:45.542 INFO (qtp1291367132-19) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score=1=0.1=100=2=test=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=69033
And =0.0
2021-02-27 15:20:38.194 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score=1=0.0=100=2=test=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=2841
I left our setting at 0.0 – this presumably how it was in 7.7.1 (fully
left aligned). I am not too sure as to how many time a word has to
occur in a record for performance to go right down – but if too many it
can have a BIG impact.
I also noticed that setting =9 did not break out of the
query until it finished. Perhaps because the query finished quickly and
what took the time was the highlighting. It might be an idea to get
to also cover any highlighting so that the query does not
run until the jetty timeout is hit. The machine 100% one core for about
20 mins!.
Hope this helps.
Regards
Matthew
*Matthew Flowerday*| Consultant | ULEAF
Unisys | 01908 774830| matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes |
MK17 8LX
unisys_logo Grey_TW
Grey_YT
Grey_FB
Grey_Vimeo
Grey_UB
