Partial results with not enough hits
In all of my queries I have timeAllowed parameter. My application is
ready for partial results. However, whenever Solr returns partial result
it is a very bad result.
For example, I have a test query and here its execution log with the
strict time allowed:
WARNING: Query: ; Elapsed time: 120Exceeded allowed search
time: 100 ms.
INFO: [] webapp=/solr path=/select
params={&timeAllowed=100} hits=189 status=0 QTime=119
Here it is without such a strict limitation:
INFO: [] webapp=/solr path=/select
params={&timeAllowed=1} hits=582 status=0 QTime=124
The total execution time is different by mere 5 ms, but the partial
result has only about 1/3 of the full result.
Is it the expected behaviour? Does that mean I can never rely on the
partial results?
I added timeAllowed to protect from too expensive wide queries, but I
still want to return something relevant to the user. This query returned
30% of the full result, but I have other queries in the log where
partial result is just empty. Am I doing something wrong?
P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory.
Load Average on the Solr box is very low.
-- Aleksey
Re: Partial results with not enough hits
It could be that the time to get set up to return even the first result is
high and then each additional document is a minimal increment in time.
Do a query with &rows=1 (or even 0) and see what the minimum query time is
for your query, index, and environment.
-- Jack Krupansky
-Original Message-
From: Aleksey Vorona
Sent: Wednesday, November 21, 2012 6:04 PM
To: solr-user@lucene.apache.org
Subject: Partial results with not enough hits
In all of my queries I have timeAllowed parameter. My application is
ready for partial results. However, whenever Solr returns partial result
it is a very bad result.
For example, I have a test query and here its execution log with the
strict time allowed:
WARNING: Query: ; Elapsed time: 120Exceeded allowed search
time: 100 ms.
INFO: [] webapp=/solr path=/select
params={&timeAllowed=100} hits=189 status=0 QTime=119
Here it is without such a strict limitation:
INFO: [] webapp=/solr path=/select
params={&timeAllowed=1} hits=582 status=0 QTime=124
The total execution time is different by mere 5 ms, but the partial
result has only about 1/3 of the full result.
Is it the expected behaviour? Does that mean I can never rely on the
partial results?
I added timeAllowed to protect from too expensive wide queries, but I
still want to return something relevant to the user. This query returned
30% of the full result, but I have other queries in the log where
partial result is just empty. Am I doing something wrong?
P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory.
Load Average on the Solr box is very low.
-- Aleksey
Re: Partial results with not enough hits
Hi,
Maybe your goal should be to make your queries faster instead of fighting
with timeouts which are known not to work well.
What is your hardware like?
How about your queries?
What do you see in debugQuery=true output?
Otis
--
SOLR Performance Monitoring - http://sematext.com/spm
On Nov 21, 2012 6:04 PM, "Aleksey Vorona" wrote:
> In all of my queries I have timeAllowed parameter. My application is ready
> for partial results. However, whenever Solr returns partial result it is a
> very bad result.
>
> For example, I have a test query and here its execution log with the
> strict time allowed:
> WARNING: Query: ; Elapsed time: 120Exceeded allowed search
> time: 100 ms.
> INFO: [] webapp=/solr path=/select params={&timeAllowed=**100}
> hits=189 status=0 QTime=119
> Here it is without such a strict limitation:
> INFO: [] webapp=/solr path=/select params={&timeAllowed=**1}
> hits=582 status=0 QTime=124
>
> The total execution time is different by mere 5 ms, but the partial result
> has only about 1/3 of the full result.
>
> Is it the expected behaviour? Does that mean I can never rely on the
> partial results?
>
> I added timeAllowed to protect from too expensive wide queries, but I
> still want to return something relevant to the user. This query returned
> 30% of the full result, but I have other queries in the log where partial
> result is just empty. Am I doing something wrong?
>
> P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory.
> Load Average on the Solr box is very low.
>
> -- Aleksey
>
Re: Partial results with not enough hits
Thank you!
That seems to be the case, I tried to execute queries without sorting
and only one document in the response and I got execution time in the
same range as before.
-- Aleksey
On 12-11-21 04:07 PM, Jack Krupansky wrote:
It could be that the time to get set up to return even the first result is
high and then each additional document is a minimal increment in time.
Do a query with &rows=1 (or even 0) and see what the minimum query time is
for your query, index, and environment.
-- Jack Krupansky
-Original Message-
From: Aleksey Vorona
Sent: Wednesday, November 21, 2012 6:04 PM
To: solr-user@lucene.apache.org
Subject: Partial results with not enough hits
In all of my queries I have timeAllowed parameter. My application is
ready for partial results. However, whenever Solr returns partial result
it is a very bad result.
For example, I have a test query and here its execution log with the
strict time allowed:
WARNING: Query: ; Elapsed time: 120Exceeded allowed search
time: 100 ms.
INFO: [] webapp=/solr path=/select
params={&timeAllowed=100} hits=189 status=0 QTime=119
Here it is without such a strict limitation:
INFO: [] webapp=/solr path=/select
params={&timeAllowed=1} hits=582 status=0 QTime=124
The total execution time is different by mere 5 ms, but the partial
result has only about 1/3 of the full result.
Is it the expected behaviour? Does that mean I can never rely on the
partial results?
I added timeAllowed to protect from too expensive wide queries, but I
still want to return something relevant to the user. This query returned
30% of the full result, but I have other queries in the log where
partial result is just empty. Am I doing something wrong?
P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory.
Load Average on the Solr box is very low.
-- Aleksey
Re: Partial results with not enough hits
Thanks for the response.
I have increased the timeout and it did not increase execution time or
system load. It is really that I misused the timeout.
Just to give you a bit of perspective, we added timeout to guarantee
some level of QoS from the search engine. Our UI allows user to
construct very complex queries and (what is worse) not all the time user
really understands what she needs. That may become a problem if we have
lots of users doing that. In this case I do not want to run such a
complex query for seconds and want to return some result with a warning
to the user that she is doing something wrong. But clearly, I set a
timeout too low for that and started to harm even normal queries.
Anyway, thanks everyone for the replies. The issue is fixed and I now
understand how timeout works much better (which was the reason to post
to this list). Thanks!
-- Aleksey
On 12-11-22 06:37 AM, Otis Gospodnetic wrote:
Hi,
Maybe your goal should be to make your queries faster instead of fighting
with timeouts which are known not to work well.
What is your hardware like?
How about your queries?
What do you see in debugQuery=true output?
Otis
--
SOLR Performance Monitoring - http://sematext.com/spm
On Nov 21, 2012 6:04 PM, "Aleksey Vorona" wrote:
In all of my queries I have timeAllowed parameter. My application is ready
for partial results. However, whenever Solr returns partial result it is a
very bad result.
For example, I have a test query and here its execution log with the
strict time allowed:
WARNING: Query: ; Elapsed time: 120Exceeded allowed search
time: 100 ms.
INFO: [] webapp=/solr path=/select params={&timeAllowed=**100}
hits=189 status=0 QTime=119
Here it is without such a strict limitation:
INFO: [] webapp=/solr path=/select params={&timeAllowed=**1}
hits=582 status=0 QTime=124
The total execution time is different by mere 5 ms, but the partial result
has only about 1/3 of the full result.
Is it the expected behaviour? Does that mean I can never rely on the
partial results?
I added timeAllowed to protect from too expensive wide queries, but I
still want to return something relevant to the user. This query returned
30% of the full result, but I have other queries in the log where partial
result is just empty. Am I doing something wrong?
P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory.
Load Average on the Solr box is very low.
-- Aleksey
