Partial results with not enough hits
In all of my queries I have timeAllowed parameter. My application is ready for partial results. However, whenever Solr returns partial result it is a very bad result. For example, I have a test query and here its execution log with the strict time allowed: WARNING: Query: ; Elapsed time: 120Exceeded allowed search time: 100 ms. INFO: [] webapp=/solr path=/select params={&timeAllowed=100} hits=189 status=0 QTime=119 Here it is without such a strict limitation: INFO: [] webapp=/solr path=/select params={&timeAllowed=1} hits=582 status=0 QTime=124 The total execution time is different by mere 5 ms, but the partial result has only about 1/3 of the full result. Is it the expected behaviour? Does that mean I can never rely on the partial results? I added timeAllowed to protect from too expensive wide queries, but I still want to return something relevant to the user. This query returned 30% of the full result, but I have other queries in the log where partial result is just empty. Am I doing something wrong? P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory. Load Average on the Solr box is very low. -- Aleksey
Re: Partial results with not enough hits
It could be that the time to get set up to return even the first result is high and then each additional document is a minimal increment in time. Do a query with &rows=1 (or even 0) and see what the minimum query time is for your query, index, and environment. -- Jack Krupansky -Original Message- From: Aleksey Vorona Sent: Wednesday, November 21, 2012 6:04 PM To: solr-user@lucene.apache.org Subject: Partial results with not enough hits In all of my queries I have timeAllowed parameter. My application is ready for partial results. However, whenever Solr returns partial result it is a very bad result. For example, I have a test query and here its execution log with the strict time allowed: WARNING: Query: ; Elapsed time: 120Exceeded allowed search time: 100 ms. INFO: [] webapp=/solr path=/select params={&timeAllowed=100} hits=189 status=0 QTime=119 Here it is without such a strict limitation: INFO: [] webapp=/solr path=/select params={&timeAllowed=1} hits=582 status=0 QTime=124 The total execution time is different by mere 5 ms, but the partial result has only about 1/3 of the full result. Is it the expected behaviour? Does that mean I can never rely on the partial results? I added timeAllowed to protect from too expensive wide queries, but I still want to return something relevant to the user. This query returned 30% of the full result, but I have other queries in the log where partial result is just empty. Am I doing something wrong? P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory. Load Average on the Solr box is very low. -- Aleksey
Re: Partial results with not enough hits
Hi, Maybe your goal should be to make your queries faster instead of fighting with timeouts which are known not to work well. What is your hardware like? How about your queries? What do you see in debugQuery=true output? Otis -- SOLR Performance Monitoring - http://sematext.com/spm On Nov 21, 2012 6:04 PM, "Aleksey Vorona" wrote: > In all of my queries I have timeAllowed parameter. My application is ready > for partial results. However, whenever Solr returns partial result it is a > very bad result. > > For example, I have a test query and here its execution log with the > strict time allowed: > WARNING: Query: ; Elapsed time: 120Exceeded allowed search > time: 100 ms. > INFO: [] webapp=/solr path=/select params={&timeAllowed=**100} > hits=189 status=0 QTime=119 > Here it is without such a strict limitation: > INFO: [] webapp=/solr path=/select params={&timeAllowed=**1} > hits=582 status=0 QTime=124 > > The total execution time is different by mere 5 ms, but the partial result > has only about 1/3 of the full result. > > Is it the expected behaviour? Does that mean I can never rely on the > partial results? > > I added timeAllowed to protect from too expensive wide queries, but I > still want to return something relevant to the user. This query returned > 30% of the full result, but I have other queries in the log where partial > result is just empty. Am I doing something wrong? > > P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory. > Load Average on the Solr box is very low. > > -- Aleksey >
Re: Partial results with not enough hits
Thank you! That seems to be the case, I tried to execute queries without sorting and only one document in the response and I got execution time in the same range as before. -- Aleksey On 12-11-21 04:07 PM, Jack Krupansky wrote: It could be that the time to get set up to return even the first result is high and then each additional document is a minimal increment in time. Do a query with &rows=1 (or even 0) and see what the minimum query time is for your query, index, and environment. -- Jack Krupansky -Original Message- From: Aleksey Vorona Sent: Wednesday, November 21, 2012 6:04 PM To: solr-user@lucene.apache.org Subject: Partial results with not enough hits In all of my queries I have timeAllowed parameter. My application is ready for partial results. However, whenever Solr returns partial result it is a very bad result. For example, I have a test query and here its execution log with the strict time allowed: WARNING: Query: ; Elapsed time: 120Exceeded allowed search time: 100 ms. INFO: [] webapp=/solr path=/select params={&timeAllowed=100} hits=189 status=0 QTime=119 Here it is without such a strict limitation: INFO: [] webapp=/solr path=/select params={&timeAllowed=1} hits=582 status=0 QTime=124 The total execution time is different by mere 5 ms, but the partial result has only about 1/3 of the full result. Is it the expected behaviour? Does that mean I can never rely on the partial results? I added timeAllowed to protect from too expensive wide queries, but I still want to return something relevant to the user. This query returned 30% of the full result, but I have other queries in the log where partial result is just empty. Am I doing something wrong? P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory. Load Average on the Solr box is very low. -- Aleksey
Re: Partial results with not enough hits
Thanks for the response. I have increased the timeout and it did not increase execution time or system load. It is really that I misused the timeout. Just to give you a bit of perspective, we added timeout to guarantee some level of QoS from the search engine. Our UI allows user to construct very complex queries and (what is worse) not all the time user really understands what she needs. That may become a problem if we have lots of users doing that. In this case I do not want to run such a complex query for seconds and want to return some result with a warning to the user that she is doing something wrong. But clearly, I set a timeout too low for that and started to harm even normal queries. Anyway, thanks everyone for the replies. The issue is fixed and I now understand how timeout works much better (which was the reason to post to this list). Thanks! -- Aleksey On 12-11-22 06:37 AM, Otis Gospodnetic wrote: Hi, Maybe your goal should be to make your queries faster instead of fighting with timeouts which are known not to work well. What is your hardware like? How about your queries? What do you see in debugQuery=true output? Otis -- SOLR Performance Monitoring - http://sematext.com/spm On Nov 21, 2012 6:04 PM, "Aleksey Vorona" wrote: In all of my queries I have timeAllowed parameter. My application is ready for partial results. However, whenever Solr returns partial result it is a very bad result. For example, I have a test query and here its execution log with the strict time allowed: WARNING: Query: ; Elapsed time: 120Exceeded allowed search time: 100 ms. INFO: [] webapp=/solr path=/select params={&timeAllowed=**100} hits=189 status=0 QTime=119 Here it is without such a strict limitation: INFO: [] webapp=/solr path=/select params={&timeAllowed=**1} hits=582 status=0 QTime=124 The total execution time is different by mere 5 ms, but the partial result has only about 1/3 of the full result. Is it the expected behaviour? Does that mean I can never rely on the partial results? I added timeAllowed to protect from too expensive wide queries, but I still want to return something relevant to the user. This query returned 30% of the full result, but I have other queries in the log where partial result is just empty. Am I doing something wrong? P.S. I am using Solr 3.6.1, index size is 3Gb and easily fits in memory. Load Average on the Solr box is very low. -- Aleksey