Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Grant Ingersoll
On Dec 24, 2009, at 11:09 AM, Fuad Efendi wrote: > I used pagination for a while till found this... > > > I have filtered query ID:[* TO *] returning 20 millions results (no > faceting), and pagination always seemed to be fast. However, fast only with > low values for start=12345. Queries like

Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Walter Underwood
When do users do a query like that? --wunder On Dec 24, 2009, at 8:09 AM, Fuad Efendi wrote: > I used pagination for a while till found this... > > > I have filtered query ID:[* TO *] returning 20 millions results (no > faceting), and pagination always seemed to be fast. However, fast only with

Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Joe Calderon
fwiw, when implementing distributed search i ran into a similar problem, but then i noticed even google doesnt let you go past page 1000, easier to just set a limit on start On Thu, Dec 24, 2009 at 8:36 AM, Walter Underwood wrote: > When do users do a query like that? --wunder > > On Dec 24, 200

Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Erik Hatcher
On Dec 24, 2009, at 11:36 AM, Walter Underwood wrote: When do users do a query like that? --wunder Well, SolrEntityProcessor "users" do :) http://issues.apache.org/jira/browse/SOLR-1499 (which by the way I plan on polishing and committing over the holidays) Erik On Dec 24

Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Walter Underwood
Some bots will do that, too. Maybe badly written ones, but we saw that at Netflix. It was causing search timeouts just before a peak traffic period, so we set a page limit in the front end, something like 200 pages. It makes sense for that to be very slow, because a request for hit 28838540 mea

RE: SOLR Performance Tuning: Pagination

2009-12-24 Thread Fuad Efendi
; To: solr-user@lucene.apache.org > Subject: Re: SOLR Performance Tuning: Pagination > > Some bots will do that, too. Maybe badly written ones, but we saw that at > Netflix. It was causing search timeouts just before a peak traffic period, > so we set a page limit in the front end, something

Re: SOLR Performance Tuning: Pagination

2009-12-24 Thread Paul Rosen
olr call that is applied to the relevance before sorting. [It also made me jump through hoops when I wrote some unit tests for the indexing.] -Original Message- From: Walter Underwood [mailto:wun...@wunderwood.org] Sent: December-24-09 1:51 PM To: solr-user@lucene.apache.org Subj

RE: SOLR Performance Tuning: Pagination

2009-12-24 Thread Fuad Efendi
. > -Original Message- > From: Walter Underwood [mailto:wun...@wunderwood.org] > Sent: December-24-09 11:37 AM > To: solr-user@lucene.apache.org > Subject: Re: SOLR Performance Tuning: Pagination > > When do users do a query like that? --wunder > > On Dec

RE: SOLR Performance Tuning: Pagination

2009-12-24 Thread Fuad Efendi
). But some queries may return huge nuber of documents (better is to tune "stop-word" list) -Fuad > -Original Message- > From: Walter Underwood [mailto:wun...@wunderwood.org] > Sent: December-24-09 1:51 PM > To: solr-user@lucene.apache.org > Subject: Re: SOLR Perfo

Re: SOLR Performance Tuning: Pagination

2009-12-25 Thread Grant Ingersoll
On Dec 24, 2009, at 1:51 PM, Walter Underwood wrote: > Some bots will do that, too. Maybe badly written ones, but we saw that at > Netflix. It was causing search timeouts just before a peak traffic period, so > we set a page limit in the front end, something like 200 pages. > > It makes sense

Re: SOLR Performance Tuning: Pagination

2010-01-03 Thread Peter Wolanin
At the NOVA Apache Lucene/Solr Meetup last May, one of the speakers from Near Infinity (Aaron McCurry I think) mentioned that he had a patch for lucene that enabled unlimited depth memory-efficient paging. Is anyone in contact with him? -Peter On Thu, Dec 24, 2009 at 11:27 AM, Grant Ingersoll w

Re: SOLR Performance Tuning: Pagination

2010-01-07 Thread Otis Gospodnetic
nin > To: solr-user@lucene.apache.org > Sent: Sun, January 3, 2010 3:37:01 PM > Subject: Re: SOLR Performance Tuning: Pagination > > At the NOVA Apache Lucene/Solr Meetup last May, one of the speakers > from Near Infinity (Aaron McCurry I think) mentioned that he had a > patch

Re: SOLR Performance Tuning: Pagination

2010-01-07 Thread Peter Wolanin
un, January 3, 2010 3:37:01 PM >> Subject: Re: SOLR Performance Tuning: Pagination >> >> At the NOVA Apache Lucene/Solr Meetup last May, one of the speakers >> from Near Infinity (Aaron McCurry I think) mentioned that he had a >> patch for lucene that enabled unlimi

Re: SOLR Performance Tuning: Pagination

2010-01-07 Thread Otis Gospodnetic
Si si, that issue. Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message > From: Peter Wolanin > To: solr-user@lucene.apache.org > Sent: Thu, January 7, 2010 9:27:04 PM > Subject: Re: SOLR Performance Tuning: Pagination > > Great -