Abhishek, There is a wiki regarding this:
http://wiki.apache.org/solr/CommonQueryParameters search "pageDoc and pageScore". On Mon, Apr 29, 2013 at 1:17 PM, Abhishek Sanoujam <abhi.sanou...@gmail.com>wrote: > We have a single shard, and all the data is in a single box only. > Definitely looks like "deep-paging" is having problems. > > Just to understand, is the searcher looping over the result set everytime > and skipping the first "start" count? This will definitely take a toll when > we reach higher "start" values. > > > > > On 4/29/13 2:28 PM, Jan Høydahl wrote: > >> Hi, >> >> How many shards do you have? This is a known issue with deep paging with >> multi shard, see >> https://issues.apache.org/**jira/browse/SOLR-1726<https://issues.apache.org/jira/browse/SOLR-1726> >> >> You may be more successful in going to each shard, one at a time (with >> &distrib=false) to avoid this issue. >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> Solr Training - www.solrtraining.com >> >> 29. apr. 2013 kl. 09:17 skrev Abhishek Sanoujam <abhi.sanou...@gmail.com >> >: >> >> We have a solr core with about 115 million documents. We are trying to >>> migrate data and running a simple query with *:* query and with start and >>> rows param. >>> The performance is becoming too slow in solr, its taking almost 2 mins >>> to get 4000 rows and migration is being just too slow. Logs snippet below: >>> >>> INFO: [coreName] webapp=/solr path=/select params={start=55438000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=168308 >>> INFO: [coreName] webapp=/solr path=/select params={start=55446000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=122771 >>> INFO: [coreName] webapp=/solr path=/select params={start=55454000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=137615 >>> INFO: [coreName] webapp=/solr path=/select params={start=55450000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=141223 >>> INFO: [coreName] webapp=/solr path=/select params={start=55462000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=97474 >>> INFO: [coreName] webapp=/solr path=/select params={start=55458000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=98115 >>> INFO: [coreName] webapp=/solr path=/select params={start=55466000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=143822 >>> INFO: [coreName] webapp=/solr path=/select params={start=55474000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=118066 >>> INFO: [coreName] webapp=/solr path=/select params={start=55470000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=121498 >>> INFO: [coreName] webapp=/solr path=/select params={start=55482000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=164062 >>> INFO: [coreName] webapp=/solr path=/select params={start=55478000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=165518 >>> INFO: [coreName] webapp=/solr path=/select params={start=55486000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=118163 >>> INFO: [coreName] webapp=/solr path=/select params={start=55494000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=141642 >>> INFO: [coreName] webapp=/solr path=/select params={start=55490000&q=*:*& >>> **wt=javabin&version=2&rows=**4000} hits=115760479 status=0 QTime=145037 >>> >>> >>> I've taken some thread dumps in the solr server and most of the time the >>> threads seem to be busy in the following stacks mostly: >>> Is there anything that can be done to improve the performance? Is it a >>> known issue? Its very surprising that querying for some just rows starting >>> at some points is taking in order of minutes. >>> >>> >>> "395883378@qtp-162198005-7" prio=10 tid=0x00007f4aa0636000 nid=0x295a >>> runnable [0x00007f42865dd000] >>> java.lang.Thread.State: RUNNABLE >>> at org.apache.lucene.util.**PriorityQueue.downHeap(** >>> PriorityQueue.java:252) >>> at org.apache.lucene.util.**PriorityQueue.pop(** >>> PriorityQueue.java:184) >>> at org.apache.lucene.search.**TopDocsCollector.** >>> populateResults(**TopDocsCollector.java:61) >>> at org.apache.lucene.search.**TopDocsCollector.topDocs(** >>> TopDocsCollector.java:156) >>> at org.apache.solr.search.**SolrIndexSearcher.**getDocListNC(** >>> SolrIndexSearcher.java:1499) >>> at org.apache.solr.search.**SolrIndexSearcher.getDocListC(** >>> SolrIndexSearcher.java:1366) >>> at org.apache.solr.search.**SolrIndexSearcher.search(** >>> SolrIndexSearcher.java:457) >>> at org.apache.solr.handler.**component.QueryComponent.** >>> process(QueryComponent.java:**410) >>> at org.apache.solr.handler.**component.SearchHandler.** >>> handleRequestBody(**SearchHandler.java:208) >>> at org.apache.solr.handler.**RequestHandlerBase.**handleRequest( >>> **RequestHandlerBase.java:135) >>> at org.apache.solr.core.SolrCore.**execute(SolrCore.java:1817) >>> at org.apache.solr.servlet.**SolrDispatchFilter.execute(** >>> SolrDispatchFilter.java:639) >>> at org.apache.solr.servlet.**SolrDispatchFilter.doFilter(** >>> SolrDispatchFilter.java:345) >>> at org.apache.solr.servlet.**SolrDispatchFilter.doFilter(** >>> SolrDispatchFilter.java:141) >>> >>> >>> "1154127582@qtp-162198005-3" prio=10 tid=0x00007f4aa0613800 nid=0x2956 >>> runnable [0x00007f42869e1000] >>> java.lang.Thread.State: RUNNABLE >>> at org.apache.lucene.util.**PriorityQueue.downHeap(** >>> PriorityQueue.java:252) >>> at org.apache.lucene.util.**PriorityQueue.updateTop(** >>> PriorityQueue.java:210) >>> at org.apache.lucene.search.**TopScoreDocCollector$** >>> InOrderTopScoreDocCollector.**collect(TopScoreDocCollector.**java:62) >>> at org.apache.lucene.search.**Scorer.score(Scorer.java:64) >>> at org.apache.lucene.search.**IndexSearcher.search(** >>> IndexSearcher.java:605) >>> at org.apache.lucene.search.**IndexSearcher.search(** >>> IndexSearcher.java:297) >>> at org.apache.solr.search.**SolrIndexSearcher.**getDocListNC(** >>> SolrIndexSearcher.java:1491) >>> at org.apache.solr.search.**SolrIndexSearcher.getDocListC(** >>> SolrIndexSearcher.java:1366) >>> at org.apache.solr.search.**SolrIndexSearcher.search(** >>> SolrIndexSearcher.java:457) >>> at org.apache.solr.handler.**component.QueryComponent.** >>> process(QueryComponent.java:**410) >>> at org.apache.solr.handler.**component.SearchHandler.** >>> handleRequestBody(**SearchHandler.java:208) >>> at org.apache.solr.handler.**RequestHandlerBase.**handleRequest( >>> **RequestHandlerBase.java:135) >>> at org.apache.solr.core.SolrCore.**execute(SolrCore.java:1817) >>> at org.apache.solr.servlet.**SolrDispatchFilter.execute(** >>> SolrDispatchFilter.java:639) >>> at org.apache.solr.servlet.**SolrDispatchFilter.doFilter(** >>> SolrDispatchFilter.java:345) >>> at org.apache.solr.servlet.**SolrDispatchFilter.doFilter(** >>> SolrDispatchFilter.java:141) >>> >>> >>> -- >>> --------- >>> Cheers, >>> Abhishek >>> >>> > > -- > --------- > Cheers, > Abhishek > >
