Hi Erick, My OOM problem starts when I query the core with 13217121 documents. My schema and other details are given below,
1> how is your sort field defined? String? Integer? If it's a string and you could change it to a numeric type, you'd use a lot less memory. We primarily use two different sort criteria one is a date field and the other is string (id). I cannot change the "id" field as this is also the uniquekey for my schema. 2> How many distinct terms? I'm guessing one/document actually,this is somewhat of an anti-pattern in Solr for all it's sometimes necessary. Since one of the field is a timestamp instance and the other a unique key all are distinct. (These are tweets happening for keyword) 3> How much memory are you allocating for the JVM? I am starting solr with the following command java -Xms1024M -Xmx-2048M start.jar All out test case for moving to solr has passed, this is proving to be a big set back. Help would be greatly appreciated. Regards, Rohit -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: 19 May 2011 18:21 To: solr-user@lucene.apache.org Subject: Re: Out of memory on sorting The warming queries warm up the caches used in sorting. So just including the &sort=..... will warm the sort caches. the terms searched are not important. The same is true with facets... However, I don't understand how that relates to your OOM problems. I'd expect the OOM to start happening on startup, you'd be doing the operation that runs you out of memory on startup... So, we need more details: 1> how is your sort field defined? String? Integer? If it's a string and you could change it to a numeric type, you'd use a lot less memory. 2> How many distinct terms? I'm guessing one/document actually, this is somewhat of an anti-pattern in Solr for all it's sometimes necessary. 3> How much memory are you allocating for the JVM? 4> What other fields are you sorting on and how many unique values in each? Solr Admin can help you here.... Best Erick On Thu, May 19, 2011 at 6:20 AM, Rohit <ro...@in-rev.com> wrote: > Thanks for pointing me in the right direction, now I see the configuration > for firstsearcher or newsearcher, the <str name="q"> needs to configured > previously. In my case the q is every changing, users can actually search > for anything and the possibilities of queries unlimited. > > How can I make this generic? > > -Rohit > > > > -----Original Message----- > From: rajini maski [mailto:rajinima...@gmail.com] > Sent: 19 May 2011 14:53 > To: solr-user@lucene.apache.org > Subject: Re: Out of memory on sorting > > Explicit Warming of Sort Fields > > If you do a lot of field based sorting, it is advantageous to add explicitly > warming queries to the "newSearcher" and "firstSearcher" event listeners in > your solrconfig which sort on those fields, so the FieldCache is populated > prior to any queries being executed by your users. > firstSearcher > <lst> <str name="q">solr rocks</str><str name="start">0</str><str > name="rows">10</str><str name="sort">empID asc</str></lst> > > > > On Thu, May 19, 2011 at 2:39 PM, Rohit <ro...@in-rev.com> wrote: > >> Hi, >> >> >> >> We are moving to a multi-core Solr installation with each of the core >> having >> millions of documents, also documents would be added to the index on an >> hourly basis. Everything seems to run find and I getting the expected >> result and performance, except where sorting is concerned. >> >> >> >> I have an index size of 13217121 documents, now when I want to get >> documents >> between two dates and then sort them by ID solr goes out of memory. This >> is >> with just me using the system, we might also have simultaneous users, how >> can I improve this performance? >> >> >> >> Rohit >> >> > >
