I'm on a development box currently and production servers will be bigger, but at the same time the index will be too.
Each query requests at most 20 stored fields. Why doesn't help lazyfieldloading in this situation? I don't need to retrieve all stored fields and I thought I wasn't doing this (through limiting the fields returned using the FL-param), but if I read your comment correctly, apparently I am retrieving them all, I'm just not displaying them all? Also, if I understand correctly, for optimal performance I need to have at least enough RAM to put the entire Index size in OS cache (thus RAM) + the amount of RAM that SOLR / Lucene consumes directly through the JVM? (which among other things includes the Lucene field-cache + all of SOlr's caches on top of that). I've never read the requirement of having the entire index in OS cache before, is this because in normal situations (with less stored fields) it doesn't matter much? I'm just surprised to hear of this for the first time, since it will likely give a big impact on my design. Luckily most of the normal queries return 10 documents each, which results in a discrepancy between total elapsed time and qTIme of about 15-30 ms. Doesn't this seem strange, since to me it would seem logical that the discrepancy would be at least 1/10th of fetching 100 documents. hmm, hope you can shine some light on this, Thanks a lot, Britske Yonik Seeley wrote: > > That's a bit too tight to have *all* of the index cached...your best > bet is to go to 4GB+, or figure out a way not to have to retrieve so > many stored fields. > > -Yonik > > On Mon, Jul 28, 2008 at 4:27 PM, Britske <[EMAIL PROTECTED]> wrote: >> >> Size on disk is 1.84 GB (of which 1.3 GB sits in FDT files if that >> matters) >> Physical RAM is 2 GB with -Xmx800M set to Solr. >> >> >> Yonik Seeley wrote: >>> >>> That high of a difference is due to the part of the index containing >>> these particular stored fields not being in OS cache. What's the size >>> on disk of your index compared to your physical RAM? >>> >>> -Yonik >>> >>> On Mon, Jul 28, 2008 at 4:10 PM, Britske <[EMAIL PROTECTED]> wrote: >>>> >>>> Hi all, >>>> >>>> For some queries I need to return a lot of rows at once (say 100). >>>> When performing these queries I notice a big difference between qTime >>>> (which >>>> is mostly in the 15-30 ms range due to caching) and total time taken to >>>> return the response (measured through SolrJ's elapsedTime), which takes >>>> between 500-1600 ms. >>>> >>>> For queries which return less rows the difference becomes less big. >>>> >>>> I presume (after reading some threads in the past) that this is due to >>>> solr >>>> constructing and streaming the response (which includes retrieving the >>>> stored fields) , which is something that is not calculated in qTime. >>>> >>>> Documents have a lot of stored fields (more than 10.000), but at any >>>> given >>>> query a maximum of say 20 are returned (through fl-field ) or used (as >>>> part >>>> of filtering, faceting, sorting) >>>> >>>> I would have thought that enabling enableLazyFieldLoading for this >>>> situation >>>> would mean a lot, since so many stored fields can be skipped, but I >>>> notice >>>> no real difference in measuring total elapsed time (or qTime for that >>>> matter). >>>> >>>> Am I missing something here? What criteria would need to be met for a >>>> field >>>> to not be loaded for instance? Should I see a big performance boost in >>>> this >>>> situation? >>>> >>>> Thanks, >>>> Britske >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698590.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698909.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699550.html Sent from the Solr - User mailing list archive at Nabble.com.
