Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
: We will take this approach in our production environment but meanwhile I am : curious if this issue will be addressed: it seems the new/first searchers do : not really buy any performance benefits because it uses so much memory, : especially at core loading time. There's nothing inheriently wrong with using newSearcher/firstSearcher -- for many people they do in fact provide a perf improvement for real users (at the cost of some initial time spent warming before those users ever get access to the searcher) As i udnerstand it from this thread, your issue is not actually the firstSearcher/newSearcher -- your issue (per yonik's ocmments) is that with per Segment sorting in 1.4, the FieldCache for some of your fields requires a lot more ram in 1.4 then it would have been for Solr 1.3 -- which caused GC thrashing during initialization. Even w/o using firstSearcher/newSearcher, all that RAM is still going to be used if/when you do sort on those fields -- all removing the firstSearcher/newSearcher queries on those fields has done for you is delay when the time spent initializing those FieldCaches happens and when that RAM first starts getting used. It's posisbly you never actual sort on those fields, in which case removing those warming queries completely is definitely the way to go -- but if you do sort on them then the warming queries can still legitimately be helpful (in thta they pay the cost up front before a real user issues queries) As yonik mentioned the real fix for the amount of memory being used is to switch to the TrieDateFields which use much more efficient FieldCache's for sorting -- with that change you can probably start using the warming queries again. (Depending on how you tested, you may not have noticed much advantage to having them because you'll really only see the advantages on the initial queries that do sorting -- those should show huge outlier times w/o the warming queries, but once those poor unlucky users have paid the price for initializing hte FieldCache, every one elses sorts should be fast) -Hoss
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
just a blind shot (didn't read the full thread): what is your maxWarmingSearchers settings? For large indices we set it to 2 (maximum) Regards, Peter. just update on this issue... we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran benchmark tests, there is no noticeable performance impact on the queries we perform comparing with Solr 1.3 benchmark tests WITH new/first searchers. Also the memory usage reduced by 5.5 GB after loading the cores with our data volume by turning these static warm caches off. We will take this approach in our production environment but meanwhile I am curious if this issue will be addressed: it seems the new/first searchers do not really buy any performance benefits because it uses so much memory, especially at core loading time. thanks Renee -- http://jetwick.com twitter search prototype
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
just update on this issue... we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran benchmark tests, there is no noticeable performance impact on the queries we perform comparing with Solr 1.3 benchmark tests WITH new/first searchers. Also the memory usage reduced by 5.5 GB after loading the cores with our data volume by turning these static warm caches off. We will take this approach in our production environment but meanwhile I am curious if this issue will be addressed: it seems the new/first searchers do not really buy any performance benefits because it uses so much memory, especially at core loading time. thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1697609.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, I tried the fix suggested in your comments (using solr.TrieDateField ), and it loaded up 130 cores in 1 minute, 1.3GB memory (a little more than 1GB when turning off static warm cache, and much less than 6.5GB when use 'solr.DateField'). Will this have any impact on first query or performance? I am about to run some benchmark test and compare with old data, will update you. Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1637176.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Could Solr just load cores one at a time, waiting for loader events to finish? Or continuously stage 2 or three simultaneously? On Sat, Oct 2, 2010 at 7:08 AM, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun renee_...@mcafee.com wrote: Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. For other's reference: here is the warming query (it's the same for newSearcher too): listener event=firstSearcher class=solr.QuerySenderListener arr name=queries lst str name=qtype:message/str str name=start0/str str name=rows10/str str name=sortmessage_date desc/str /lst /arr /listener The sort field message_date is what will be taking up the memory. Starting with Lucene 2.9 (which is used in Solr 1.4), searching and sorting is per-segment. This is generally beneficial, but in this case I believe it is causing the extra memory usage because the same date value that would have been shared across all documents in the fieldcache is now repeated in each segment it is used in. One potential fix (that requires you to reindex) is to use the date fieldType as defined in the new 1.4 schema: fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ This will use 8 bytes per document in your index, rather than 4 bytes per doc + an array of unique string-date values per index. Trunk (4.0-dev) is also much more efficient at storing string-based fields in the FieldCache - but that will only help you if you're comfortable with using development versions. -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 -- Lance Norskog goks...@gmail.com
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun renee_...@mcafee.com wrote: Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. For other's reference: here is the warming query (it's the same for newSearcher too): listener event=firstSearcher class=solr.QuerySenderListener arr name=queries lst str name=qtype:message/str str name=start0/str str name=rows10/str str name=sortmessage_date desc/str /lst /arr /listener The sort field message_date is what will be taking up the memory. Starting with Lucene 2.9 (which is used in Solr 1.4), searching and sorting is per-segment. This is generally beneficial, but in this case I believe it is causing the extra memory usage because the same date value that would have been shared across all documents in the fieldcache is now repeated in each segment it is used in. One potential fix (that requires you to reindex) is to use the date fieldType as defined in the new 1.4 schema: fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ This will use 8 bytes per document in your index, rather than 4 bytes per doc + an array of unique string-date values per index. Trunk (4.0-dev) is also much more efficient at storing string-based fields in the FieldCache - but that will only help you if you're comfortable with using development versions. -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. I assuem the consequence of commenting out the static warm requests will be it will slow down first time we hit the core for query? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
http://lucene.472066.n3.nabble.com/file/n1617135/solrconfig.xml solrconfig.xml Hi Yonik, I have uploaded our solrconfig.xml file for your reference. we also tried 1.4.1, for same index data, it took about 30-55 minutes to load up all 130 cores, it did not help at all. There is no query running when we tried to upload the cores. Since JConsole is not responding at all when this happens, I am not sure if there is any command link memory profiler I can use to collect information, any suggestions? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617135.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
VERY interesting. Looking forward to what happens on the first queries. Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Fri, 10/1/10, Renee Sun renee_...@mcafee.com wrote: From: Renee Sun renee_...@mcafee.com Subject: Re: Upgrade to Solr 1.4, very slow at start up when loading all cores To: solr-user@lucene.apache.org Date: Friday, October 1, 2010, 2:42 PM Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. I assuem the consequence of commenting out the static warm requests will be it will slow down first time we hit the core for query? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Thu, Sep 30, 2010 at 10:41 AM, Renee Sun renee_...@mcafee.com wrote: Hi - I posted this problem but no response, I guess I need to post this in the Solr-User forum. Hopefully you will help me on this. We were running Solr 1.3 for long time, with 130 cores. Just upgrade to Solr 1.4, then when we start the Solr, it took about 45 minutes. The catalina.log shows Solr is very slowly loading all the cores. Have you tried 1.4.1 yet? Could you open a JIRA issue for this and give whatever info you can? Info like: - do you have any warming queries configured? - do the cores have documents already, and if so, how many per core? - are you using the same schema solrconfig, or did you upgrade? - have you tried finding out what is taking up all the memory (or all the CPU time)? -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, thanks for your reply. I entered a bug for this at : https://issues.apache.org/jira/browse/SOLR-2138 to answer your questions here: - do you have any warming queries configured? no, all autowarmingcount are set to 0 for all caches - do the cores have documents already, and if so, how many per core? yes, 130 cores total, 2,3 of them already have 1~2.4 million documents, others have about 50,000 documents - are you using the same schema solrconfig, or did you upgrade? yes, absolutely no change - have you tried finding out what is taking up all the memory (or all the CPU time)? yes, JConsole shows after 70 cores are loaded in about 4 minutes, all 16GB memory are taken and rest of cores load extremely slow. The memory remain high and never dropped. We are in process to upgrade to 1.4.1 -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1611030.html Sent from the Solr - User mailing list archive at Nabble.com.
