I have 2 ways to update the index, either I use solrj using SolrEmbeddedServer or I do it with an http query. If I do it with an http query I indeed don't stop tomcat but I have to do some operations (mainly taking instance out of the cluster) and I can't automate this process when I can automate update which is my goal that's why I'm trying to do update without having problems with garbage collection.
I have disabled cache as I don't need it. Is it running a number of old queries to re-generate the cache anyway or it is a different cache you are talking about? But I believe it still has to register a new searcher. I don't know what the impact of this is though. I guess I will go for running more tomcat running less indexes with lower JVM heap. Thank you for the link and thanks for the reply. Marc -----Original Message----- From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Sent: 12 January 2010 07:49 To: solr-user@lucene.apache.org Subject: Re: update solr index On Mon, Jan 11, 2010 at 7:42 PM, Marc Des Garets <marc.desgar...@192.com>wrote: > > I am running solr in tomcat and I have about 35 indexes (between 2 and > 80 millions documents each). Currently if I try to update few documents > from an index (let's say the one which contains 80 millions documents) > while tomcat is running and therefore receiving requests, I am getting > few very long garbage collection (about 60sec). I am running tomcat with > -Xms10g -Xmx10g -Xmn2g -XX:PermSize=256m -XX:MaxPermSize=256m. I'm using > ConcMarkSweepGC. > > I have 2 questions: > 1. Is solr doing something specific while an index is being updated like > updating something in memory which would cause the garbage collection? > Solr's caches are thrown away and a fixed number of old queries are re-executed to re-generated the cache on the new index (known as auto-warming). This happens on a commit. > > 2. Any idea how I could solve this problem? Currently I stop tomcat, > update index, start tomcat. I would like to be able to update my index > while tomcat is running. I was thinking about running more tomcat > instance with less memory for each and each running few of my indexes. > Do you think it would be the best way to go? > > If you stop tomcat, how do you update the index? Are you running a multi-core setup? Perhaps it is better to split up the indexes among multiple boxes. Also, you should probably lower the JVM heap so that the full GC pause doesn't make your index unavailable for such a long time. Also see http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles /Scaling-Lucene-and-Solr -- Regards, Shalin Shekhar Mangar. ---------------------------------------------------------- This transmission is strictly confidential, possibly legally privileged, and intended solely for the addressee. Any views or opinions expressed within it are those of the author and do not necessarily represent those of 192.com, i-CD Publishing (UK) Ltd or any of it's subsidiary companies. If you are not the intended recipient then you must not disclose, copy or take any action in reliance of this transmission. If you have received this transmission in error, please notify the sender as soon as possible. No employee or agent is authorised to conclude any binding agreement on behalf of i-CD Publishing (UK) Ltd with another party by email without express written confirmation by an authorised employee of the Company. http://www.192.com (Tel: 08000 192 192). i-CD Publishing (UK) Ltd is incorporated in England and Wales, company number 3148549, VAT No. GB 673128728.
