Your autocommit settings are still pretty aggressive causing very frequent commits, and that is using your CPU. Yes, splitting the servers into a master and slaves tends to be the performant/scalable way to go. There is no real downside to replication, really, just a bit of network traffic.
Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: kalidoss <kalidoss.muthuramalin...@sifycorp.com> > To: solr-user@lucene.apache.org > Sent: Wed, November 18, 2009 2:25:05 AM > Subject: Re: Solr - Load Increasing. > > There seems to be some improvement. The writes speeds are faster. Server > restarts are lower. > > We changed the configuration to: > 50 > 10000 > > Before the Change: > - Server Restarts: 10 times in 12 hours > - CPU load: Average:50 and Peak:90 > > After the Change: > - Server Restarts: 4 times in 12 hours. > - CPU load: Average:30 and Peak:~70 > > Our every day writes are around 60k and reads are around 1 million. > > We are now changing the MaxDocs to 300 and MaxTime will be 10000 ms and > hoping to some more improvements. > > The system configuration is 4GB RAM and 4 core x 2 CPUs. We start the > solr (1.3) like this: java -Xms512M -Xmx1024M -jar start.jar > > Is there any other way we can reduce the high CPU load in the system? > > Do you guys think that upgrading to 1.4 and having the replication in > place with reads and writes split into separate solrs will help? How > efficient will the replication be with above mentioned scenarios? Is > there any place we can look at for info on the disadvantages of > replication... > > Please help. > Kalidoss.m, > Tom Alt wrote: > > Nice to learn a new word for the day! > > > > But to answer your question, or at least part of it, I don't really think > > you want a configuration like > > > > > > 1 > > 10 > > > > > > Committing every doc, and every 10 milliseconds? That's just asking for > > problems. How about starting with 1000 docs, and five minutes for maxTime > > (5*60*1000) or about 3 laks of milliseconds. > > > > That should help performance a lot. Try that, and see how it works. > > > > Tom > > > > On Mon, Nov 16, 2009 at 2:43 PM, Shashi Kant wrote: > > > > > >> I think it would be useful for members of this list to realize that not > >> everyone uses the same metrology and terms. > >> > >> It is very easy for "Americans" to use the imperial system and presume > >> everyone does the same; Europeans to use the metric system etc. Hopefully > >> members on this list would be persuaded to use or at least clarify their > >> terminology. > >> > >> While the apocryphal saying goes " the great thing about standards is they > >> are so many choose from", we should all make an effort to communicate > >> across > >> cultures and nations. > >> > >> > >> > >> On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo wrote: > >> > >> > >>> On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood > >>> > >>>> wrote: > >>>> > >>>> Probably "lakh": 100,000. > >>>> > >>>> So, 900k qpd and 3M docs. > >>>> > >>>> http://en.wikipedia.org/wiki/Lakh > >>>> > >>>> wunder > >>>> > >>>> On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote: > >>>> > >>>> > >>>>> Hi, > >>>>> > >>>>> Your autoCommit settings are very aggressive. I'm guessing that's > >>>>> > >>> what's > >>> > >>>> causing the CPU load. > >>>> > >>>>> btw. what is "laks"? > >>>>> > >>>>> Otis > >>>>> -- > >>>>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls > >>>>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR > >>>>> > >>>>> > >>>>> > >>>>> ----- Original Message ---- > >>>>> > >>>>>> From: kalidoss > >>>>>> To: solr-user@lucene.apache.org > >>>>>> Sent: Mon, November 16, 2009 9:11:21 AM > >>>>>> Subject: Solr - Load Increasing. > >>>>>> > >>>>>> Hi All. > >>>>>> > >>>>>> My server solr box cpu utilization increasing b/w 60 to 90% and > >>>>>> > >>> some > >>> > >>>> time > >>>> > >>>>>> solr is getting down and we are restarting it manually. > >>>>>> > >>>>>> No of documents in solr 30 laks. > >>>>>> No of add/update requrest solr 30 thousand / day. Avg of every 30 > >>>>>> > >>>> minutes > >>>> > >>>>>> around 500 writes. > >>>>>> No of search request 9laks / day. > >>>>>> Size of the data directory: 4gb. > >>>>>> > >>>>>> > >>>>>> My system ram is 8gb. > >>>>>> System available space 12gb. > >>>>>> processor Family: Pentium Pro > >>>>>> > >>>>>> Our solr data size can be increase in number like 90 laks. and > >>>>>> > >>> writes > >>> > >>>> per day > >>>> > >>>>>> will be around 1laks. - Hope its possible by solr. > >>>>>> > >>>>>> For write commit i have configured like > >>>>>> > >>>>>> 1 > >>>>>> 10 > >>>>>> > >>>>>> > >>>>>> Is all above can be possible? 90laks datas and 1laks per day > >>>>>> > >> writes > >> > >>>> and > >>>> > >>>>>> 30laks per day read?? - if yes what type of system configuration > >>>>>> > >>> would > >>> > >>>> require. > >>>> > >>>>>> Please suggest us. > >>>>>> > >>>>>> thanks, > >>>>>> Kalidoss.m, > >>>>>> > >>>>>> > >>>>>> Get your world in your inbox! > >>>>>> > >>>>>> Mail, widgets, documents, spreadsheets, organizer and much more with > >>>>>> > >>>> your > >>>> > >>>>>> Sifymail WIYI id! > >>>>>> Log on to http://www.sify.com > >>>>>> > >>>>>> ********** DISCLAIMER ********** > >>>>>> Information contained and transmitted by this E-MAIL is proprietary > >>>>>> > >> to > >> > >>>> Sify > >>>> > >>>>>> Limited and is intended for use only by the individual or entity to > >>>>>> > >>>> which it is > >>>> > >>>>>> addressed, and may contain information that is privileged, > >>>>>> > >>> confidential > >>> > >>>> or > >>>> > >>>>>> exempt from disclosure under applicable law. If this is a forwarded > >>>>>> > >>>> message, the > >>>> > >>>>>> content of this E-MAIL may not have been sent with the authority of > >>>>>> > >>> the > >>> > >>>> Company. > >>>> > >>>>>> If you are not the intended recipient, an agent of the intended > >>>>>> > >>>> recipient or a > >>>> > >>>>>> person responsible for delivering the information to the named > >>>>>> > >>>> recipient, you > >>>> > >>>>>> are notified that any use, distribution, transmission, printing, > >>>>>> > >>> copying > >>> > >>>> or > >>>> > >>>>>> dissemination of this information in any way or in any manner is > >>>>>> > >>>> strictly > >>>> > >>>>>> prohibited. If you have received this communication in error, please > >>>>>> > >>>> delete this > >>>> > >>>>>> mail & notify us immediately at ad...@sifycorp.com > >>>>>> > >>>> > >>> Thanks Walter for clarifying that. > >>> > >>> I too was wondering what "laks" meant. > >>> > >>> It was a bit distracting when I read the original post. > >>> -- > >>> "Good Enough" is not good enough. > >>> To give anything less than your best is to sacrifice the gift. > >>> Quality First. Measure Twice. Cut Once. > >>> > >>> > > > >
