[Dhis2-users] Fwd: Improving Tomcat memory parameters
This discussion can be of interest to DHIS2 administrators as well. -- Forwarded message -- From: James Arbaugh Date: Sun, Apr 14, 2013 at 10:25 PM Subject: RE: Improving Tomcat memory parameters To: implement...@openmrs.org Hi Ellen, ** ** We ran into those painful problems in the past too. One thing we did was added a crontab job to restart tomcat automatically during the night twice a week as “preventative” maintenance. ** ** FYI: our production server (with 46GB physical memory) runs stable with the following tomcat configuration… CATALINA_OPTS="-Xms10240m -XX:PermSize=6144m -XX:MaxPermSize=6144m -XX:NewSize=4096m" ** ** Thanks, James ** ** *From:* implement...@openmrs.org [mailto:implement...@openmrs.org] *On Behalf Of *Ellen Ball *Sent:* Thursday, April 11, 2013 5:33 PM *To:* implement...@openmrs.org *Subject:* Improving Tomcat memory parameters *Importance:* Low ** ** We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3 production server with this memory error: "Java.lang.OutOfMemoryError: PermGen" - The server has 32GB of RAM. - tomcat is using JAVA_OPTS = "-Xmx512m -Xms512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=128m" Has anyone else seen this error? Has anyone experimented with increased memory parameter values? It seems that the Xmx and Xms values should be the same for a server ("In a server environment, you normally want Xms and Xmx set to the same value to avoid heap thrashing."). The heap space shouldn't be too high, but if anyone know the optimal size that would be helpful. ** ** Thanks, ** ** Ellen Ball Partners In Health ** ** -- OpenMRS Implementers: http://go.openmrs.org/implementers Post: implement...@openmrs.org Unsubscribe: implementers+unsubscr...@openmrs.org Manage your OpenMRS subscriptions at https://id.openmrs.org/ -- OpenMRS Implementers: http://go.openmrs.org/implementers Post: implement...@openmrs.org Unsubscribe: implementers+unsubscr...@openmrs.org Manage your OpenMRS subscriptions at https://id.openmrs.org/ -- Knut Staring Dept. of Informatics, University of Oslo +4791880522 http://dhis2.org ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp
[Dhis2-users] Fwd: Improving Tomcat memory parameters
This discussion can be of interest to DHIS2 administrators as well. -- Forwarded message -- From: Saptarshi Purkayastha Date: Fri, Apr 12, 2013 at 11:25 PM Subject: Re: Improving Tomcat memory parameters To: Mark Goodrich Cc: implement...@openmrs.org, Rowan Seymour , augustine12th thomas , Gilbert Tuwei < giltuwei2...@gmail.com> If you are running dual processor system, definitely use CMS. If its a single processor with quad-core still use it. I've seen people set one core affinity to tomcat/java process, but I would recommend that in experimental cases. If its a dual-core or worse, one of those fake-extra thread Intel processors (I forget their name), don't enable CMS. yes, I meant by default... It is also not meant to be compacting GC and hence it's time to GC is unpredictable The problem with jdk7 at the moment in OpenMRS is only compiler related. The runtime issues with the infamous PorterStemmer bug or the Lucene/Solr bugs were with the first couple of releases. Since, then I haven't seen any in-the-face stability problems. Performance gains are quite good. If the benchmarksgame@alioth wouldn't have removed java6, I could have shown some evidence :-/ --- Regards, Saptarshi PURKAYASTHA My Tech Blog: http://sunnytalkstech.blogspot.com You Live by CHOICE, Not by CHANCE On 12 April 2013 23:53, Mark Goodrich wrote: > A couple follow-on questions... (forgive my ignorance! :) > > How many cores would you recommend for using CMS? > > You said that CMS does not do class unloading... but yet the parameter I > saw was "CMSClassUnloadingEnabled". Do you mean that CMS doesn't do > class unloading by default? > > I'll talk with others about moving up to jdk7, but I'm not sure if we want > to risk the potential instability at this point... > > Take care, > Mark > > > On 04/12/2013 05:30 PM, Saptarshi Purkayastha wrote: > > Hi Mark, > > If you are using CMS for the JVM, you should ensure that your processor > has enough cores, which I expect most modern servers would. > But please check... or else your system would hang which GC goes on. > > I'll be a little scared to enable PermGenSweeping on OpenMRS or for that > matter any system that uses Spring, lazy loading and localization. > From my use of profiling OpenMRS core, PermGen creates a sawtooth graph if > GC is enabled on PermGen. Unless you are really constrained on RAM, please > do not enable it. > > On the other hand, class unloading might work much better. This is > especially true for scripting languages that run on JVM. Rhino for sure > speeds up quite a bit with CMS class unloading enabled. I've not tried > JRuby or JPython, but I've read somewhere that these languages are the > biggest gainers from G1 style compacting. CMS is not a compacting garbage > collector and thus, doesn't do class unloading. > > So I'd recommend that you try enabling it and running YourKit or another > profiler. > Simplest I'd suggest using jdk7_update 4 and higher, which enables G1 by > default > > --- > Regards, > Saptarshi PURKAYASTHA > > My Tech Blog: http://sunnytalkstech.blogspot.com > You Live by CHOICE, Not by CHANCE > > > On 12 April 2013 16:57, Jeremy Keiper wrote: > >> AMPATH's production machine uses these values, and we have fought with >> the PermGen errors before: >> >> *-Xmx3584M -Xms512M -XX:PermSize=512M -XX:MaxPermSize=512M >> -XX:NewSize=256M* >> >> We also turned on *-XX:+UseConcMarkSweepGC * >> >> Right now we are fighting with CPU load, and have not explored the >> options Mark just mentioned. >> >> >> >> Jeremy Keiper >> OpenMRS Core Developer >> AMPATH / IU-Kenya Support >> >> >> On Fri, Apr 12, 2013 at 10:10 AM, Mark Goodrich wrote: >> >>> Saptarshi, >>> >>> Thanks so much! This is very helpful... >>> >>> +1 to Rowan's suggestion of updating the wiki... if you don't have time, >>> let me know, and I can at least paste in what you've written here. >>> >>> This is a good page as well: >>> >>> https://wiki.openmrs.org/display/docs/Performance+Tuning >>> >>> Do you have any experience with these parameters? >>> >>> -XX:+CMSPermGenSweepingEnabled >>> -XX:+CMSClassUnloadingEnabled** >>> >>> >>> It looks like we are experiences a classloading leak due to the user of >>> Groovy in the UI framework... I checked yesterday and there were 4,000+ >>> dead classloaders on our production most of which were Groovy >>> classloaders. Darius put a tweak into the UI Framework last night, but I >>> don't think it has resolved it. >>> >>> Mark >>> >>> >>> >>> On 04/12/2013 04:09 AM, Rowan Seymour wrote: >>> >>> @Saptarshi maybe you could update >>> https://wiki.openmrs.org/display/docs/Troubleshooting+Memory+Errors >>> >>> I think every implementation runs into these kind of problems >>> eventually and just googling for answers gives all sorts of conflicting >>> advice. >>> >>> >>> On 12 April 2013 00:21, Saptarshi Purkayastha wrote: >>> To add to what I wrote earlier, so that this is not considered Voodoo knowledge, lemme q
Re: [Dhis2-users] Fwd: Improving Tomcat memory parameters
Great!!! Needless to say on a 64bit machine. And are you usinge an enterprise kernel (from Red Hat Enterprise or CentOS one)? Thank you Caveman On Mon, Apr 15, 2013 at 9:38 AM, Knut Staring wrote: > This discussion can be of interest to DHIS2 administrators as well. > > -- Forwarded message -- > From: James Arbaugh > Date: Sun, Apr 14, 2013 at 10:25 PM > Subject: RE: Improving Tomcat memory parameters > To: implement...@openmrs.org > > > Hi Ellen, > > ** ** > > We ran into those painful problems in the past too. One thing we did was > added a crontab job to restart tomcat automatically during the night twice > a week as “preventative” maintenance. > > ** ** > > FYI: our production server (with 46GB physical memory) runs stable with > the following tomcat configuration… > > CATALINA_OPTS="-Xms10240m -XX:PermSize=6144m -XX:MaxPermSize=6144m > -XX:NewSize=4096m" > > ** ** > > Thanks, > > James > > ** ** > > *From:* implement...@openmrs.org [mailto:implement...@openmrs.org] *On > Behalf Of *Ellen Ball > *Sent:* Thursday, April 11, 2013 5:33 PM > *To:* implement...@openmrs.org > *Subject:* Improving Tomcat memory parameters > *Importance:* Low > > ** ** > > We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3 > production server with this memory error: "Java.lang.OutOfMemoryError: > PermGen" > >- The server has 32GB of RAM. >- tomcat is using JAVA_OPTS = "-Xmx512m -Xms512m -XX:PermSize=256m >-XX:MaxPermSize=256m -XX:NewSize=128m" > > Has anyone else seen this error? Has anyone experimented with increased > memory parameter values? It seems that the Xmx and Xms values should be > the same for a server ("In a server environment, you normally want Xms > and Xmx set to the same value to avoid heap thrashing."). The heap space > shouldn't be too high, but if anyone know the optimal size that would be > helpful. > > ** ** > > Thanks, > > ** ** > > Ellen Ball > > Partners In Health > > ** ** > > > > -- > OpenMRS Implementers: http://go.openmrs.org/implementers > Post: implement...@openmrs.org > Unsubscribe: implementers+unsubscr...@openmrs.org > Manage your OpenMRS subscriptions at https://id.openmrs.org/ > > > > -- > OpenMRS Implementers: http://go.openmrs.org/implementers > Post: implement...@openmrs.org > Unsubscribe: implementers+unsubscr...@openmrs.org > Manage your OpenMRS subscriptions at https://id.openmrs.org/ > > > > > > -- > Knut Staring > Dept. of Informatics, University of Oslo > +4791880522 > http://dhis2.org > > ___ > Mailing list: https://launchpad.net/~dhis2-users > Post to : dhis2-users@lists.launchpad.net > Unsubscribe : https://launchpad.net/~dhis2-users > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp
Re: [Dhis2-users] Fwd: Improving Tomcat memory parameters
Hi, We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3 production server with this memory error: "Java.lang.OutOfMemoryError: PermGen" · The server has 32GB of RAM. · tomcat is using JAVA_OPTS = "-Xmx512m -Xms512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=128m" The JVM has a lot of optimization and intelligence for dynamic memory allocation, so doing things like making -Xms equal to -Xmx or-XX:PermSize equal to -XX:MaxPermSize will remove some of the optimizations the JVM can apply to its allocation strategies and therefor reduce your application performance instead of improving it. Note that these parameters always depend on your application. So the best way is to monitor memory consumption of your app to perform some tuning. Jconsole can be your best friend. <http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.h tml> http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.ht ml Regards, <mailto:*%20your-email%20*> Paulo Grácio Technical Manager (+258) 822 544 603 <http://www.criticalsoftware.co.uk> Critical Software Mozambique Dependable Technologies for Critical Systems Critical Software Mozambique is a subsidiary of <http://cmmiinstitute.com/> Critical Software, a CMMI® Level 5 rated Company CMMI® is registered in the USPTO by <http://www.cmu.edu/> CMU Rua Pereira Marinho, 179 Bairro de Sommerchield Maputo Moçambique Phone: (+258) 214 951 45 Fax: (+258) 214 951 46 DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system. From: Dhis2-users [mailto:dhis2-users-bounces+pgracio=criticalsoftware@lists.launchpad.net ] On Behalf Of Orvalho Augusto Sent: segunda-feira, 15 de Abril de 2013 09:48 To: Knut Staring Cc: dhis2-users@lists.launchpad.net Subject: Re: [Dhis2-users] Fwd: Improving Tomcat memory parameters Great!!! Needless to say on a 64bit machine. And are you usinge an enterprise kernel (from Red Hat Enterprise or CentOS one)? Thank you Caveman On Mon, Apr 15, 2013 at 9:38 AM, Knut Staring wrote: This discussion can be of interest to DHIS2 administrators as well. -- Forwarded message -- From: James Arbaugh Date: Sun, Apr 14, 2013 at 10:25 PM Subject: RE: Improving Tomcat memory parameters To: implement...@openmrs.org Hi Ellen, We ran into those painful problems in the past too. One thing we did was added a crontab job to restart tomcat automatically during the night twice a week as preventative maintenance. FYI: our production server (with 46GB physical memory) runs stable with the following tomcat configuration CATALINA_OPTS="-Xms10240m -XX:PermSize=6144m -XX:MaxPermSize=6144m -XX:NewSize=4096m" Thanks, James From: implement...@openmrs.org [mailto:implement...@openmrs.org] On Behalf Of Ellen Ball Sent: Thursday, April 11, 2013 5:33 PM To: implement...@openmrs.org Subject: Improving Tomcat memory parameters Importance: Low We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3 production server with this memory error: "Java.lang.OutOfMemoryError: PermGen" * The server has 32GB of RAM. * tomcat is using JAVA_OPTS = "-Xmx512m -Xms512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=128m" Has anyone else seen this error? Has anyone experimented with increased memory parameter values? It seems that the Xmx and Xms values should be the same for a server ("In a server environment, you normally want Xms and Xmx set to the same value to avoid heap thrashing."). The heap space shouldn't be too high, but if anyone know the optimal size that would be helpful. Thanks, Ellen Ball Partners In Health -- OpenMRS Implementers: http://go.openmrs.org/implementers Post: implement...@openmrs.org Unsubscribe: implementers+unsubscr...@openmrs.org Manage your OpenMRS subscriptions at https://id.openmrs.org/ -- OpenMRS Implementers: http://go.openmrs.org/implementers Post: implement...@openmrs.org Unsubscribe: implementers+unsubscr...@openmrs.org <mailto:implementers%2bunsubscr...@openmrs.org> Manage your OpenMRS subscriptions at https://id.openmrs.org/ -- Knut Staring Dept. of Informatics, University of Oslo +4791880522 http://dhis2.org ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help