ugh, yuck. If the VM memory use in task manager is correct, it is more than the JVM args are set at. Is that really possible? If so the heap size settings seem worthless, eh?
hmm, I'm using the JRun logging service, not any JVM args. This is setup in the jrun.xml file. I've read that it doesn't add noticeable load at all. Plan to turn it off once things are more stable again. Note we do not get any server crashes, we get server hangs. If I understand this all correctly, if the JVM runs out of memory D On Tue, 1 Mar 2005 14:48:54 -0000, Robertson-Ravo, Neil (RX) <[EMAIL PROTECTED]> wrote: > If Task Manager is that High it is a sure fire case that your app is using > that amount of Memory! > > Again, you will get a far clearer picture of things if you run the VisualGC > - I wouldn't trust JRun metrics at all. > > N > > -----Original Message----- > From: Douglas Knudsen [mailto:[EMAIL PROTECTED] > Sent: 01 March 2005 14:50 > To: CF-Server > Subject: Re: memory use and CF > > ok > > On Tue, 1 Mar 2005 14:11:21 -0000, Robertson-Ravo, Neil (RX) > <[EMAIL PROTECTED]> wrote: > > Hey, > > > > All of this really stems on how your application interacts with Java/JVM. > > There is no really technotes from Macromedia or otherwise which will > provide > > you with a best fit setting. Whatever else has been said ConcMarkSweep > will > > give you worse performance than a Parallel GC - for sure. However f you > are > > not sure, then simply remove any setting of the GC type and the JVM will > > take / use Parallel - if you have multiple CPU's you need to use them ;-) > > > > Good point! danke. > > > What metrics have you ran before to give you memory consumption? (DO NOT > > rely on Task manager!) > > I have the JRun metrics loggin enabled. Example entries during load > DateTime,,listenTh,delayTh,idleTh,busyTh,totalTh,delayRq,droppedRq,delayMs,T > otalMem,FreeMem,J2EESessions > 03/01 09:20:41,metrics ,3,0,6,2,11,0,0,0,655296,445742,1 > 03/01 09:21:14,metrics ,3,1,1,6,11,0,0,0,655296,487605,1 > 03/01 09:21:44,metrics ,3,0,6,3,12,0,0,0,655296,421309,1 > 03/01 09:22:17,metrics ,3,0,2,10,15,0,0,0,655296,506024,1 > 03/01 09:22:47,metrics ,2,0,10,2,14,0,0,0,655296,480495,1 > 03/01 09:23:19,metrics ,2,0,6,6,14,0,0,0,655296,537053,1 > 03/01 09:23:49,metrics ,2,0,8,4,14,0,0,0,655296,481596,1 > 03/01 09:24:21,metrics ,3,0,1,10,14,0,0,0,655296,532518,1 > 03/01 09:24:51,metrics ,5,0,9,2,16,0,0,0,655296,457535,1 > 03/01 09:25:23,metrics ,5,0,7,4,16,0,0,0,655296,528284,1 > > > > > In Task manager ensure that you have the VM column visible, What you are > > probably seeing is timedout requests or queued requests being buffered > into > > the system. Do you have a lot of Connection Reset by Peer's in the > runtime > > logs? If you do, then CF is backlogging processing of the requests which > > will consume memory. > > Doh! I need a refresher course in using task manager. JVM heap is > set to 640MB. Task mangers VM reports 850,000K and mem use is at > 754,000k. what is the discrepency? > > > > > I would certainly update your JVM as the GC in later versions is greatly > > improved. > > ok, on the task list. and thanks again for your help on this. > > > > > Thanks > > > > Neil > > > > > > -----Original Message----- > > From: Douglas Knudsen [mailto:[EMAIL PROTECTED] > > Sent: 01 March 2005 14:04 > > To: CF-Server > > Subject: Re: memory use and CF > > > > thanks for the reply Neil, very helpful. Info out there on this is > > scarce. I'm going to reply inline below. > > > > On Tue, 1 Mar 2005 10:03:52 -0000, Robertson-Ravo, Neil (RX) > > <[EMAIL PROTECTED]> wrote: > > > OK, > > > > > > Sorry I was delayed, damn timezones! OK, it looks as though you have a > > > fairly beefy machine...A few things you can do here (and these are never > > > documents by MM as they are no doubt out of scope of CF!) > > > > > > I would remove the -XX:+UseConcMarkSweepGC from your arguments and > replace > > > it with -XX:+UseParallelGC. If you have a 4 processor box then you > going > > to > > > suffer some performance problems with a ConcMarkSweep. > > > > This is interesting, Robi Sen's blog mentions the opposite and I think > > B Purcell's does too > > > http://www.robisen.com/index.cfm?mode=entry&entry=FD4BE2FC-55DC-F2B1-FED0717 > > CC1C7E0AF > > we are using the 1.4.2 JVM shipping with CFMX. Interesting. > > > > > > > > Your Heap size looks scarily low - if its too low you will begin to get > a > > > lot of GC's happening. For a 2GB box IO would set the heap to 1GB for > Min > > > and Max. One thing to note is that there is no ideal setting for the > heap > > - > > > it is entirely based on how much memory your application uses - and each > > > application is different. If you have not already I suggest you go and > > run > > > the Sun VisualGC application which is part of the JVMStat tool. > > > > > > http://java.sun.com/performance/jvmstat/ > > > > > > > I am working on getting this tool to work. I'm not a JAR head yet :) > > Hmm, I raised the heap to 1GB before and got the same issues. From > > the metrics monitoring, free heap space rarely goes below 400MB. > > Increasing the heap too big is also bad correct? Causing larger GCs? > > What I can't figure oiut is why windows task manager says the JRun > > instance is taking up more memory then it si though, see my first post > > in this thread. > > > > > Your symptoms all point to the fact that your app is a memory hogger - > > > probably due to largish queries (as MX uses around 4x more memory for a > > > query than CF5 does) or indeed just extensive use of compels procedures. > > > > > > > Yes, this is what I suspect too. One of the apps on this box is a > > heavy query laden piece of work that I have fought with teh developer > > on since inception. The developer sees no issue in a web page running > > 30 to 60 seconds if that's what the user wants. geesh. I'm > > segregating his app to a seperate instance soon. I expect some of > > these issues to go away then. In fact, I can sometimes see a large > > chunk, ~100MB, of free heap space get used at the same time this app > > has a long running request. Seconds later teh heap space is returned > > though, so it seems GC works. > > > > > I would redefined the following values (settings inline) > > -XX:MaxPermSize=64m > > > -XX:PermSize=48m -XX:NewSize=256m > > > > Ok, the MaxPermSize conflicts with this > > > www.macromedia.com/support/coldfusion/ts/documents/java_lang_outofmemory.htm > > any reason to set it lower? What's your thoughts? Setting NewSize > > this high reduces GC frequency, correct? > > > > > > > > As for your XML settings - there a few things you can do. First off, > what > > > is your Simultaneous connections set to? Within your proxy service the > > > activeHandlerThreads setting should be the same as your Simultaneous > > > connections value. And within your Scheduler service (this is sometimes > > > totally missing in CFMX) you should have > > > > > > <service class="jrunx.scheduler.SchedulerService" > name="SchedulerService"> > > > <attribute name="bindToJNDI">true</attribute> > > > <attribute name="activeHandlerThreads">20</attribute> > > > <attribute name="maxHandlerThreads">1000</attribute> > > > <attribute name="minHandlerThreads">20</attribute> > > > <attribute name="threadWaitTimeout">60</attribute> > > > <attribute name="timeout">300</attribute> > > > </service> > > > > > > Where min and activehandler threads are equal to the value set in > > > simultaneous threads. > > > > yup, have these set, I read much about this. I failed to list them > > before. I have no simultaneous thread setting in CFAdmin as we are > > using the J2EE install on JRun. > > > > > > > > Also consider that it is IIS which is causing the slowdown. We have had > > > some serious issues with IIS5 which has the IIS Lockdown tool applied. > > > Sometimes it would just hang the server - CF would look to be serving > > pages > > > OK and the IIS service is running but not pages get served....the only > way > > > to get them back (not always succcessful) is to perform an IISReset from > > the > > > command line. What to do is go download IISState tool here: > > > > > > http://www.iisfaq.com/Default.aspx?tabid=2513 > > > > > > When you are getting slow//blocked pages then run this and it will > > generate > > > a log which you can then review to see if any IIS processes are blocking > > CF. > > > You can also post it to the Microsoft Newsgroup to get an MVP input. > > > > > > > ah, thanks, I never saw this tool before. I have been looking at > > IISTracer too, a nice tool. We are thinking about buying seefusion > > too (www.seefusion.com) > > > > > I see you are using large query sets. Are you running these via CFQUERY > > or > > > in an SP? I would suggest that any of these large result sets are set > to > > be > > > dirty reads in order to stop blocking/locks. > > > > Yup, using cfquery for most everything. Kind of a pain to use an SP > > to return a result set in Oracle/CF. > > > > > > > > You may also want to consider creating separate CF Datasource > connections > > > (to the same Database) for different processes - i.e. if you have a > report > > > which runs ad-hoc and takes time you can increase performance and speed > by > > > running this in its own connection pool. > > > > > > > We have about 15 apps on this box. Each app gets its own DSN. Except > > we have 'maintain connections' off for every one at the moment, > > thinking this hanging server issue was related to JDBC drivers. > > > > > Let me know how you get on, if you need to contact me offline you can > via > > > MSN...let me know if you want it offline. > > > > > > N > > > > > > > > > -----Original Message----- > > > From: Douglas Knudsen [mailto:[EMAIL PROTECTED] > > > Sent: 28 February 2005 21:04 > > > To: CF-Server > > > Subject: Re: memory use and CF > > > > > > thankyou Neil! > > > > > > We are using win2K Server, IIS5, CFMX6.1 UPD 1, 4 Xeon P4s, and 2GB of > RAM > > > We use Oracle 9i and we use the JDBC drivers that come with CF > > > I have this as teh JVM settings, > > > java.args=-server -Xms640m -Xmx640m -Dsun.io.useCanonCaches=false > > > > > > -Xbootclasspath/a:"{application.home}/servers/cfusion/cfusion-ear/cfusion-wa > > > r/WEB-INF/cfusion/lib/webchartsJava2D.jar" > > > -XX:MaxPermSize=128m -XX:PermSize=64m -XX:+UseConcMarkSweepGC > > > -XX:NewSize=48m -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS > > > > > > I have metrics logging on, here is a sample > > > > > > DateTime,,listenTh,delayTh,idleTh,busyTh,totalTh,delayRq,droppedRq,delayMs,T > > > otalMem,FreeMem,J2EESessions > > > 02/28 08:51:11,metrics ,2,0,7,4,13,0,0,0,655296,499693,1 > > > 02/28 08:51:43,metrics ,2,0,9,2,13,0,0,0,655296,541514,1 > > > 02/28 08:52:13,metrics ,2,0,11,0,13,0,0,0,655296,511963,1 > > > 02/28 08:52:45,metrics ,11,0,5,2,18,0,0,0,655296,571183,1 > > > When the server hangs, the metrics are similar, no indication of > > > nastiness at all > > > > > > have some logging of GCs on, a sample here > > > 02/28 15:35:36,debug MM GC collected 0 messages, 0 remaining. > > > 02/28 15:43:36,debug MM GC collected 0 messages, 0 remaining. > > > 02/28 15:51:36,debug MM GC collected 0 messages, 0 remaining. > > > > > > We have disabled 'maintain connections' for all DSNs thinking its > > > related to JDBC stuffs. > > > > > > relevant jrun.xml settings > > > <service class="jrun.servlet.jrpp.JRunProxyService" name="ProxyService"> > > > <attribute name="activeHandlerThreads">25</attribute> > > > <attribute name="backlog">500</attribute> > > > <attribute name="deactivated">false</attribute> > > > <attribute name="interface">*</attribute> > > > <attribute name="maxHandlerThreads">1000</attribute> > > > <attribute name="minHandlerThreads">20</attribute> > > > <attribute name="port">51002</attribute> > > > <attribute name="threadWaitTimeout">180</attribute> > > > <attribute name="timeout">300</attribute> > > > considering teh metrics logging, there does not appear anything wrong > > > with these. > > > > > > At this poitn, the only thing I can see to do is stack/thread traces. > > > But time constraints, I'm supposed to be developing!, are leading us > > > to think of a schedule restart of the cf instance. > > > > > > We have one app that continually gets page request times in the 15 to > > > 20 second range, but the metrics logging seems ok with that. We are > > > planning on moving it to its own instance, thinking its the problem > > > child. > > > > > > hope this helps some. > > > > > > Doug > > > > > > On Mon, 28 Feb 2005 19:59:01 -0000, Robertson-Ravo, Neil (RX) > > > <[EMAIL PROTECTED]> wrote: > > > > Sorry, coming in latee on this one...what are your Java settings (Heap > > > Size, > > > > Perm Size, New etc..). What are the specs of your server? This looks > > > like > > > > a GC issue. > > > > > > > > Have you ran any extensive monitoring? What SQL backend do you have? > > What > > > > drivers? > > > > > > > > I can help you on this issue as I have had 6-8 months of memory/cf > > > debugging > > > > under my belt - believe me, I have tweaked and set thousands of values > > for > > > a > > > > decent performance setup. > > > > > > > > Give me as much info as you can on settings/configs etc..and we can > > > > hopefully resolve this. > > > > > > > > Neil > > > > Member of TMM CF > > > > > > > > > > > > -----Original Message----- > > > > From: Douglas Knudsen > > > > To: CF-Server > > > > Sent: 28/02/2005 19:10 > > > > Subject: Re: memory use and CF > > > > > > > > thinking about this....we do use CFLDAP for all authentication. I > > > > have read somewhere that this tag can be an issue. Anyone experience > > > > this? > > > > > > > > On Mon, 28 Feb 2005 13:20:27 -0500, Douglas Knudsen > > > > <[EMAIL PROTECTED]> wrote: > > > > > nope, debugging is off in our production environments, something I > > > > > speak LOUDLY about amongst my team. Thanks for the input. > > > > > > > > > > > > > > > On Mon, 28 Feb 2005 10:06:26 -0800, Cameron Childress > > > > > <[EMAIL PROTECTED]> wrote: > > > > > > I've seen several reports lately that leaving Debugging turned on > > > > will > > > > > > also cause memory leaks in CF. Is it turned on on these boxes? > > > > > > > > > > > > -Cameron > > > > > > > > > > > > On Mon, 28 Feb 2005 08:42:08 -0500, Douglas Knudsen > > > > > > <[EMAIL PROTECTED]> wrote: > > > > > > > Running CFMX6.1 UPD1 on JRun with J2EE install. > > > > > > > What is the difference between memory use as seen in Windows > Task > > > > > > > Manager and memory use in the metrics logging via JRun? I got > the > > > > JVM > > > > > > > memory set to 640m in the jvm.config file. According to metrics > > > > > > > logging I see many entries like > > > > > > > > > > > > > > TotalMem,FreeMem > > > > > > > 655296,545928 > > > > > > > > > > > > > > with the FreeMem rarely less than 400000. > > > > > > > > > > > > > > At the same time I see that according to task manager, the jrun > > > > > > > instance is using about 750,000k. This figure slowy climbs from > > > > about > > > > > > > 350,000k when the JRun service is started. When it reachs a bit > > > > over > > > > > > > 800,000k the server starts to hang. > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > -- > > > > > > > Douglas Knudsen > > > > > > > http://www.cubicleman.com > > > > > > > this is my signature, like it? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Get help! RoboHelp http://www.houseoffusion.com/banners/view.cfm?bannerid=58 Message: http://www.houseoffusion.com/lists.cfm/link=i:10:5200 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/10 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:10 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.10 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
