Re: ColdFusion|JRun CPU and|or memory problem.
On 1/21/2010 8:37 PM, b...@bradwood.com wrote: > >> I've been use the Trigger Snapshot function from the ColdFusion Monitor >> and I have been getting threads, but I am not, in my limited experience >> reading threads, I can see anything. In fact every thread dump I have >> done I have not seen a single thread processing and cfm or cfc code >> except for the on running the monitor cfc I used to get the thread dump. >> > - > > Sorry, I'm getting to these a little out of order... there's so much > static on the list today it's taking me a while to church through it > all. :) > > If you don't see any .cfm or .cfc files running then the answer it > probably quite simply that nothing is running. The big question is > whether or not you took those stack traces WHILE the CPU was spinning or > during normal operation. If the latter, than you probably just need to > wait until it starts spinning again and then capture some traces. > If you DID take the traces while the CPU was spinning then look > carefully again an any thread that isn't sleeping. There could be a > background process like client var purge, or mail spooler running that > would show as an executing thread but wouldn't necessarily involve any > .cfm or .cfc files. If the CPU is spinning on the Jrun process and NONE > of the threads are doing ANYTHING, then my next place to check out would > be stop-the-world Garbage Collection. > > I guess I'll let you answer those before we dive into that since I know > you've been getting a lot of advice and following all of it at once can > be a little crazy. > > ~Brad Yes, the snap shots where taken when the CPU was not spinning. After nearly two weeks of the CPU spinning every day, it never once did it yesterday while I was prepared to monitor it. And since today is a Furlough day, I won't be back on this until Monday. I appreciate all the advice, they have put several new tools into my box. Now I just need to get proficient at using them. Look forward to follow ups next week. Ian ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:330019 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
> I've been use the Trigger Snapshot function from the ColdFusion Monitor > and I have been getting threads, but I am not, in my limited experience > reading threads, I can see anything. In fact every thread dump I have > done I have not seen a single thread processing and cfm or cfc code > except for the on running the monitor cfc I used to get the thread dump. - Sorry, I'm getting to these a little out of order... there's so much static on the list today it's taking me a while to church through it all. :) If you don't see any .cfm or .cfc files running then the answer it probably quite simply that nothing is running. The big question is whether or not you took those stack traces WHILE the CPU was spinning or during normal operation. If the latter, than you probably just need to wait until it starts spinning again and then capture some traces. If you DID take the traces while the CPU was spinning then look carefully again an any thread that isn't sleeping. There could be a background process like client var purge, or mail spooler running that would show as an executing thread but wouldn't necessarily involve any .cfm or .cfc files. If the CPU is spinning on the Jrun process and NONE of the threads are doing ANYTHING, then my next place to check out would be stop-the-world Garbage Collection. I guess I'll let you answer those before we dive into that since I know you've been getting a lot of advice and following all of it at once can be a little crazy. ~Brad ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:330001 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
> What are my options here? Is there an easier way to get to this thread > information? JConsole is _not_ necessary to get a thread dump of your JVM. All you need for that is the snapshot tab in ColdFusion's built-in server monitor. JConsole will help you track your memory usage. If it were me, I would make the thread dumps your first priority and worry about JConsole if you can't identify what is running from the stack trace and/or you determine that memory usage has in fact been a-typical. Getting a stack trace is trivial, and JConsole might be a bit of a red herring right now for you. I'm not saying you shouldn't get JConsole working, but I personally would make it a second priority to analyzing some stack traces taken WHILE the CPU is spinning. ~Brad ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:33 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/21/2010 2:41 PM, b...@bradwood.com wrote: > In general, you need to add some GC args to take a look at your garbage > collections and how often they are running as well as how much memory > you generally use. (jConsole will also be a good step) An advice on what it takes to set up jConsole? When I tried, I got the error I posted earlier that indicate to me that the required class(es) are not available for the jConsole hook on the server. Can I use jConsole with the out of the box JRun4 box. If not what do I need to do to use it? I've been use the Trigger Snapshot function from the ColdFusion Monitor and I have been getting threads, but I am not, in my limited experience reading threads, I can see anything. In fact every thread dump I have done I have not seen a single thread processing and cfm or cfc code except for the on running the monitor cfc I used to get the thread dump. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329991 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/21/2010 2:56 PM, Maureen wrote: > Did anything change with the configuration of the server on January 5? > Java update, maybe? > Only one application update, and no mater what we do with that application we can not trigger the condition. > Most of the time I've found that problems like this are related to > either a large data query that is hanging or timing out, and not > notifying the Coldfusion server. Or a program that has an infinite > loop - a cfc that is calling itself, perhaps. > That's our thought as well, we just can't any hint on what code might be causing this. That is why I am trying to learn how to do and read these thread dumps, so that maybe I can get a pointer to some suspect code. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329989 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
Did anything change with the configuration of the server on January 5? Java update, maybe? Most of the time I've found that problems like this are related to either a large data query that is hanging or timing out, and not notifying the Coldfusion server. Or a program that has an infinite loop - a cfc that is calling itself, perhaps. Are you seeing anything unusual in the logs when this happening? On Wed, Jan 20, 2010 at 1:37 PM, Ian Skinner wrote: > > Any good information on how to track down what is causing a ColdFusion > server to behave badly. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329987 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
Questions like that hard to answer generically.
In general, you need to add some GC args to take a look at your garbage
collections and how often they are running as well as how much memory
you generally use. (jConsole will also be a good step) Chances are, if
your server has enough RAM and CF is the only thing running, you can
probably safely increase your Xmx. There are many good blogs out there
by people who know more than I do. Adobe also has good info on the
subject:
http://www.adobe.com/devnet/coldfusion/articles/coldfusion_performance_02.html
The perm generation is where classes are stored. If your app is heavy
on CFCs, you might need more. Generally, if that really is problem,
you'll start seeing out of perm space errors.
~Brad
--
For reference here is the arguments from out development server which
should be the same on our production server
# Arguments to VM
java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=192m
-XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/
Are there any concerns or performance problems having this argument
configured on a production server?
~|
Want to reach the ColdFusion community with something they want? Let them know
on the House of Fusion mailing lists
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329985
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
> The difficult part of my job is that I administer the ColdFusion server, but > I am not allowed direct access to the server. Yeah, those sort of setups suck because you can't just get in and do what you need to. The problem with a couple isolated readings of memory usage without a reference point is you really have no clue if that amount of memory usage is normal or not for your app. No one here can tell you that because every app is different. Some people probably have servers that never see more than 400 Megs of memory assigned to Jrun, other people here probably are used to each serer consuming 2 Gigs per instance on a regular basis. I monitor and log my memory usage (With SeeFusion) so I can get average values and trending and I know my servers sit at about 700 Megs to 1 Gig depending on traffic. You pretty much need to have baseline statistics to know if a given value is higher or lower than the normal for your app. Anything else is just guessing. You also never told me what your -Xms and -Xmx JVM args were. The reason I asked is because some people (like myself) set them equal to the same number (Though it appears you don't). This means that the amount of memory that jrun gets allocated by the OS is immediately granted (and shows up in Windows process monitor as being "used" by jrun) even though the heap may only be partially used. At that point, looking at the Windows process list will do you no good since it only shows you how much memory the OS has reserved for CF, not how much is being used. The server monitor will show you how much memory is in use without turning on memory tracking. ~Brad ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329981 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: ColdFusion|JRun CPU and|or memory problem.
> Well, that is probably the better solution, but for my first experiment, > I just planned to restart one instance, so I presume only that instance > would get the new settings. At least until such a time as the whole > server was restarted or something. Yes, that's correct. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329961 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/21/2010 11:00 AM, Ian Skinner wrote: > So if I am understanding your comment and the documentation I just need > to add this argument to the VM Arguments line in the jvm.config file in > our E:\JRun4\bin folder. > Well, I tried to add the JConsole hook to the JVM and I got the following error. java.lang.NoClassDefFoundError: com/sun/management/jmxremote/port=13013 I presume that is telling me that this class is not available in the JRun flavor of Java that we are running. What are my options here? Is there an easier way to get to this thread information? If not I presume I need to download and install a Sun Java SDK to get access to the require class. Does the ColdFusion instance need to be running on the Sun SDK, or just the SDK just need to be on the box? I'm am finding it very frustrating to try and get some information about what is suddenly causing this server to spin the CPU so hard. P.S. I may not have communicated yet, that I tried the information found in TechNote 18339 Debugging Thread Dumps and Server Problems in ColdFusion MX 6.1 and 7.0. http://kb2.adobe.com/cps/183/tn_18339.html. Unfortunatly, when we followed the instructions to configure the Windows service to "Allow service to interact with Desktop", we saw that this only exists if the service runs as a LocalSystem user. We don't run our ColdFusion instance as a LocalSystem user, we have it configured to run under a domain account we have created so that our ColdFusion code and access other network resources outside the local system. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329960 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/21/2010 11:24 AM, Dave Watts wrote: > Yes, changing jvm.config will affect all instances by default. I > recommend that you configure the specific instance in question to use > a different jvm.config, which is very easy to do, and then just change > that jvm.config. > > http://kb2.adobe.com/cps/182/tn_18206.html > Well, that is probably the better solution, but for my first experiment, I just planned to restart one instance, so I presume only that instance would get the new settings. At least until such a time as the whole server was restarted or something. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329959 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
> Are there in considerations to this because we are running ColdFusion in > a multi-home configuration? Is putting this argument in that one > configuration file going to effect all the JRun instances? Once each > one is restarted of course. We are using a default multi-home > configuration, we have taken no steps to have different ColdFusion|JRun > instances using separate configuration files of which I am aware. Yes, changing jvm.config will affect all instances by default. I recommend that you configure the specific instance in question to use a different jvm.config, which is very easy to do, and then just change that jvm.config. http://kb2.adobe.com/cps/182/tn_18206.html Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsi ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329956 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/20/2010 3:15 PM, Dominic Watson wrote:
> Something else to look at is jConsole. It comes with the Java SDK and will
> let you look at any java app's memory, thread and cpu usage in detail
> (you'll need to change jrun's JVM args to have it report those stats).
>
So if I am understanding your comment and the documentation I just need
to add this argument to the VM Arguments line in the jvm.config file in
our E:\JRun4\bin folder.
Are there in considerations to this because we are running ColdFusion in
a multi-home configuration? Is putting this argument in that one
configuration file going to effect all the JRun instances? Once each
one is restarted of course. We are using a default multi-home
configuration, we have taken no steps to have different ColdFusion|JRun
instances using separate configuration files of which I am aware.
For reference here is the arguments from out development server which
should be the same on our production server
# Arguments to VM
java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=192m
-XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/
Are there any concerns or performance problems having this argument
configured on a production server?
~|
Want to reach the ColdFusion community with something they want? Let them know
on the House of Fusion mailing lists
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329952
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/20/2010 4:08 PM, b...@bradwood.com wrote: > I never got a clear answer from you on whether or not > you are seeing memory jumps (real ones, not the petabytes that the CF > server mon was showing you) or if your main problem was just CPU usage. > I don't know about a "Jump" yet, but when we look at the task manager that main JRun instance is currently consuming over 600,000K. When it is restarted, the JRun starts up with ~50,000K I believe. The difficult part of my job is that I administer the ColdFusion server, but I am not allowed direct access to the server. So I have to ask others for information like this. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329919 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
I'll second that on the memory tracking - I wish I could disable the button so an unwary admin doesn't accidentaly crash the server for me (yikes). Mark A. Kruger, CFG, MCSE (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: b...@bradwood.com [mailto:b...@bradwood.com] Sent: Wednesday, January 20, 2010 5:04 PM To: cf-talk Subject: RE: ColdFusion|JRun CPU and|or memory problem. Is the jrun process specifically consuming the CPU usage, or is another process using it? Is SQL Server or any thing else also running on this box? Also, how much memory does jrun usually use? What are you min and max heap arguments? Windows may have 1 Gig allocated to jrun, but you may only be using a portion of it. Another quick place to check are scheduled tasks in CF. Ditto on the CF server monitoring. It is bad news, period. I hate to say it but memory tracking is virtually unusable in my experience because it has instantly crashed the server any time I've even tried to enable it-- even if only for a few seconds. And finally, buy SeeFusion. It's like 300 bucks and it makes it dang easy to watch your memory, get stack traces (for a specific thread, or the whole server) and it will show you what is running and what SQL it is calling. ~Brad Original Message Subject: ColdFusion|JRun CPU and|or memory problem. From: Ian Skinner Date: Wed, January 20, 2010 3:37 pm To: cf-talk Any good information on how to track down what is causing a ColdFusion server to behave badly. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329878 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
> Yes, looking at the task manager we clearly see one of the JRun services > spinning at 50% (we have a 2 cpu server). In that case, I would focus in on getting some stack traces of exactly what your server is doing while experiencing these problems. That can be done easily cfom the CF server monitor without having to enable any monitoring stuff. The second thing would be some in-depth memory monitoring like Dominic suggested, however I never got a clear answer from you on whether or not you are seeing memory jumps (real ones, not the petabytes that the CF server mon was showing you) or if your main problem was just CPU usage. ~Brad ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329871 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
+1 to Dominic Very useful tool! and can be setup so you can access the server remotely - it doesn't have to run on the same server. Mark On Thu, Jan 21, 2010 at 10:15 AM, Dominic Watson < watson.domi...@googlemail.com> wrote: > > Something else to look at is jConsole. It comes with the Java SDK and will > let you look at any java app's memory, thread and cpu usage in detail > (you'll need to change jrun's JVM args to have it report those stats). > Something like SeeFusion will have some more ColdFusion specific stuff > (long > running pages, queries, etc), but jConsole gets into the memory side of > things in more detail (and it's free). > > HTH > > Dominic > -- E: mark.man...@gmail.com T: http://www.twitter.com/neurotic W: www.compoundtheory.com Hands-on ColdFusion ORM Training @ cf.Objective() 2010 www.ColdFusionOrmTraining.com/ ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329870 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/20/2010 3:03 PM, b...@bradwood.com wrote: > Is the jrun process specifically consuming the CPU usage, or is another > process using it? Is SQL Server or any thing else also running on this > box? > Yes, looking at the task manager we clearly see one of the JRun services spinning at 50% (we have a 2 cpu server). > Also, how much memory does jrun usually use? What are you min and max > heap arguments? Windows may have 1 Gig allocated to jrun, but you may > only be using a portion of it. > > Another quick place to check are scheduled tasks in CF. > > Ditto on the CF server monitoring. It is bad news, period. I hate to > say it but memory tracking is virtually unusable in my experience > because it has instantly crashed the server any time I've even tried to > enable it-- even if only for a few seconds. > Well, it has not caused my any troubles today when I have turned it on. I guess mileage does vary. I have not left it on of course, but I have used it to spot check the situation for several minutes at at time. I even ran it for a couple of hours yesterday. > And finally, buy SeeFusion. It's like 300 bucks and it makes it dang > easy to watch your memory, get stack traces (for a specific thread, or > the whole server) and it will show you what is running and what SQL it > is calling. > Unfortunately money is not a good topic right now, but I if I can ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329869 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
Something else to look at is jConsole. It comes with the Java SDK and will let you look at any java app's memory, thread and cpu usage in detail (you'll need to change jrun's JVM args to have it report those stats). Something like SeeFusion will have some more ColdFusion specific stuff (long running pages, queries, etc), but jConsole gets into the memory side of things in more detail (and it's free). HTH Dominic 2010/1/20 Mark Mandel > > On Thu, Jan 21, 2010 at 9:51 AM, Ian Skinner wrote: > > > > First of all - having memory tracking enabled in your production server > > is > > > going to make it crawl ;o) > > > > > > > Well, this is an Intranet web server that has a minimal load probably > > measured in a dozen or two simultaneous requests at most. So while the > > monitor is definatly a load, I beleive the server has plenty of capacity > > to spare in its normal state. > > > > No seriously - don't use it in production, ever. Ever. Ever. > > The thread dumps are pretty straight forward to read - just ignore most of > the the Java gobbley-de-gook. > > Mark > > -- > E: mark.man...@gmail.com > T: http://www.twitter.com/neurotic > W: www.compoundtheory.com > > Hands-on ColdFusion ORM Training @ cf.Objective() 2010 > www.ColdFusionOrmTraining.com/ > > > ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329868 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On Thu, Jan 21, 2010 at 9:51 AM, Ian Skinner wrote: > > First of all - having memory tracking enabled in your production server > is > > going to make it crawl ;o) > > > > Well, this is an Intranet web server that has a minimal load probably > measured in a dozen or two simultaneous requests at most. So while the > monitor is definatly a load, I beleive the server has plenty of capacity > to spare in its normal state. > No seriously - don't use it in production, ever. Ever. Ever. The thread dumps are pretty straight forward to read - just ignore most of the the Java gobbley-de-gook. Mark -- E: mark.man...@gmail.com T: http://www.twitter.com/neurotic W: www.compoundtheory.com Hands-on ColdFusion ORM Training @ cf.Objective() 2010 www.ColdFusionOrmTraining.com/ ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329867 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
Is the jrun process specifically consuming the CPU usage, or is another process using it? Is SQL Server or any thing else also running on this box? Also, how much memory does jrun usually use? What are you min and max heap arguments? Windows may have 1 Gig allocated to jrun, but you may only be using a portion of it. Another quick place to check are scheduled tasks in CF. Ditto on the CF server monitoring. It is bad news, period. I hate to say it but memory tracking is virtually unusable in my experience because it has instantly crashed the server any time I've even tried to enable it-- even if only for a few seconds. And finally, buy SeeFusion. It's like 300 bucks and it makes it dang easy to watch your memory, get stack traces (for a specific thread, or the whole server) and it will show you what is running and what SQL it is calling. ~Brad Original Message Subject: ColdFusion|JRun CPU and|or memory problem. From: Ian Skinner Date: Wed, January 20, 2010 3:37 pm To: cf-talk Any good information on how to track down what is causing a ColdFusion server to behave badly. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329866 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
On 1/20/2010 1:57 PM, Mark Mandel wrote: > First of all - having memory tracking enabled in your production server is > going to make it crawl ;o) > Well, this is an Intranet web server that has a minimal load probably measured in a dozen or two simultaneous requests at most. So while the monitor is definatly a load, I beleive the server has plenty of capacity to spare in its normal state. > I would start with thread dumps. You can do this through the monitor (there > is a snapshot feature that I believe does this), although setting up a cron > job or similar to fire the Java commands neccessary to get it to happen on > interval wouldn't be bad. > > May just be easier to do it when the CPU spikes - I leave it up to you, but > grab a few at least. > > Then use a tool like Thread Dump Analyzer - so you can see what ColdFusion > is actually doing under the hood. > > Once you can see what is actually going on - you should be able to see what > is causing the problem. > > Mark > > Thanks, I just read and printed the TechNote on how to dump the threads. That is probably the first thing I will get to do tomorrow, today is about done and the Governor has forbidden overtime. Any other resources to how I am going to read these thread dumps once I am looking at them? ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329864 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: ColdFusion|JRun CPU and|or memory problem.
First of all - having memory tracking enabled in your production server is going to make it crawl ;o) I would start with thread dumps. You can do this through the monitor (there is a snapshot feature that I believe does this), although setting up a cron job or similar to fire the Java commands neccessary to get it to happen on interval wouldn't be bad. May just be easier to do it when the CPU spikes - I leave it up to you, but grab a few at least. Then use a tool like Thread Dump Analyzer - so you can see what ColdFusion is actually doing under the hood. Once you can see what is actually going on - you should be able to see what is causing the problem. Mark On Thu, Jan 21, 2010 at 8:37 AM, Ian Skinner wrote: > > Any good information on how to track down what is causing a ColdFusion > server to behave badly. > > The symptoms we have to date: > > 1) Starting January 5th, our Windows 2003, ColdFusion Web Server will > suddenly jump from under 5% CPU usage to an average over 50% with > frequent spikes to 80%, 90% and over. > > 2) When this happens the JRun process for our main JRun process will be > running with over 600,000kb of memory. > > 3) Restarting this ColdFusion instances resets the server back to it's > normal 5% CPU usage. > > 4) When I point the Administrator monitor at the server nothing seems to > be obviously wrong > > 5) *EXCEPT* The monitor will sometimes report an Application Scope of > nearly 821,000,000,000,000kb. Yes, that is over 100 petabytes of data > according to Wolfram-Alpha. I am pretty sure we do not have that much > memory on this server. > > The weird thing is that there does not seem to be a correlation between > that impossible memory spike and the problem with the server. I have > been able to see the spike twice since I have started monitoring things > yesterday. Once this occurred during a problem period, once it did not. > > Using the monitor and looking at the various log files, I can not find > any clear smoking gun, so I am looking for advice on how one my track > down an elusive problem like this. > > ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329862 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: ColdFusion|JRun CPU and|or memory problem.
Ian, Doing triage I always look at (in order). 1) client vars - are they stored in the registry? If so you could have an issue deleting that that "shows up" as the purge process starts hitting actual values to attempt to delete. 2) DB issues - Trouble on the DB spells trouble for the CF Server for most aps - even though CF gets the blame :) Look for locks and blocks. After that I start looking pot luck of other things -Mark -Original Message- From: Ian Skinner [mailto:h...@ilsweb.com] Sent: Wednesday, January 20, 2010 3:38 PM To: cf-talk Subject: ColdFusion|JRun CPU and|or memory problem. Any good information on how to track down what is causing a ColdFusion server to behave badly. The symptoms we have to date: 1) Starting January 5th, our Windows 2003, ColdFusion Web Server will suddenly jump from under 5% CPU usage to an average over 50% with frequent spikes to 80%, 90% and over. 2) When this happens the JRun process for our main JRun process will be running with over 600,000kb of memory. 3) Restarting this ColdFusion instances resets the server back to it's normal 5% CPU usage. 4) When I point the Administrator monitor at the server nothing seems to be obviously wrong 5) *EXCEPT* The monitor will sometimes report an Application Scope of nearly 821,000,000,000,000kb. Yes, that is over 100 petabytes of data according to Wolfram-Alpha. I am pretty sure we do not have that much memory on this server. The weird thing is that there does not seem to be a correlation between that impossible memory spike and the problem with the server. I have been able to see the spike twice since I have started monitoring things yesterday. Once this occurred during a problem period, once it did not. Using the monitor and looking at the various log files, I can not find any clear smoking gun, so I am looking for advice on how one my track down an elusive problem like this. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:329861 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
