RE: Garbage Collection Lag
Correct, PermGen (class objects and meta data) are not on the heap. Also correct is the the query/session/application variable memory is stored on the heap. These are the memory objects that are getting moved around and collected when obsolete. I think once you increase your New Generation size and give your PermGen a larger minimum, you'll see drastic improvements. In fact, your info that UseConcMarkSweepGC actually decreases your performance makes me believe that the PermGen was causing most of your problems. When the PermGen space needs to be expanded, the JVM forces a full GC. If you don't specify a minimum size, the default for PermGen is only 4m. Since you're working with over 900 classes, I think your PermGen space is being increased almost constantly. -Original Message- From: Terry Ford [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 04, 2006 6:24 PM To: CF-Talk Subject: Re: Garbage Collection Lag Thanks -- I'll give that a try. Have got approx 950 classes in this app, so I'll watch closely for out of memory errors wrto MaxPermSize (classes are in Perm memory and not Heap, right?) After fiddling a little further, -XX:+UseConcMarkSweepGC seems to do more harm than good on my setup. The problem is that the normal garbage collection appears to run less frequently but for a longer period of time. So instead of frequent 1-2 second delays with UseConcMarkSweepGC , you get infrequent 3-4 second delays. The problem is that even 1-2 second delays are nasty when you've got incoming requests of 60+ templates/second. By the time the GC is complete you've got ~150 incoming requests that have to sort themselves out. By turning the GCConcMarkSweep off you don't get as many interrupting events. I like to completely avoid queued requests if possible. One last question... is it correct that all cached queries / session / application variables are stored in the heap, and not perm memory? I use a *ton* of application variables (tables and parts of tables in structs of arrays and arrays of structs and all sorts of other fun configurations) and a fair number of cached queries. Just wondering if the heavy use of memory in application variables affects jvm optimization in any way. Regards, Terry >Permanent Generation isn't involved in the garbage collection process >at all. The PermGen space is where your JVM stores the meta-data about >the classes that ColdFusion is creating constantly. The parameter you need to >worry about is your NewSize. One thing I notice that you probably don't >want to do is set your MaxPermSize so big. ColdFusion uses a lot of >PermGen space, but not that much - especially since you only have 512m >set aside for your total memory size. And yes, you're right, your JVM >size is Xmx + PermSize. > >Here are some recommendations: > >- You have 4GB of RAM on the machine, you should dedicate much more to >the size of the JVM (assuming this is your production machine). >Perhaps 2GB >- Set a minimum PermSize. -XX:PermSize=80m. >- I've never seen ColdFusion use more than 120m in PermGen space even >for massive applications, so set MaxPermSize somewhere in that range >- Make your New Generation space larger. Perhaps 25% of your total JVM >size. >- The -XX:+UseConcMarkSweepGC setting seems to make no difference in >any of the tests I've run either so you can not worry about that > >Last but not least, if you're using JVM 1.5 look into using jconsole to >monitor your memory usage. If you're still on 1.4.2, you're limited to >seeing just the total JVM usage, but you can still monitor it to see >how much it's fluctuating during use. Bottom line is that every >application is different, so you need to tinker with the size of your >memory spaces until you get the performance you need. In the >applications I've looked at, simply increasing the New Generation size helps tremendously. > >-Original Message- >From: Terry Ford [mailto:[EMAIL PROTECTED] >Sent: Wednesday, January 04, 2006 4:39 PM >To: CF-Talk >Subject: Garbage Collection Lag > >Hey CFers, > >As traffic increases and the server has been up for a while I'm >starting to notice periods where the JVM seems to be taking vacations >when it ought to be processing templates. I suspect that this is garbage collection. > >The symptom is a cfstat period for 1 or 2 seconds where no templates >are processed, yet at the same time no templates are queued. There are >no queries running on mysql either, so it does look as if the JVM is at >work and ignoring CFMX. > >It gets pretty bad at times, with such a pause occuring every 30 >seconds or so. > >Any suggestions on how to reduce this garbage collection lag? > >Here's what I'm u
Re: Garbage Collection Lag
Thanks -- I'll give that a try. Have got approx 950 classes in this app, so
I'll watch closely for out of memory errors wrto MaxPermSize (classes are in
Perm memory and not Heap, right?)
After fiddling a little further, -XX:+UseConcMarkSweepGC seems to do more harm
than good on my setup. The problem is that the normal garbage collection
appears to run less frequently but for a longer period of time. So instead of
frequent 1-2 second delays with UseConcMarkSweepGC , you get infrequent 3-4
second delays.
The problem is that even 1-2 second delays are nasty when you've got incoming
requests of 60+ templates/second. By the time the GC is complete you've got
~150 incoming requests that have to sort themselves out. By turning the
GCConcMarkSweep off you don't get as many interrupting events. I like to
completely avoid queued requests if possible.
One last question... is it correct that all cached queries / session /
application variables are stored in the heap, and not perm memory? I use a
*ton* of application variables (tables and parts of tables in structs of arrays
and arrays of structs and all sorts of other fun configurations) and a fair
number of cached queries. Just wondering if the heavy use of memory in
application variables affects jvm optimization in any way.
Regards,
Terry
>Permanent Generation isn't involved in the garbage collection process at
>all. The PermGen space is where your JVM stores the meta-data about the
>classes that ColdFusion is creating constantly. The parameter you need to
>worry about is your NewSize. One thing I notice that you probably don't
>want to do is set your MaxPermSize so big. ColdFusion uses a lot of PermGen
>space, but not that much - especially since you only have 512m set aside for
>your total memory size. And yes, you're right, your JVM size is Xmx +
>PermSize.
>
>Here are some recommendations:
>
>- You have 4GB of RAM on the machine, you should dedicate much more to the
>size of the JVM (assuming this is your production machine). Perhaps 2GB
>- Set a minimum PermSize. -XX:PermSize=80m.
>- I've never seen ColdFusion use more than 120m in PermGen space even for
>massive applications, so set MaxPermSize somewhere in that range
>- Make your New Generation space larger. Perhaps 25% of your total JVM
>size.
>- The -XX:+UseConcMarkSweepGC setting seems to make no difference in any of
>the tests I've run either so you can not worry about that
>
>Last but not least, if you're using JVM 1.5 look into using jconsole to
>monitor your memory usage. If you're still on 1.4.2, you're limited to
>seeing just the total JVM usage, but you can still monitor it to see how
>much it's fluctuating during use. Bottom line is that every application is
>different, so you need to tinker with the size of your memory spaces until
>you get the performance you need. In the applications I've looked at,
>simply increasing the New Generation size helps tremendously.
>
>-Original Message-
>From: Terry Ford [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, January 04, 2006 4:39 PM
>To: CF-Talk
>Subject: Garbage Collection Lag
>
>Hey CFers,
>
>As traffic increases and the server has been up for a while I'm starting to
>notice periods where the JVM seems to be taking vacations when it ought to
>be processing templates. I suspect that this is garbage collection.
>
>The symptom is a cfstat period for 1 or 2 seconds where no templates are
>processed, yet at the same time no templates are queued. There are no
>queries running on mysql either, so it does look as if the JVM is at work
>and ignoring CFMX.
>
>It gets pretty bad at times, with such a pause occuring every 30 seconds or
>so.
>
>Any suggestions on how to reduce this garbage collection lag?
>
>Here's what I'm using right now. dual xeon machine, 4GB RAM, linux. A ps
>-e at steady state shows the cfusion process using ~650MB of memory (which
>is an unrelated question: how can CF be using 650MB if Xmx = 512M? Unless
>the process memory = Xmx + Maxperm?)
>
>java.args=-server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS
>-Xbootclasspath/a:{application.home}/lib/webchartsJava2D.jar
>-Djava.awt.graphicsenv=com.gp.java2d.Ex
>HeadlessGraphicsEnvironment -Xmx512m -Dsun.io.useCanonCaches=false
>-XX:MaxPermSize=256m
>-Djavax.xml.parsers.SAXParserFactory=com.macromedia.crimson.jaxp.SAXParserFa
>cto
>ryImpl
>-Djavax.xml.parsers.DocumentBuilderFactory=com.macromedia.crimson.jaxp.Docum
>entBuilderFactoryImpl -XX:NewSize=48m
>
>Anyway, any ideas would be much appreciated... does a larger MaxPermSize
>increase or decrease garbage collection lag? any other params that might
>help out here? I tried -XX:+UseConcMarkSweepGC but it didnt seem to help at
>all.
>
>Thanks
~|
Message: http://www.houseoffusion.com/lists.cfm/link=i:4:228401
Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4
Subscription: http://www.houseoffusion.com/lists.c
RE: Garbage Collection Lag
Permanent Generation isn't involved in the garbage collection process at
all. The PermGen space is where your JVM stores the meta-data about the
classes that ColdFusion is creating constantly. The parameter you need to
worry about is your NewSize. One thing I notice that you probably don't
want to do is set your MaxPermSize so big. ColdFusion uses a lot of PermGen
space, but not that much - especially since you only have 512m set aside for
your total memory size. And yes, you're right, your JVM size is Xmx +
PermSize.
Here are some recommendations:
- You have 4GB of RAM on the machine, you should dedicate much more to the
size of the JVM (assuming this is your production machine). Perhaps 2GB
- Set a minimum PermSize. -XX:PermSize=80m.
- I've never seen ColdFusion use more than 120m in PermGen space even for
massive applications, so set MaxPermSize somewhere in that range
- Make your New Generation space larger. Perhaps 25% of your total JVM
size.
- The -XX:+UseConcMarkSweepGC setting seems to make no difference in any of
the tests I've run either so you can not worry about that
Last but not least, if you're using JVM 1.5 look into using jconsole to
monitor your memory usage. If you're still on 1.4.2, you're limited to
seeing just the total JVM usage, but you can still monitor it to see how
much it's fluctuating during use. Bottom line is that every application is
different, so you need to tinker with the size of your memory spaces until
you get the performance you need. In the applications I've looked at,
simply increasing the New Generation size helps tremendously.
-Original Message-
From: Terry Ford [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 04, 2006 4:39 PM
To: CF-Talk
Subject: Garbage Collection Lag
Hey CFers,
As traffic increases and the server has been up for a while I'm starting to
notice periods where the JVM seems to be taking vacations when it ought to
be processing templates. I suspect that this is garbage collection.
The symptom is a cfstat period for 1 or 2 seconds where no templates are
processed, yet at the same time no templates are queued. There are no
queries running on mysql either, so it does look as if the JVM is at work
and ignoring CFMX.
It gets pretty bad at times, with such a pause occuring every 30 seconds or
so.
Any suggestions on how to reduce this garbage collection lag?
Here's what I'm using right now. dual xeon machine, 4GB RAM, linux. A ps
-e at steady state shows the cfusion process using ~650MB of memory (which
is an unrelated question: how can CF be using 650MB if Xmx = 512M? Unless
the process memory = Xmx + Maxperm?)
java.args=-server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS
-Xbootclasspath/a:{application.home}/lib/webchartsJava2D.jar
-Djava.awt.graphicsenv=com.gp.java2d.Ex
HeadlessGraphicsEnvironment -Xmx512m -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=256m
-Djavax.xml.parsers.SAXParserFactory=com.macromedia.crimson.jaxp.SAXParserFa
cto
ryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.macromedia.crimson.jaxp.Docum
entBuilderFactoryImpl -XX:NewSize=48m
Anyway, any ideas would be much appreciated... does a larger MaxPermSize
increase or decrease garbage collection lag? any other params that might
help out here? I tried -XX:+UseConcMarkSweepGC but it didnt seem to help at
all.
Thanks
~|
Message: http://www.houseoffusion.com/lists.cfm/link=i:4:228397
Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4
Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
