Hi everyone,
thanks for all the good feedback.
Let me first tell you WHY I think its a JVM issue, then show you what
results I have had.
We took the exact same code, and connected it to the exact same DB (and DB
server) and ran the exact same code in isolation. The comparison was
outstanding about 2s on the old machine and 10s as you have seen on the new.
The connection between the SQL and both web machines pinged at <1ms
We tried the connection from various locations and ruled out network issues
(although there could be ). We looked at execution times and logged the
information pretty heavily in the intensive parts of the app and saw that,
on comparison between machines the differences. Given that, and the
isolation tests, the only thing we could see that was different was the JVM
settings, thus the conclusion we came too.
so having taken some of the suggestions into account I have ended up with
this setting:
java.args=-server -Xms4096m -Xmx8192m -Dsun.io.useCanonCaches=false
-XX:PermSize=1024m -XX:MaxPermSize=1024m
-Dsun.rmi.dgc.client.gcInterval=150000
-Dsun.rmi.dgc.server.gcInterval=150000 -XX:+UseParallelGC -Xbatch
-Dcoldfusion.rootDir={application.home}/
-Djava.security.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/coldfusion.policy
-Djava.security.auth.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/neo_jaas.policy
This puts us at an execution time parallel to the old 32 bit machine, down
at 1.5-2 seconds. while this is a vast improvement I would hope that more
gains could be had out of tuning the JVM some more. I will tweak the
GCinterval and the permSize tomorrow night and see if there is an
incremental gain to be had.
Any other suggestions would be great!
Thanks.
On Thu, Feb 17, 2011 at 3:20 AM, Mark A. Kruger <mkru...@cfwebtools.com>wrote:
>
> When you get varied response times like that I often take a closer look at
> the network (duplexing, flow control etc).
>
>
> -----Original Message-----
> From: Wil Genovese [mailto:jugg...@trunkful.com]
> Sent: Wednesday, February 16, 2011 10:11 AM
> To: cf-talk
> Subject: Re: JVM help!
>
>
> Duncan,
>
> Sorry about _MY_ confusion. I was reading and responding to your request,
> while 'out', with my phone and the small screen sometimes jumbles up a
> message enough to be confusing. I think I saw this "running out of memory"
> and thought that was the issue. I just noticed that the entire message
> didn't show up on my phone. Sorry.
>
> Still the JVM settings should be set as many have mentioned. However, the
> most critical thing you can do is monitor everything. DB, CF, IIS, OS,
> network, name resolution (DNS) just to name a few possible things. Is the
> DB on the same machine? or on a network somewhere? make sure there is
> nothing wrong on the network to slow things down. Is the new server
> connection to network resources via IP addresses or using DNS? If using DNS,
> make sure name resolution happens fast from the server. I actually seen slow
> name resolution (or timeouts of resolution) cause issues on a server. There
> may not even be an issue with ColdFusion or the JVM.
>
> When I go to http://www.hairbodyskin.com.au/ my FF plugins tells me it
> took 10.449s from the time my http request was sent to the first response
> from the server, and then a total of 15s to complete the response. (LORI
> plugin for FF). Then sometimes the requests happen fast. This response
> times are all over the spectrum here.
>
> The other apps, are those heavily used? This is a new machine, but even so
> are you overloading it with too many high load apps?
>
> There are a lot of possible issues. Monitor everything.
>
> Everyone including myself can offer advice on settings and potential
> causes, but we are all blind to the real issue without accurate monitoring.
>
>
>
>
> Wil Genovese
> Sr. Web Application Developer/
> Systems Administrator
> CF Webtools
> www.cfwebtools.com
>
> wilg...@trunkful.com
> www.trunkful.com
>
> On Feb 16, 2011, at 1:27 AM, Duncan wrote:
>
> >
> > Hi Will,
> >
> > Thanks - but I am confused about your terminology. You state we are
> looking
> > for a memory issue, is this inextricably linked to the execution time?
> i.e.
> > could it be the JVM trying to cleanup the stuff in memory prior to the
> end
> > of the request?
> >
> > As I pointed out the exact same code, with the exact same data works
> faster
> > on the old 32bit machine. If I can put some logic around this, on the
> > possible causes and how that relates to the JVM settings, I will perhaps
> be
> > able to make some better decisions.
> >
> > Can you help expand on what we are looking for?
> >
> > Thanks!
> >
> > On Wed, Feb 16, 2011 at 4:09 PM, Wil Genovese <wilg...@gmail.com> wrote:
> >
> >>
> >> Set min max heap to be equal.
> >>
> >> Set gc intervals to 10000
> >>
> >> And try again. These are not hard set numbers but in many cases works
> good
> >> enough to start figuring out what is really happening.
> >>
> >> You should install monitoring package and look for the cause of the
> memory
> >> issue. Typically it's unexpected behavior in code. Many times a query
> >> returning far more results than expected.
> >>
> >> Monitor, fix code monitor again, adjust settings as needed.
> >>
> >>
> >>
> >>
> >> Wil Genovese
> >> Sr Web Application Developer / Sys Admin
> >>
> >> CFWebtools
> >>
> >> On Feb 15, 2011, at 10:48 PM, Duncan <duncan.lox...@gmail.com> wrote:
> >>
> >>>
> >>> Hi Everyone,
> >>>
> >>> We have recently put together a new CF server, moving from 32 bit, 4Gb
> >> RAM
> >>> Windows over to 64bit, 16Gb RAM Windows.
> >>>
> >>> We have pretty much done a default set up, and placed the same set of
> >>> applications on the new server, and have run in the machine for a bit.
> >> After
> >>> a few weeks we have been noticing some cracks appear, and we now have
> >> some
> >>> issues that we have narrowed down to the JVM/CF setup.
> >>>
> >>> Here is the setup on the old and new boxes:
> >>>
> >>> 4GB, Dual Xeon, 32bit windows, jdk1.6.0_13
> >>>
> >>>
> >>>
> >>> java.args=-server -Xms512m -Xmx1000m -Dsun.io.useCanonCaches=false
> >>> -Dcoldfusion.fckupload=true -XX:PermSize=128m -XX:MaxPermSize=240m
> >>> -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError
> >>> -Dsun.rmi.dgc.client.gcInterval=150000
> >>> -Dsun.rmi.dgc.server.gcInterval=150000
> >>> -Dcoldfusion.rootDir={application.home}/../
> >>> -Dcoldfusion.libPath={application.home}/../lib
> >>>
> >>
> -Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/flex/jars,{application.home}/../wwwroot/WEB-INF/cfform/jars
> >>>
> >>>
> >>> Crashed a lot, had out of memory issues once every hour on a good day.
> >>>
> >>>
> >>>
> >>> www.hairbodyskin.com.au takes 2,781ms (from FusionReactor)
> >>>
> >>>
> >>>
> >>>
> >>> 16Gb, Quad Core, 64bit Windows, jdk1.6.0_13
> >>>
> >>>
> >>>
> >>> java.args=-server âââ‰â¬ÅXmx8192m -Xms8192m
> -Dsun.io.useCanonCaches=false
> >>> -XX:MaxPermSize=1024m -XX:+UseParallelGC -Xbatch
> >>> -Dcoldfusion.rootDir={application.home}/
> >>>
> >>
> -Djava.security.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/coldfusion.policy
> >>>
> >>
> -Djava.security.auth.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/neo_jaas.policy
> >>>
> >>>
> >>> Runs great, has no crashes, happily inside memory limits, but runs so
> >> slow.
> >>>
> >>>
> >>> www.hairbodyskin.com.au takes 10,361ms (from FusionReactor)
> >>>
> >>> I must add that this is NOT the only application on this server, there
> >> are
> >>> many others, mostly built on the same code base, some older some newer.
> >>>
> >>> I would like some advice on how to better adjust my settings - I think
> >> this
> >>> should be reasonably for those educated in the art because we are
> talking
> >>> (in some cases) a 5x difference in execution speeds. I am thinking:
> >>>
> >>> 1) increasing GC interval to 600000
> >>> 2) fixing max and min perm size to 1024
> >>> 3) putting xms down to 4096
> >>>
> >>> Are these sensible? Any other suggestions?
> >>>
> >>> Thanks in advance!
> >>>
> >>> --
> >>> Duncan I Loxton
> >>> duncan.lox...@gmail.com
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
>
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342347
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
