But what do you mean by the stack trace showing that "fine so where is it caught up and not resolving"?
A stack trace in and of itself isn't valuable. You need to see it over time. You also said, "I also monitored several of the long running threads (around 2 minutes) and capture several stack traces. They were all the same." So do you really mean they all (what? All request running? And over several thread dumps) showed being stuck with " at java.util.HashMap.get(HashMap.java:325)" being the first line in the stack trace? If not, then you've not really got anything to conclude from the stack traces. Sorry if it sounds like I'm being contentious. I just don't know what you do or don't understand about all this, so am going only on your words. As for the sudden burden of processing at startup, that could indeed be from a bunch of requests all coming in at once. Perhaps you could queue them up in such a case. There's no built-in singleton processing in CF, but you could use CFLOCK surrounding some troublesome code that you permit to only run one at a time, but your problem is that you only want to do it at startup, or when things are going bad, so you need to use something other than CFLOCK. You could code something in CFML, though, that acts like a singleton (only let one request at a time run this during a period you detect as a trouble time). Just thinking off the top of my head here. But I'll share one other thought: others who experience slow startup times often find that it's because that flush of new requests is triggering a lot of java class loading, and there's a known issue in the JVM 1.6 version implemented with CF. Some moved back to 1.5 to solve it, while others updated to 1.6.10 (which fixed the problem) to make it go away. Have you tried either? There are various blogs about this. The first here explains the problem and how to do it: http://www.ghidinelli.com/2008/08/18/java16-u10-gives-big-boost-to-modelglue -transfer-coldspring-performance Others with more other detail are: http://www.compoundtheory.com/?action=displayPost&ID=270 http://corfield.org/blog/index.cfm/do/blog.entry/entry/Java_6_and_ColdFusion _8 Hope that's helpful. /charlie -----Original Message----- From: cfaussie@googlegroups.com [mailto:cfaus...@googlegroups.com] On Behalf Of Matthew Sent: Wednesday, January 14, 2009 1:30 AM To: cfaussie Subject: [cfaussie] Re: JRUN hanging Hi guys We've just a server "hang". It was a little different to usual (perhaps because Fusion Reactor is installed). This time the website was showing the JRun error message. When I looked at the JRun process the RAM was up at 615MB!!! I restarted JRun.exe. As the website came back up it started to chock again. There is obviously extra load as all the objects, web services etc run for the first time. I know that with the 3rd party web service which I've spoken of in previous posts that always the first call takes a good couple of minutes. Unfortunately when it happens on the live server you usually get at least half a dozen users trying to hit the page which consumes this web service so all requests take a long time and chew up a lot of resources. I was watching things through FR and the Task Manager: the CPU was up near 100% plus the RAM was quite high (it normally sits around 200Mb but was 350+). Meanwhile I couldn't even hit a very light weight page on the site. I used the KILL feature in FR to terminate about 20 threads and eventually the CPU usage and RAM dropped down to stable levels. I also monitored several of the long running threads (around 2 minutes) and capture several stack traces. They were all the same. Perhaps someone will be able to explain it to me so I can understand what I'm reading. For example I read the below that the CFINVOKE was fine so where is it caught up and not resolving? at java.util.HashMap.get(HashMap.java:325) at org.apache.axis.utils.JavaUtils.isEnumClass(JavaUtils.java:1040) at org.apache.axis.encoding.ser.BeanSerializerFactory.init (BeanSerializerFactory.java:49) at org.apache.axis.encoding.ser.BeanSerializerFactory. (BeanSerializerFactory.java:42) at org.apache.axis.encoding.ser.BaseSerializerFactory.createFactory (BaseSerializerFactory.java:235) at org.apache.axis.client.Call.registerTypeMapping(Call.java:2296) at com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub.createCall (RailEuropeWebServiceSoapBindingStub.java:2454) - locked <0x1534d630> (a com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub) at com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub.doBuildPackag eForCityPair (RailEuropeWebServiceSoapBindingStub.java:2663) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:223) at coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:143) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:1594) at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:388) at cfen_p2p2ecfc1610607676$funcSEARCHP2P.runFunction(C:\Inetpub \wwwroot\model\search.cfc:423) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:344) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke (UDFMethod.java:290) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke (UDFMethod.java:254) at coldfusion.filter.FunctionAccessFilter.invoke (FunctionAccessFilter.java:56) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:207) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:169) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:194) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:146) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:1634) at cfpoint2dto2dpoint2ecfm401551261.runPage(C:\Inetpub\wwwroot \search.cfm:33) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:152) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:349) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java: 225) at coldfusion.filter.RequestMonitorFilter.invoke (RequestMonitorFilter.java:51) at coldfusion.filter.PathFilter.invoke(PathFilter.java:86) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69) at coldfusion.filter.BrowserDebugFilter.invoke (BrowserDebugFilter.java:52) at coldfusion.filter.ClientScopePersistenceFilter.invoke (ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java: 22) at coldfusion.filter.RequestThrottleFilter.invoke (RequestThrottleFilter.java:115) at coldfusion.CfmServlet.service(CfmServlet.java:107) at coldfusion.bootstrap.BootstrapServlet.service (BootstrapServlet.java:78) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorFilter.B(Unknown Source) at com.intergral.fusionreactor.filter.FusionReactorFilter.A(Unknown Source) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter (Unknown Source) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke (JRunRequestDispatcher.java:257) at jrun.servlet.ServletEngineService.dispatch (ServletEngineService.java:541) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable (JRunProxyService.java:204) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable (ThreadPool.java:318) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable (ThreadPool.java:426) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable (ThreadPool.java:264) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) By the way: this doesn't really reflect what I raised in my first thread because this shows what happen straight after a CF restart where as the original issue happens when everything is stable and then suddenly CF hangs. I'm still waiting for that to happen. Cheers Matthew --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "cfaussie" group. To post to this group, send email to cfaussie@googlegroups.com To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en -~----------~----~----~----~------~----~------~--~---
