Hi guys
We've just a server "hang". It was a little different to usual
(perhaps because Fusion Reactor is installed). This time the website
was showing the JRun error message. When I looked at the JRun process
the RAM was up at 615MB!!! I restarted JRun.exe. As the website came
back up it started to chock again. There is obviously extra load as
all the objects, web services etc run for the first time. I know that
with the 3rd party web service which I've spoken of in previous posts
that always the first call takes a good couple of minutes.
Unfortunately when it happens on the live server you usually get at
least half a dozen users trying to hit the page which consumes this
web service so all requests take a long time and chew up a lot of
resources. I was watching things through FR and the Task Manager: the
CPU was up near 100% plus the RAM was quite high (it normally sits
around 200Mb but was 350+). Meanwhile I couldn't even hit a very light
weight page on the site. I used the KILL feature in FR to terminate
about 20 threads and eventually the CPU usage and RAM dropped down to
stable levels.
I also monitored several of the long running threads (around 2
minutes) and capture several stack traces. They were all the same.
Perhaps someone will be able to explain it to me so I can understand
what I'm reading. For example I read the below that the CFINVOKE was
fine so where is it caught up and not resolving?
at java.util.HashMap.get(HashMap.java:325)
at org.apache.axis.utils.JavaUtils.isEnumClass(JavaUtils.java:1040)
at org.apache.axis.encoding.ser.BeanSerializerFactory.init
(BeanSerializerFactory.java:49)
at org.apache.axis.encoding.ser.BeanSerializerFactory.
(BeanSerializerFactory.java:42)
at org.apache.axis.encoding.ser.BaseSerializerFactory.createFactory
(BaseSerializerFactory.java:235)
at org.apache.axis.client.Call.registerTypeMapping(Call.java:2296)
at
com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub.createCall
(RailEuropeWebServiceSoapBindingStub.java:2454)
- locked <0x1534d630> (a
com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub)
at
com.raileurope.web.ws.soap.RailEuropeWebServiceSoapBindingStub.doBuildPackageForCityPair
(RailEuropeWebServiceSoapBindingStub.java:2663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:223)
at coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:143)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:1594)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:388)
at cfen_p2p2ecfc1610607676$funcSEARCHP2P.runFunction(C:\Inetpub
\wwwroot\model\search.cfc:423)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:344)
at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke
(UDFMethod.java:290)
at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke
(UDFMethod.java:254)
at coldfusion.filter.FunctionAccessFilter.invoke
(FunctionAccessFilter.java:56)
at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:207)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:169)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:194)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:146)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:1634)
at cfpoint2dto2dpoint2ecfm401551261.runPage(C:\Inetpub\wwwroot
\search.cfm:33)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:152)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:349)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:
225)
at coldfusion.filter.RequestMonitorFilter.invoke
(RequestMonitorFilter.java:51)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69)
at coldfusion.filter.BrowserDebugFilter.invoke
(BrowserDebugFilter.java:52)
at coldfusion.filter.ClientScopePersistenceFilter.invoke
(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:
22)
at coldfusion.filter.RequestThrottleFilter.invoke
(RequestThrottleFilter.java:115)
at coldfusion.CfmServlet.service(CfmServlet.java:107)
at coldfusion.bootstrap.BootstrapServlet.service
(BootstrapServlet.java:78)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at com.intergral.fusionreactor.filter.FusionReactorFilter.B(Unknown
Source)
at com.intergral.fusionreactor.filter.FusionReactorFilter.A(Unknown
Source)
at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter
(Unknown Source)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke
(JRunRequestDispatcher.java:257)
at jrun.servlet.ServletEngineService.dispatch
(ServletEngineService.java:541)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable
(JRunProxyService.java:204)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable
(ThreadPool.java:318)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable
(ThreadPool.java:426)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable
(ThreadPool.java:264)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
By the way: this doesn't really reflect what I raised in my first
thread because this shows what happen straight after a CF restart
where as the original issue happens when everything is stable and then
suddenly CF hangs. I'm still waiting for that to happen.
Cheers
Matthew
On Jan 8, 6:27 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:
> Well, before you get too excited, be careful that you're not
> misunderstanding the report. The memory reported there is NOT for the
> current request, but rather for the entire CF server. So that number alone
> isn't too meaningful. But if it ROSE by 32 meg when you ran a request, that
> would be different (but even then you can't be positive that a given request
> caused the rise. Other things can be running to increase memory besides the
> request you're running.)
>
> Rather than view the memory in the stack trace, to observe how it's changing
> over time, I'd recommend instead you watch either the graphical interface
> they offer for the memory graph, or view the info in the resource-x.log
> file, both of which report it at 5 second intervals.
>
> But if that indeed is a stack trace of the call to the web service, notice
> the top line which says:
>
> java.net.SocketInputStream.socketRead0(Native Method)
>
> That would be the kind of thing (a native method) that CF's request timeout
> feature can't interrupt normally, but again I have confirmed that in my test
> where the TIMEOUT of a CFINVOKE webservice call works, it is indeed timing
> out while sitting in that same state. My assertion is that CF has added some
> additional timeout callback mechanism in such a case, which it doesn't do as
> a matter of course on other code. Makes sense to me.
>
> The question is why it doesn't timeout for you--but that will be the thing
> to confirm now. When you have a request that you see runs long, and you KNOW
> it has a timeout, if you sit there refreshing the stack trace, does it
> remain in this socketread0 method beyond that timeout time? It doesn't for
> me. :-)
>
> /charlie
>
> -----Original Message-----
> From: cfaussie@googlegroups.com [mailto:cfaus...@googlegroups.com] On Behalf
>
> Of Matthew
> Sent: Wednesday, January 07, 2009 11:32 PM
> To: cfaussie
> Subject: [cfaussie] Re: JRUN hanging
>
> Hi Charlie,
>
> Out of interest (while I await a hung server) I've configured FR 2.0.4
> on my dev PC and run a couple of calls to the web service whilst I
> watch them in FR. I was alarmed to see that my single request was
> costing 32MB of memory (32,438KB)!!!! This snapshot was taken at about
> the 30th second (the whole request finished after 55s).
>
> Here is the top part of the "Thread Stack Trace":
>
> Thread Stack Trace
> Trace Time: 15:26:34.224 08-Jan-2009
> Request ID: 25
> Script Name: http://foo
> Started: 15:26:04.177 08-Jan-2009
> Exec Time: 30047ms
> Memory Used: (6%)32,438KB
> Memory Free: 472,457KB
> Thread ID: jrpp-8
> Priority: 5
> Hashcode: 30900283
>
> "jrpp-8" prio=5 tid=0x03c5f468 nid=0x3c4 runnable [559d000..559fd90]
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA12275)
> at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA12275)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA12275)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275)
> - locked <0x158afd20> (a java.lang.Object)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA12275)
> - locked <0x158afc88> (a
> com.sun.net.ssl.internal.ssl.AppInputStream)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:220)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
> - locked <0x158afad8> (a java.io.BufferedInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:111)
> at
> org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read
> (Unknown Source)
>
> Cheers
> Matthew
>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com
To unsubscribe from this group, send email to
cfaussie+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/cfaussie?hl=en
-~----------~----~----~----~------~----~------~--~---
