On Feb 22, 2013, at 22:32 , Daniel Mikusa <dmik...@vmware.com> wrote:
> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>
>> Hi!
>>
>> I have 5 servers with the following configuration:
>> Windows Server 2008 R2 64-bit,
>> Oracle JDK 1.7.0_11-b21 64-bit,
>> Tomcat 7.0.35 running as a service.
>> A few webapps are running perfectly well under the Tomcat until one of
>> them is reloaded.
>>
>> Reload of a webapp (either by pressing [reload] on manager page or by
>> changing web.xml file) makes Tomcat extremely unstable.
>> The first (pre-production) server sporadically fails with absolutely
>> no traces in log files or Windows event logs:
>
> These three symptoms can often mean different things. For example, if it's
> not accepting connections then you can't really get a 404 error. Can you be
> more specific about when you see them and the status of your Tomcat server
> when you see them.
>
> Some troubleshooting thoughts...
>
>> does not accept connections,
>
> 1.) Is the process still running? Did the JVM crash? Is there a crash file?
> hs_err_podXXXX.log?
> 2.) Is the socket still listening? Have you changed any firewall rules?
> 3.) Try "telnet <port>". What happens?
> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so
> look at the connector mbean. What is its state?
> 5.) Take some thread dumps.
>
>> responds with HTTP 404 or
>
> Assuming the URL that you entered is valid, this sounds like your application
> may have failed to reload. Can you turn up the log level in your application
> to see what it is doing?
>
>> does not respond at all.
>
> Assuming that you mean it accepts your connection and request, but never
> processes and returns a response. In this case, take some thread dumps after
> you have sent the response. That should show you what is going on inside the
> JVM.
Here is more details.
All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
A webapp is serving real-time data with updates. Open the page in a browser, it
downloads an HTML (a static file), then a bunch of CSS and Javascripts (also
static files), and starts fetching the data and updates (from a servlet) - this
makes an indefinite sequence of HTTP requests.
When I open the page after the webapp is reloaded, some of this requests
(including the static files) fail with either 404 code or no response or not
accepted connection. Requests are being failed quite randomly. If I reload the
page, also a lot of fails, but slightly different - e.g. I might get the file
which was 404 earlier. The data requests also fail randomly, some of them are
successful, which means that the webapp is loaded.
So, answering to the questions above: the process is definitely running, socket
is definitely listening, telneting Tomcat gives random results, JConsole shows
nothing uncommon.
>
>> Other 4 (production) servers also become unstable but in a different
>> manner - a lot of HTTP 500 responses (logs are also clean).
>
> It would be very odd to see nothing in the logs after a 500 error. Have you
> looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"?
> That file should contain a stack trace after a 500 error. Also, what do the
> see in your browser when you get the 500 error? The default error page
> should print the stack trace for the 500 error.
That is what I am talking about - quite odd behaviour. Nothing in log files, no
body with HTTP 500 response in a browser.
>
>> Plus on
>> some of these servers I have a lot (less than HTTP 500 responses
>> though) of NPEs:
>> -------------
>> SEVERE: Exception Processing /path/here
>> java.lang.NullPointerException
>> at
>> org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>> at
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>> at
>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>> at
>> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> -------------
>> These exception occurs for any webapp, not only the one being reloaded.
>
> What's the specific version of Tomcat where you saw this error occur? Is it
> 7.0.37? or was this on an older version?
This was on 7.0.35
>
>>
>> All the above happens irregularly, quite often on the pre-production
>> server, only once on production servers. Tomcat service restart fixes
>> the problem. Updating Tomcat to 7.0.37 did not help.
>> I cannot reproduce this on any of developers hosts with the same
>> versions of JDK and Tomcat.
>>
>> Any ideas?
>
> 1.) An easy temporary fix would be to not reload the application in
> production. If you need to redeploy or reload the application, stop the
> server, undeploy, redeploy and start your server.
>
> 2.) Include your Tomcat configuration, minus comments.
Please see below:
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
<!-- A couple of custom resources here -->
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="80" protocol="HTTP/1.1"
asyncTimeout="20000"
URIEncoding="utf-8"
acceptorThreadCount="2"
compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
compression="on"
connectionTimeout="20000"
maxThreads="1024"
processorCache="512"
redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b %Dms"
resolveHosts="false"/>
</Host>
</Engine>
</Service>
</Server>
Best regards,
Kirill
>
> 3.) See troubleshooting steps above.
>
> Dan
>
>>
>>
>> Thanks a lot in advance,
>> Kirill
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
