On 25.02.2013 2:10, Daniel Mikusa wrote:
On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
On Feb 22, 2013, at 22:32 , Daniel Mikusa <dmik...@vmware.com> wrote:
On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
Hi!
I have 5 servers with the following configuration:
Windows Server 2008 R2 64-bit,
Oracle JDK 1.7.0_11-b21 64-bit,
Tomcat 7.0.35 running as a service.
A few webapps are running perfectly well under the Tomcat until one of
them is reloaded.
Reload of a webapp (either by pressing [reload] on manager page or by
changing web.xml file) makes Tomcat extremely unstable.
The first (pre-production) server sporadically fails with absolutely
no traces in log files or Windows event logs:
These three symptoms can often mean different things. For example, if it's not
accepting connections then you can't really get a 404 error. Can you be more
specific about when you see them and the status of your Tomcat server when you
see them.
Some troubleshooting thoughts...
does not accept connections,
1.) Is the process still running? Did the JVM crash? Is there a crash file?
hs_err_podXXXX.log?
2.) Is the socket still listening? Have you changed any firewall rules?
3.) Try "telnet <port>". What happens?
4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look
at the connector mbean. What is its state?
5.) Take some thread dumps.
responds with HTTP 404 or
Assuming the URL that you entered is valid, this sounds like your application
may have failed to reload. Can you turn up the log level in your application
to see what it is doing?
does not respond at all.
Assuming that you mean it accepts your connection and request, but never
processes and returns a response. In this case, take some thread dumps after
you have sent the response. That should show you what is going on inside the
JVM.
Here is more details.
All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
A webapp is serving real-time data with updates. Open the page in a browser, it
downloads an HTML (a static file), then a bunch of CSS and Javascripts (also
static files), and starts fetching the data and updates (from a servlet) - this
makes an indefinite sequence of HTTP requests.
When I open the page after the webapp is reloaded, some of this requests
(including the static files) fail with either 404 code or no response or not
accepted connection. Requests are being failed quite randomly. If I reload the
page, also a lot of fails, but slightly different - e.g. I might get the file
which was 404 earlier. The data requests also fail randomly, some of them are
successful, which means that the webapp is loaded.
So, answering to the questions above: the process is definitely running, socket
is definitely listening, telneting Tomcat gives random results, JConsole shows
nothing uncommon.
Can you attach the access log that shows these requests?
I do not have the access logs at my disposal to attach them. From what I
remember, the access logs were clean - no 404 codes, every request is
responded. In other words if you look from the server side (logs,
JConsole, Windows logs) , everything is fine.
Also, a couple more follow up questions…
1.) When this occurs does it only occur for the application that you reload?
or does it happen for all the apps that you have deployed to the Tomcat server?
It happens to all the webapps on the server, even "pre-installed" ones
(manager and ROOT).
2.) Is this application specific, in other words does this only occur when you
reload one particular application? or does reloading any application on the
Tomcat server cause the problem?
It is random. One of the application, when reloaded, causes the problem
almost for sure. Others can also cause the problem, but less frequently.
3.) Can you replicate this problem in a testing environment?
No, I can not.
Other 4 (production) servers also become unstable but in a different
manner - a lot of HTTP 500 responses (logs are also clean).
It would be very odd to see nothing in the logs after a 500 error. Have you looked at
all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file
should contain a stack trace after a 500 error. Also, what do the see in your browser
when you get the 500 error? The default error page should print the stack trace for the
500 error.
That is what I am talking about - quite odd behaviour. Nothing in log files, no
body with HTTP 500 response in a browser.
Plus on
some of these servers I have a lot (less than HTTP 500 responses
though) of NPEs:
-------------
SEVERE: Exception Processing /path/here
java.lang.NullPointerException
at
org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
at
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
-------------
These exception occurs for any webapp, not only the one being reloaded.
What's the specific version of Tomcat where you saw this error occur? Is it
7.0.37? or was this on an older version?
This was on 7.0.35
All the above happens irregularly, quite often on the pre-production
server, only once on production servers. Tomcat service restart fixes
the problem. Updating Tomcat to 7.0.37 did not help.
I cannot reproduce this on any of developers hosts with the same
versions of JDK and Tomcat.
Any ideas?
1.) An easy temporary fix would be to not reload the application in production.
If you need to redeploy or reload the application, stop the server, undeploy,
redeploy and start your server.
2.) Include your Tomcat configuration, minus comments.
Please see below:
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
/>
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
<!-- A couple of custom resources here -->
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="80" protocol="HTTP/1.1"
asyncTimeout="20000"
URIEncoding="utf-8"
acceptorThreadCount="2"
compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
compression="on"
You might want to disable compression and see if that makes a difference. I
can't say that this is specifically causing the problem, but it would allow you
to rule it out as a possibility.
Thanks, I will try disabling compression, but, as It might not help,
that is not enough for me. Servers are located in a data centre with
limited access and I should have some strong ideas of how to fix the
problem (or detect its cause) when I go there next time.
As I have said, everything worked fine with JDK 1.6. Can this be because
of some unexpected behaviour of Oracle JDK 1.7 classloader for win64?
Can it be because class files of the most of the apps were build with
JDK 1.6 and now running on Java 7? Can it be because some apps may leak
threads and ThreadLocals and Java 6 JVM could handle it but Java 7 JVM
cannot?
Thanks,
Kirill
Dan
connectionTimeout="20000"
maxThreads="1024"
processorCache="512"
redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b %Dms"
resolveHosts="false"/>
</Host>
</Engine>
</Service>
</Server>
Best regards,
Kirill
3.) See troubleshooting steps above.
Dan
Thanks a lot in advance,
Kirill
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
