Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
André,
On 4/21/15 10:56 AM, André Warnier wrote:
Thomas Boniface wrote:
The file descriptor peak show up in our monitoring application.
We have some charts showing the number of file descriptors owned
by the tomcat process (ls /proc/$(pgrep -u tomcat7)/fd/ | wc
-l).
The calatalina.out log shows errors, the most frequent being a
java.io.IOException: Broken pipe.
[..]
A "broken pipe", from the server perspective while sending a
response to the client, is a rather usual thing. It usually means
that the (human) client got tired of waiting for a response, and
clicked somewhere else in the browser (maybe a "cancel" button;
maybe he closed the window; etc..).
In this case, though, the client is nginx and not a human at a browser.
If the browser severs the connection to nginx, I'm not sure what nginx
does with the connection to Tomcat.
Nginx has no way to know that the client dropped the connection (the client-receiving part
of it), until Nginx tries to send some data (presumably coming from Tomcat) to the client
browser and finds no listener anymore. When that is the case, presumably Nginx closes its
own receiving part connected to Tomcat, which propagates the error to Tomcat.
(Buffering of all kinds neglected here).
I would expect that it either
cleans it up nicely (e.g. drains the bytes from the connection, then
closes), or just drops the connection to the back-end Tomcat (which
might be more efficient if Tomcat is expected to send relatively large
responses).
I don't know how nginx works when acting as a proxy. Does it use HTTP
keep-alive and process many requests through a single connection
(possibly not all from the same end user), or does it make and close
many connections?
I don't know how Nginx works precisely, but it must have all kinds of settings to tune
such behaviours in function of the circumstances. If the back-end Tomcat application
works under a Windows NTLM-like authentication mechanism e.g., then using different
connections for the same client (or vice-versa, sharing some connections between different
clients) would play havoc with said AAA mechanism, which is connection-oriented.
This seems to say that Nginx, by default, buffers the entire back-end server response
before starting to send it to the client :
http://nginx.com/resources/admin-guide/reverse-proxy/
But it also says that this can be tuned, and even disabled.
It also hints at the fact that even if the client specifies keep-alive with Nginx, nginx
itself, when dealing with the back-end server, disables the keep-alive (Connection: close).
This probably makes sense, in a scenario where the client may think that all responses
come from the same back-end server, but Nginx in the middle distributes the requests to
several back-end servers. It would make no sense in that case to use keep-alive with the
back-end servers, which may only ever see one request each from that client.
If it makes and closes many connections, Tomcat won't hang up the
phone unless some kind of timeout occurs.
Thomas, I'd advise you to do the following:
1. Check the nginx configuration. Specifically, the keep-alive and
timeout associated with the proxy configuration.
2. Make sure that Tomcat's timeouts are appropriate for those matching
settings in nginx.
It's common for users to misconfigure httpd+Tomcat by settings
different timeouts on either side of the connection, and the result is
many broken pipe or similar errors on the Tomcat side.
I'll +1 all that in any case.
The OP has a complex setup, where we are not even sure that the various connections in
various states are even related directly to Tomcat or not.
Graphically, we have this :
client <-- TCP --> nginx <-- TCP --> Tomcat <--> webapp <-- TCP --> external
servers
The output of netstat shows all the connections and their state, at the OS level. Even
assuming that nginx runs on a separate host, that still leaves the possibility that most
of the connections in CLOSE_WAIT state for example, would be connections between the
webapps and external servers, having not much to do with Tomcat per se.
But of course they use fd's and resources, just like the others. And for "lsof", they
would appear as "belonging" to the Tomcat process.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
