Kees Jan Koster wrote:
Dear André,
Assuming that your client is really connecting to that HTTP connector on port
8080 mentioned above..
Yes, it has a forwarded port 80 (using FreeBSD ipfw) that also points to 8080,
and there is an Apache with mod_proxy_http that hooks into 8081. My tests are
on the vanilla port, though.
Can you be a bit clearer on this part ? Do you see the problem happening for 1 in 10
posts, when your client connects directly to Tomcat's HTTP port 8080 ?
Or is it only when the client connects to Tomcat via either one of these intermediate
pieces of machinery ?
1) You are getting a
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
so this appears to happen when/while your java client is reading the response from the
server, and it appears to be that the client is expecting to be able to read more data,
but finds itself unable to, because the socket has been closed "under his nose".
The reading is one area I need to look into: did the client get all data,
partial data or none at all. I need to experiment with that.
You say that it happens "frequently", so it's not always.
Indeed, not always. About 1 in 10 posts die like this on bad days. Sometimes
hours with no issues. No pattern I can discern.
2) the server itself seems unaware that there is a problem. So it has already
written the whole response back to the client, decided it was done with this
request, and gone happily to handle other things.
Precisely.
That can happen, even if the client has not yet received all data, because between the
server and the client there is a lot of piping, and the data may buffered at various
levels or still "in transit".
thus..
- either the client is misinterpreting the amount of data that it should be
reading from the server's response (trying to read more than there actually is)
(on the other hand, I think that the kind of exception you would get in that case would
be different, more like "trying to read beyond EOF" or so).
- or something in-between the server and the client closes the connection
before all data has been returned to the client (and/or is loosing data).
It would be helpful to know if this happens when the response is particularly
large, or small, or if it is unrelated to the response size.
The response is a few bytes. I think it is about 10-20 bytes. Less than a
packet, I expect. :)
That is quite strange, I think.
See below.
If the server is configured with an AccessLogValve, you should be able to see
how big the response was, in bytes. If you have control over the client code,
you should be able to add something that logs how many bytes it has read before
the exception occurs.
What makes the request size interesting? What previous experience are you
basing this question on?
Just that intuitively, if a problem happens while reading the response, one would expect
that the larger the response is, the more likely that some network issue would show up in
the middle.
But now that I say this, going back to your initial message and the stacktrace
in it, I see
..
at sun.net.www.http.HttpClient.parseHTTPHeader
...
so the problem seems to show up right away, while the response's HTTP *headers* are being
read. So it looks like when the problem happens, the client is not able to read anything
at all, not even the headers..
Do all problems show up the same stacktrace, all with a problem while reading/parsing the
response headers ?
Dumping the response HTTP headers to the client logfile would also help finding out what
happens. (If the client is an applet running inside of a browser, then a browser add-on
would show this easily (like "Live HTTP Headers" for Firefox, or Fiddler2 for
IE)).
I can check that I see the same problems from a browser using firebug, that is
a good idea. Thanks.
Doing a "traceroute" from the client to the server, may also give an idea of
what there is actually between the server and the client.
mtr reports no packet loss between the two machines I used for testing.
Actually, I was more thinking about some intermediate problematic proxy or
something.
But a traceroute or similar would not show that.
See the first question above, about the direct/indirect connection
client-server.
And if this all still does not provide any clues, then you're down to a network
packet trace, using Wireshark or similar.
Packet traces I was hoping to avoid. :(
So far it smells to me like there is some network issue, with some intermediate software
or hardware part which is dropping the connection between client and server, after the
client has sent the request, but before it even starts receiving the response.
Is there anything in-between client and server which could have this behaviour, such as
when it gets very busy ?
Do you have any kind of tool which can show you how many requests Tomcat is processing
over time, and if these problems happen when it is handling lots of requests ?
(Not that the problem appears to be at the Tomcat level, but just to check how busy the
network may be at such times)
Another thing : your client is effectively requesting non-keepalive connections, so Tomcat
will close the connection after sending the response to each request. And your clients
have to rebuild a new connection for each request.
If the same client(s) make lots of small requests one after another, this may be
counter-productive, because each connection build-up requires several packets going back
and forth. Also, on the server side, when a connection is being closed, it will
nevertheless "linger" for a while in CLOSE_WAIT state, waiting for the client's TCP stack
to acknowledge the CLOSE. I have seen cases where a large number of such connections
being in CLOSE_WAIT triggered bizarre issues, such as a server becoming unable to accept
new TCP connections for a while.
It may be worth checking how many of such CLOSE_WAIT connections you have over time, and
if this relates to when the problems happen.
netstat -pan | grep CLOSE_WAIT
would show this. If more than a couple of hundreds show up, I'd become suspicious of
something like that.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
