Thanks for the replies. Here’s some more info.
> Bruce,
>
> On 3/5/14, 1:11 PM, Bruce Weertman wrote:
>> We have a load balancer talking with tomcat’s running on multiple
>> backend tomcat servers. The tomcats are running web service
>> applications. In doing testing, and as reported by customers, we
>> occasionally see refused connections. Not real often, but enough
>> to be a head ache.
>>
>> We’re not sure where the problem is coming from. Suspects include
>> (1) firewall, (2) load balancer (3) tomcat and perhaps the (4) the
>> web app. The volume of traffic is so high, that it becomes a
>> needle in a hay stack issue.
>
> Can you describe your setup in a bit more detail? What kind of lb? How
> many of them? Any fronting web server(s)? How many? How many Tomcat
> instances?
There’s no front end servers. We used to do that via AJP, mod_jk and apache
but now we just have a single generic http load balancer. If a backend
it unavailable, it will send traffic to another server. We have ~5 instances
give or take.
Everything’s Linux except osx for development and the testing (see below).
> What is/are your <Connector> configuration(s)?
Pretty simple, nothing much to see here accept the keep alive
<Connector port=“8080" protocol="HTTP/1.1"
connectionTimeout="20000"
maxKeepAliveRequests="1"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,application/xml"
redirectPort="8443" bindOnInit="false”/>
> Tomcat usually complains if you run out of threads.
I don’t see that this is true.
Here’s a simple test to illustrate
I set up a tomcat with:
<Connector port=“8080" protocol="HTTP/1.1"
connectionTimeout="20000"
maxKeepAliveRequests= "1"
maxThreads="5"
acceptCount="1"
redirectPort="8443" bindOnInit="false”/>
I then have very simple JSP that looks like this.
<%@page contentType="text/plain" pageEncoding="UTF-8"%><%
long t0 = System.currentTimeMillis();
Thread.sleep(5000l);
long t1 = System.currentTimeMillis();
%>Slept for <%= t1 - t0 %> milliseconds
I then have a very simple BASH script that looks like:
#!/bin/ti
for i in {0..20}
do
wget —tries=1 -O file${i}.txt "http://localhost:8080/myapp/sleep.jsp"; &
done
After running this file, I end up with something like 8 files successfully
downloaded.
Naively, I would think that I’d get 6. Changing localhost to the host name,
I do in fact get 6. So going to localhost must have a connection queuing effect
(at the OS level) that is different from going through the interface card.
The point is, the thread limit did it’s thing. Requests were rejected.
LOOKING in the logs directory, specifically catalina.out, I do not see
any indication that tomcat ran out of threads! It says nothing about
rejected connections! EEEK. Bad stuff could be happening all the time
and I would never know. What am I missing? There’s got to be some
way to see this stuff.
>
>> In production maxConnections is the default of 200, but when
>> looking at the number of connections being processed by tomcat it’s
>> rarely more than a couple dozen per instance. There’s always the
>> possibility that it occasionally spikes.
>
> What is the resolution of the data you have? Are you sampling with
> some regularity?
It’s really just spot checking using JMX and looking at the tomcat
server-status page.
>
>> Anyway, the question is: Is there any good way to see if tomcat
>> is going into a state where it’s not accepting connections? How
>> would would I see this? Is it possible to see this ?
>
> Sure: connect to a specific Tomcat. If you get a "connection refused",
> then you are in this state.
Well of course, but this doesn’t happen very often.
>
> Your load balancer should have a setting where you can choose a
> back-end server explicitly if you want to go through the lb, or you
> can just contact one directly.
Of course, I can talk to back ends directly or not.
>
> - -chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
