-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jian
(Replying to both you and the list)
On 4/28/15 1:52 PM, Jian Wu wrote:
> I don't know how to reply to my question directly because I
> subscribed to the digest mail list.
I'm not sure, either. I suspect that replying to the digest message
will reach the list. What you did was okay, too. Consider joining the
real-time mailing list. You can always throw the messages into a
folder so they don't choke-up your inbox.
> First, thank you very much for replying to my question. To clarify
> my question, the load balancer, and three web servers are on
> different virtual machines. I use heartbeat-ldirectord for loading
> balancing, and Tomcat to deploy my web application (not httpd).
>
> --- virtual machine1 (tomcat instance 1) | --> load balancer ---
> virtual machine 2 (tomcat instance 2) | --- virtual machine 3
> (tomcat instance 3)
>
> All the three Tomcat instances are the same.
Okay, this is good information.
> I know 2000 is a lot, but our site receives more than 2 million
> hits per day (including crawlers), and sometimes, it is easy to
> reach to the default value, which makes server unreachable. That is
> why we set it to 2000. But even though, sometimes, this limit was
> reached. I don't know why.
>
> The BRecv and BSend are stats shown by Tomcat manager under
> "Server Status". To be more clear, I attach a screen shot.
[The screenshot just shows a sample of threads on the lb. Also
includes this text:
Max threads: 1000 Current thread count: 228 Current thread busy: 42
Keeped alive sockets count: 1
Max processing time: 3453257ms Processing time: 2726939.0s Request
count: 62453 Error count: 14423 Bytes received: 0.27 MB Bytes sent:
51310.35 MB]
What do those values represent? The stats on the lb? Or the stats on a
single (or multiple) Tomcat instance?
> I also send you the <Connectors> section in my server.xml file
> (same for all Tomcat instances).
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="20000" redirectPort="8443" maxThreads="1500"/>
You said that you had maxThreads="2000" but it's set to 1500 here.
- From the screenshot, it looks like maxThread="1000". Can you clarify
the mismatch between these three values?
What is your peak load, in terms of simultaneous connections? How is
the lb deciding whether a node is available, and how to distribute load?
With a completely even load, 2M requests per day is only ~1400
req/sec, and you have the capacity to handle 4500 simultaneous
requests (3 * 1500, as configured above). What is your average
response rate? IF you can respond within 1000ms, then you should be
able to handle that amount of load without a problem.
Now... if you have bursty-load -- which is usually the case -- it's
important to know what the maximum you can expect might be. For
instance, perhaps you get all 2M of those requests during regular
business hours in your time zone. That means that you will be handling
~4200 req/sec. Under that kind of load, can you still respond within
1000ms? You still have enough threads to handle that load, but if they
can't respond fast enough, you'll slow down to a halt.
The nice thing about Nio is that it can accept lots of connections and
only allocate a processing thread once the request line and headers
have been read. But, if you accept too many requests, you can make it
hard to process the load.
So the first thing I'd like to know if whether your 3-node cluster is
capable of handling the load you actually have. If so, you ought to be
able to configure it appropriately. If not, no amount of configuration
is going to fix your problem.
As for configuration, I have a few suggestions:
1. The default maxConnections for the NIO connector is 10000, which
might be too low for you.
2. If your lb can use HTTP KeepAlive with the backend Tomcat
instances, you'll probably improve your performance.
3. Especially if you have multiple <Connector>s (but even if you
don't), you might want to consider using an <Executor>:
<Connector executor="myExecutor"
... /?
<Executor name="myExecutor"
namePrefix="catalina-exec-"
maxThreads="1500"
minSpareThreads="500"
/>
If you have more than one <Connector> you can share the <Executor>
between them. This will allow you to use fewer threads to handle your
load, rather than having 1500 threads available for one <Connector>
and some other (possibly large) number of threads for another <Connector
>.
Now, if you want to have an administrative <Connector> that can't be
starved by "regular" requests, then you'll want to use a separate
<Executor>.
4. Your connectionTimeout is 20000 which is a looong time. For a
high-volume site, you should probably lower that value. The
load-balancer shouldn't be connecting to you and then not doing
anything: it should be sending the request pretty much immediately.
So, consider lowering that timeout a bit.
If you are using HTTP KeepAlive, then it *does* make sense to have a
long timeout, because the lb will have a (somewhat) persistent
connection to the Tomcat node and might not get a request every 20
seconds or so. 20s seems okay to me for a lb using HTTP KeepAlive.
Just make sure that HTTP KeepAlive is actually being used.
5. The connectionTimeout on Tomcat should be the same as the timeout
configured on the load-balancer for HTTP KeepAlive. That means that
the lb and Tomcat will both hang-up the connection at the same time if
the lb doesn't get an incoming request from a client before the
KeepAlive timeout expires.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVP87MAAoJEBzwKT+lPKRYU7kP/0er7EgqtrsT2MMkFlQfMJwq
tLzRip3WPabSpE7FXUlI8TCNr1XJRfsiG9z00qoJ3LghtzyNgTDvpoNZisQIhaUo
XYzq0G925NbepPSFnFWg2uG3dafvf9utBLy2b1ZcaqWIuu5X6GoYi3e/lisBSmjy
a/Q9e+a2q6manwV2rFHuRhTktuOJSE+TVHdhL1xy4n6/aV4e/kW+/Wv+H23pWhSe
ZXol1OHxK5GeRETfiAAwWvT4eUx/FuGLNrRMH4CJ1J75dD6SkXJFoMtJxNOo9bpt
TI0/6PgxwlX2gy5BAc4T3QecS0lV7kNoJUpOwrJSLuh4UwasfSolth8OZr0ogtR/
Eg3AkttTvtlnKZJ57XgDr4ORk+tmGG/TpoqcqLbjYXajl/R5xiVbzXR+XRJAO9GB
8VJAksSg99m3I+RsUAGeKL5lxEA9KyJAd2F/7bGyNtRmxF7TzjmIKbimkMbcRhC8
nkJr53OpoDumLApUIouIRtGmr4Xw0YjevUcyn7c4KhxLecxRSB/cAvZFXNUjsaJn
IVlXXteyRPv7MCgb3eUlh9PDgCPe0Q0B/pLrcKzPyA9eVWGHSZmqo8ABgrE3t8mI
6RYPFj+ZCcbSaTyuhVopU2OjjFMDZdvsj3wfi5VFgV8hz45gjG++R+Lj/KAdThNs
+C+TE1ESUkXyXsyYIJj8
=t/KD
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
