On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be
something that would be left on in a live> system. We can set these
parameters, so it would be useful to know if we
were hitting the set limits.
For the connection limit:
How timely do you need the information to be? It is much easier to add a
check when the connection is closed since a) it returns the current
count and b) we are already checking the return value for mis-counting.
For the thread limit:
We have a couple of options here. (We use a custom queue which
effectively changes the way threads in the pool re used.) The simplest
is probably to add log in TaskQueue.offer() if all threads are currently
being used.
I'm not sure I fully grasp how this additional logging presents a
significant incremental DOS risk. I mean, if an attacker is flooding you
with enough traffic or connections where this becomes an issue, aren't you
already logging various aspects of the attempts anyway (e.g., access logs,
etc)? At that point aren't your logs already being filled anyway?
It is an area we need to be careful with. Log messages that include
exceptions are of greatest concern since they are relatively expensive
to generate. If an invalid HTTP message triggers a log message with an
exception (it does, but only when configured for debug logging) then
there is a degree of DoS risk there.
Access logs are a special case and one that we have taken care to try
and optimise. Still, if you want to make a performance test look good
turning off access logging is one of the first things to do (and of
course results in totally meaningless results that don't reflect real
world usage).
I think the concern Chris was trying to express (although I'm sure he'll
correct me if I am wrong) is that when these limits are reached there is
a risk of a lot of log messages being generated and that can cause
issues both from a performance issue and from just filling the logs with
the same message over and over again.
I think what we would need is some form of hysteresis. Log a message
when the limit is reached, log another message when usage falls below
80%? of the limit. That way, administrators see the peaks in usage but
aren't inundated with log messages.
My only concern is that at least some this code is going to run for
every single request. It needs to be efficient. We should measure the
impact of adding this code before we decide on whether to include it.
Mark
On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:
Baron,
On 4/9/24 16:33, Baron Fujimoto wrote:
I'm investigating occasional 503 errors for our CAS service running in a
Tomcat 10.1.x container. The 503s appear to correlate with some traffic
spikes at the same time.
The connector is configured as follows:
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="8443"
maxThreads="2500"
maxConnections="50000"
maxPostSize="100000"
maxParameterCount="1000"
scheme="https" secure="true"
SSLEnabled="true"
>
Can Tomcat log info such as when the maxThreads or maxConnections limits
are reached? I'm basically trying to see if there is a good way to
more definitively determine what may have caused the 503s and what may be
feasible to mitigate them.
Are you thinking of a debugging feature or something to be left-on for a
live production system?
Such logging might be considered a DOS vector for a live system: you can
fill the logs by asking lots of trivial requests.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
