>From our perspective, it needn't be super timely. It would be more for forensic confirmation that there's something we should consider. I think a hysteresis behavior would be compatible with this.
On Mon, Apr 15, 2024 at 12:00 AM Mark Thomas <ma...@apache.org> wrote: > On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be > something that would be left on in a live> system. We can set these > parameters, so it would be useful to know if we > > were hitting the set limits. > > For the connection limit: > > How timely do you need the information to be? It is much easier to add a > check when the connection is closed since a) it returns the current > count and b) we are already checking the return value for mis-counting. > > For the thread limit: > > We have a couple of options here. (We use a custom queue which > effectively changes the way threads in the pool re used.) The simplest > is probably to add log in TaskQueue.offer() if all threads are currently > being used. > > > I'm not sure I fully grasp how this additional logging presents a > > significant incremental DOS risk. I mean, if an attacker is flooding you > > with enough traffic or connections where this becomes an issue, aren't > you > > already logging various aspects of the attempts anyway (e.g., access > logs, > > etc)? At that point aren't your logs already being filled anyway? > > It is an area we need to be careful with. Log messages that include > exceptions are of greatest concern since they are relatively expensive > to generate. If an invalid HTTP message triggers a log message with an > exception (it does, but only when configured for debug logging) then > there is a degree of DoS risk there. > > Access logs are a special case and one that we have taken care to try > and optimise. Still, if you want to make a performance test look good > turning off access logging is one of the first things to do (and of > course results in totally meaningless results that don't reflect real > world usage). > > I think the concern Chris was trying to express (although I'm sure he'll > correct me if I am wrong) is that when these limits are reached there is > a risk of a lot of log messages being generated and that can cause > issues both from a performance issue and from just filling the logs with > the same message over and over again. > > I think what we would need is some form of hysteresis. Log a message > when the limit is reached, log another message when usage falls below > 80%? of the limit. That way, administrators see the peaks in usage but > aren't inundated with log messages. > > My only concern is that at least some this code is going to run for > every single request. It needs to be efficient. We should measure the > impact of adding this code before we decide on whether to include it. > > Mark > > > > > > On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> Baron, > >> > >> On 4/9/24 16:33, Baron Fujimoto wrote: > >>> I'm investigating occasional 503 errors for our CAS service running in > a > >>> Tomcat 10.1.x container. The 503s appear to correlate with some traffic > >>> spikes at the same time. > >>> > >>> The connector is configured as follows: > >>> > >>> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" > >>> port="8443" > >>> maxThreads="2500" > >>> maxConnections="50000" > >>> maxPostSize="100000" > >>> maxParameterCount="1000" > >>> scheme="https" secure="true" > >>> SSLEnabled="true" > >>> > > >>> > >>> Can Tomcat log info such as when the maxThreads or maxConnections > limits > >>> are reached? I'm basically trying to see if there is a good way to > >>> more definitively determine what may have caused the 503s and what may > be > >>> feasible to mitigate them. > >> > >> Are you thinking of a debugging feature or something to be left-on for a > >> live production system? > >> > >> Such logging might be considered a DOS vector for a live system: you can > >> fill the logs by asking lots of trivial requests. > >> > >> -chris > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
