Thanks, Mark. Submitted < https://bz.apache.org/bugzilla/show_bug.cgi?id=68934>
On Tue, Apr 16, 2024 at 4:44 AM Mark Thomas <ma...@apache.org> wrote: > It would be worth creating an enhancement request for this in Bugzilla > to ensure the request doesn't get forgotten about. > > Mark > > > On 16/04/2024 01:06, Baron Fujimoto wrote: > > From our perspective, it needn't be super timely. It would be more for > > forensic confirmation that there's something we should consider. I think > a > > hysteresis behavior would be compatible with this. > > > > On Mon, Apr 15, 2024 at 12:00 AM Mark Thomas <ma...@apache.org> wrote: > > > >> On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be > >> something that would be left on in a live> system. We can set these > >> parameters, so it would be useful to know if we > >>> were hitting the set limits. > >> > >> For the connection limit: > >> > >> How timely do you need the information to be? It is much easier to add a > >> check when the connection is closed since a) it returns the current > >> count and b) we are already checking the return value for mis-counting. > >> > >> For the thread limit: > >> > >> We have a couple of options here. (We use a custom queue which > >> effectively changes the way threads in the pool re used.) The simplest > >> is probably to add log in TaskQueue.offer() if all threads are currently > >> being used. > >> > >>> I'm not sure I fully grasp how this additional logging presents a > >>> significant incremental DOS risk. I mean, if an attacker is flooding > you > >>> with enough traffic or connections where this becomes an issue, aren't > >> you > >>> already logging various aspects of the attempts anyway (e.g., access > >> logs, > >>> etc)? At that point aren't your logs already being filled anyway? > >> > >> It is an area we need to be careful with. Log messages that include > >> exceptions are of greatest concern since they are relatively expensive > >> to generate. If an invalid HTTP message triggers a log message with an > >> exception (it does, but only when configured for debug logging) then > >> there is a degree of DoS risk there. > >> > >> Access logs are a special case and one that we have taken care to try > >> and optimise. Still, if you want to make a performance test look good > >> turning off access logging is one of the first things to do (and of > >> course results in totally meaningless results that don't reflect real > >> world usage). > >> > >> I think the concern Chris was trying to express (although I'm sure he'll > >> correct me if I am wrong) is that when these limits are reached there is > >> a risk of a lot of log messages being generated and that can cause > >> issues both from a performance issue and from just filling the logs with > >> the same message over and over again. > >> > >> I think what we would need is some form of hysteresis. Log a message > >> when the limit is reached, log another message when usage falls below > >> 80%? of the limit. That way, administrators see the peaks in usage but > >> aren't inundated with log messages. > >> > >> My only concern is that at least some this code is going to run for > >> every single request. It needs to be efficient. We should measure the > >> impact of adding this code before we decide on whether to include it. > >> > >> Mark > >> > >> > >>> > >>> On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz < > >>> ch...@christopherschultz.net> wrote: > >>> > >>>> Baron, > >>>> > >>>> On 4/9/24 16:33, Baron Fujimoto wrote: > >>>>> I'm investigating occasional 503 errors for our CAS service running > in > >> a > >>>>> Tomcat 10.1.x container. The 503s appear to correlate with some > traffic > >>>>> spikes at the same time. > >>>>> > >>>>> The connector is configured as follows: > >>>>> > >>>>> <Connector > protocol="org.apache.coyote.http11.Http11NioProtocol" > >>>>> port="8443" > >>>>> maxThreads="2500" > >>>>> maxConnections="50000" > >>>>> maxPostSize="100000" > >>>>> maxParameterCount="1000" > >>>>> scheme="https" secure="true" > >>>>> SSLEnabled="true" > >>>>> > > >>>>> > >>>>> Can Tomcat log info such as when the maxThreads or maxConnections > >> limits > >>>>> are reached? I'm basically trying to see if there is a good way to > >>>>> more definitively determine what may have caused the 503s and what > may > >> be > >>>>> feasible to mitigate them. > >>>> > >>>> Are you thinking of a debugging feature or something to be left-on > for a > >>>> live production system? > >>>> > >>>> Such logging might be considered a DOS vector for a live system: you > can > >>>> fill the logs by asking lots of trivial requests. > >>>> > >>>> -chris > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> > >>>> > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
