Paul Smith wrote: > Unfortunately it seems to be down / under attack again this morning :(
I see that my attempts at mitigation of the current problem are failing. I am looking into things again now. I'll type this in stream of consciousness as I work the problem and then send it. I may have fixed the insufficient mitigation for the apache web server wedging up. I had thought that "apachectl graceful" should be sufficient to restart the apache server. But no it was not. The apache logs were filled with attempts to restart using that command and obviously it wasn't working. It appears that what is happening is that external client agents are connecting to the apache web server and hanging onto connections forever. This causes apache to spawn up to the MaxRequestWorkers number and hold there. And it doesn't matter the number. Eventually all server processes are consumed regardless of the number. Processes respond to other requests until the malicious agent eventually grabs them and holds onto them. That's when no one else can get response from the site. Meanwhile that number must be below the max memory available for use. If it is larger than available then of course that's bad and we must start swapping. I reduced it from the large 255 value to 32 yesterday on the previous pass at this tuning. The graceful action politely waits for the client to finish processing. Which in this case never occurs because the abusive agents are not polite and hold on forever. Therefore using graceful is insufficient. That is why it requires the forced stop action to have apache stop talking to the abusive agents forcibly and to drop all connections. And of course then a start works. I changed that to be a full "apachectl stop" and then followed by a "apachectl start" and that worked. Rather than simply restarting I changed the automatic mitigation test and let it work and so tested that it would automatically detect and restart okay. But I am going to configure the "restart" action. It makes me nervous that the automation triggers "stop;start" immediately with no delay. I worry that it might take a moment for the stop to propagate and all of the old apache server processes to actually exit. And if they don't exit then the restart action can't succeed. It's a race condition due to the asynchronous behavior of systemd. Though the automation would trigger again and that one might succeed if the previous action failed. Feels like the apachectl restart would handle this action more reliably since that instructs the supervisor process to perform the restart. Will configure it while the current malicious agents are active to help us test for it. [[ In this investigation I see that on systemd systems there is an "interesting" control loop. apachectl calls systemd and systemd calls apachectl. This would be a loop except for a programmed in internal variable that breaks the loop. Isn't it a wonderful world that we now live in? Not! ]] If the restart action proves insufficient (though I think it will work) then I will script up a stronger more explicit restart action. Let's hope that isn't needed. Additionally I tuned the apache server to have a smaller number of MaxRequestWorkers than the default 255 which is more than the memory available. I have it currently set to a more conservative 32. Though now that things are observable again it appears that each process uses about 25MB of memory for about 800MB of memory consumed in this configuration. We could increase that number somewhat and use more memory for server processes. Increasing this would eat into available file system buffer cache though. Life is a tradeoff. But this seems pretty responsive right now. I'll leave it this way for a while and observe. Additionally I configured MaxConnectionsPerChild to be 100 rather than unlimited. This will cause apache to restart server processes after that number of client requests have been handled. That's just a useful "random bug" guard restarting processes ever so often rather than never. > Don't people have better things to do with their lives? Sigh. It's why we can't have nice things. :-( Bob