On 27.11.2016 13:23, Jaaz Portal wrote:
hi Andre,
thank you very much this was very educative but in my case it is little
bit
different.
The server is no flooded, there is maybe dozen of very sophisticated
connections that somehow hangs apache workers threads
Can you be a bit more specific ?
When you say "apache workers threads", do you mean threads in Apache
httpd, or threads in Apache Tomcat ? (both are Apache webservers, so it is
difficult to tell what you are talking about, unless you are very precise).
Let me give you some additional explanations, and maybe you can figure out
exactly where it "hangs".
From the Apache httpd front-end point of view, mod_jk (the connector to
Apache Tomcat) is basically one among other "response generators". Apache
httpd does not "know" that behind mod_jk, there is one or more Tomcat
servers. Apache httpd receives the original client request, and depending
on the URL of the request, it will pass it to mod_jk or to another response
generator, to generate the response to the request.
That mod_jk in the background is using a Tomcat server to actually
generate the response, is none of the business of Apache httpd, and it does
not care. All it cares about, is to actually receive the response from
mod_jk, and pass it back to the client.
If httpd passes a request to mod_jk, it is because you have specified in
the Apache configuration, the type of URL that it should pass to mod_jk.
That happens via your "JkMount (URL pattern)" directives in Apache httpd.
Of course Apache httpd will not pass a request to mod_jk, before it has
received at least the URL of the request, and analysed it to determine *if*
it should pass it to mod_jk (*).
If the mod_jk logging is enabled, you can see in it, exactly *which*
requests are passed to mod_jk and to Tomcat.
Do you know *which* requests, from which clients, cause this "thread
hanging" symptom ?
Once you would know this, maybe you can design a strategy to block
specifically these requests.
and the effect is permanent. Quickly the pool is exhausted
which pool exactly ?
and the only
solution that works is to restart tomcat.
I think it is a bug similar to this one from mod_proxy:
https://tools.cisco.com/security/center/viewAlert.x?alertId=34971
Maybe, maybe not. As long as we do not know what the requests are, that
block things, we do not know this.
I think also that your solution with setting connectionTimeout will solve
the problem, at least partially. THANK YOU.
Same thing, we do not know this yet. It was only giving this explanation,
to help you think about where the problem may be.
I would like to help you further investigate this issue, as our server
comes under such attack once or twice in a week.
Other than giving you hints, there is not much I or anyone else can do to
help. You are the one with control of your servers and logfiles, so you
have to investigate and provide more precise information.
(*) actually, to be precise, Apache httpd passes *all* requests to mod_jk,
to ask it "if it wants that request". mod_jk then accepts or declines,
depending on the JkMount instructions. If mod_jk declines, then Apache
httpd will ask the next response generator if it wants this request, etc...
best regards,
artur
2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>:
Hi.
Have a look that the indicated parameters in the two pages below.
You may be the target of such a variant of DDoS attack : many clients
open
a TCP connection to your server (front-end), but then never sends a HTTP
request on that connection. In the meantime, the server accepts the TCP
connection, and passes it on to a "child" process or thread for
processing. The child then waits for the HTTP request line to arrive on
the connection (during a certain time), but it never arrives. After a
while, this triggers a timeout (see below), but the standard value of
that
timeout may be such that in the meantime, a lot of other connections have
been established by other such nefarious clients, so a lot of resources
of
the webserver are tied up, waiting for something that will never come.
Since there is never any real request sent on the connection, you would
(probably) not see this in the logs either.
The above is the basic mechanism of such an attack. There may be
variations, such as the client not "not sending" a request line, but
sending it extremely slowly, thus achieving perhaps similar kinds of
effects.
As someone pointed out, it is quite difficult to do something about this
at the level of the webserver itself, because by the time you would do
something about it, the resources have already been consumed and your
server is probably already overloaded.
There are specialised front-end devices and software available, to detect
and protect against this kind of attack.
You may want to have a look at the following parameters, but make sure to
read the caveats (side-effects, interlocking timeouts etc.), otherwise
you
may do more harm than good.
Another thing : the settings below are for Apache Tomcat, which in your
case is the back-end. It would of course be much better to detect and
eliminate this at the front-end, or even before. I had a look at the
Apache httpd documentation, and could not find a corresponding parameter.
But I am sure that it must exist. You may want to post this same question
on the Apache httpd user's list for a better response.
Tomcat configuration settings :
AJP Connector : (http://tomcat.apache.org/tomc
at-8.5-doc/config/ajp.html#
Standard_Implementations)
connectionTimeout
The number of milliseconds this Connector will wait, after accepting a
connection, for the request URI line to be presented. The default value
for
AJP protocol connectors is -1 (i.e. infinite).
(You could for example try to set this to 3000 (milliseconds) or even
lower. That should be more than enough for any legitimate client so send
the HTTP request line. Note however that by doing this at the Tomcat
level, you will probably move the problem to the Apache httpd/mod_jk
level. But at least it might confirm that this is the problem that you
are
seeing. The mod_jk logfile at the httpd level may give you some hints
there.)
HTTP Connector : (http://tomcat.apache.org/tomc
at-8.5-doc/config/http.html#Standard_Implementation)
connectionTimeout
The number of milliseconds this Connector will wait, after accepting a
connection, for the request URI line to be presented. Use a value of -1
to
indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60
seconds) but note that the standard server.xml that ships with Tomcat
sets
this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to
false, this timeout will also be used when reading the request body (if
any).
On 26.11.2016 09:57, Jaaz Portal wrote:
hi,
sorry, its mod_jk no jk2, my typo. All at latest versions. We tried with
mod proxy too.
There is no flood of the server. Nobody is flooding us, they use some
specific connections after which pool of apache workers is exhausted and
blocked
and we need to restart tomcat server.
It is some kind of exploit but do not know how to log it to obtain
details.
i had put a limit on connections per client with hope that this will
help
but once again, it is not a flood.
They open several connections that are not dropped by apache when they
disconnect. This way whole pool is quickly exhausted and the server
broken.
i would like to help you to figure details of this attack but this is
production server so it is impossible to much debugging options
best,
artur
2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu <
niranjan.bo...@gmail.com
:
you can find who is flooding site in apache access.log and block them in
firewall.
ex to find the IP:
cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort -gr
On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal <jaazpor...@gmail.com>
wrote:
hi,
we are from some weeks struggling with some Polish hackers that are
bringing our server down. After updating apache to latest version
(2.4.23)
and tomcat (8.0.38) available for debian systems we still cannot secure
our
server.
Today it has stopped to respond again and we needed to restart tomcat
process to get it back alive.
There is no too much clues in the logs. The apache error.log gives
just
this line:
[Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
1397934896385
92] AH00484: server reached MaxRequestWorkers setting, consider
raising
the
MaxR
equestWorkers setting
seems that somehow tomcat, mod-jk2 or even apache is vulnerable to
some
new
exploit, as we certainly does not have such traffic that would block
our
server otherwise
for now we have increased MaxRequestWorkers and we have limited number
of
connections from one client to 5 by mod_bw and limited number of
simultaneous connections from one ip by iptables but does not know if
this
will help
best regards,
artur
--
*Thanks*
*Niranjan*
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
