Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
i just wanted to let you know that the we have migrated to WildFly
application server
and our server is up online 24/24h from three weeks.

Since this time it has never freezed so I suppose i was right saying
somebody found DoS exploit on tomcat.

Unfortunately I cannot help you in figure causes of this freeze that
happened one or two times a week.

best wishes,
artur

2016-12-01 2:46 GMT+01:00 Christopher Schultz 
:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Jaaz,
>
> On 11/30/16 1:41 PM, Jaaz Portal wrote:
> > no it looks like dos, its dos
> >
> > i told you they dosed before bind server until we changed it to
> > other vendor, and later was scanning my host for apache
> > vulnerabilities
>
> Okay, let's just end this because obviously we've never going to get
> to the bottom of it.
>
> $ sudo iptables -A INPUT -s evil.host.pl -p tcp -m tcp -j DROP
>
> If they start coming at you from multiple IPs, use CIDR notation to
> block a whole subnet or whatever.
>
> Then when these "attacks" keep happening, you can block-out the whole
> world and you're application will never seize-up again.
>
> Review your configuration as Mark has suggested. If things are still
> not working, re-post with some actual data instead of just
> continuously posting "we are under attack, they dos bind, they dos
> mod_jk, they dos mod_proxy, what can we do?".
>
> Post. Logs.
>
> A single log line saying "server reached MaxRequestWorkers setting,
> consider raising the MaxRequestWorkers setting" is not an indication
> of any problem at all other than (lack of) accurate capacity planning.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYP4D1AAoJEBzwKT+lPKRYM6AP/2H+NEy27YU92ccllLOcnBsA
> V8RW+lGUjI/nxYqODw2B1cjE8zhnvtPRokdWIkLu0rQA4mg7p/3iLcBvdkZXMAkc
> k16aZZk+wbtTgsqK/2ORQ+5lfn3nd4QTk2tzB/z/jJnb+7vp6/qa7ESpZLXi7p3U
> TniCZz0gVrykF7NVS1Ospvw/JcEuuFeo4gMnXIupYOqF9jlI1y0HydVreIOCwgq6
> fn/4UA9Ku6iGYuPE+k8AQvgbQ3ILaODVanUPTtLQstjfXpHUVGerakclUoxIOawH
> ZkzKledtsFzLByPFdI6/Y9+WXDAVtwCKVEFNWc2lHbogy6FC+5ozubRwC5MP8CO4
> vicaCl7X+hDwDfxAZEZp1pbGrAd0G7YGgzTSKz5r61t9opv8k5HjWqGFiB2MFob1
> jedbWvE40w/d54kuy+1MUAqjH5wM1Rw4hY7glALHiNVs1z5m83YFPYQhcbGdsi31
> PxAA5OTVU8hggsbVcq9P5qjYOIQlLH+b2//K2eZK/4QV6u9u0jMuhMq/eeEy93JW
> HPb0Uri/TK+LajrJ3fsMTKhWZDS4VkCB+kkJ3BedKDnq1yQMqckJW5WKx4wgaZHA
> lxnDbxPjq4nZLI3odsG1eiU7/7yl32tPTy3b038iTacihl5hMdIFZXM5VkoKvdi8
> AGidy4ZhsZ920ZAZQqJS
> =dqVz
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jaaz,

On 11/30/16 1:41 PM, Jaaz Portal wrote:
> no it looks like dos, its dos
> 
> i told you they dosed before bind server until we changed it to
> other vendor, and later was scanning my host for apache
> vulnerabilities

Okay, let's just end this because obviously we've never going to get
to the bottom of it.

$ sudo iptables -A INPUT -s evil.host.pl -p tcp -m tcp -j DROP

If they start coming at you from multiple IPs, use CIDR notation to
block a whole subnet or whatever.

Then when these "attacks" keep happening, you can block-out the whole
world and you're application will never seize-up again.

Review your configuration as Mark has suggested. If things are still
not working, re-post with some actual data instead of just
continuously posting "we are under attack, they dos bind, they dos
mod_jk, they dos mod_proxy, what can we do?".

Post. Logs.

A single log line saying "server reached MaxRequestWorkers setting,
consider raising the MaxRequestWorkers setting" is not an indication
of any problem at all other than (lack of) accurate capacity planning.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYP4D1AAoJEBzwKT+lPKRYM6AP/2H+NEy27YU92ccllLOcnBsA
V8RW+lGUjI/nxYqODw2B1cjE8zhnvtPRokdWIkLu0rQA4mg7p/3iLcBvdkZXMAkc
k16aZZk+wbtTgsqK/2ORQ+5lfn3nd4QTk2tzB/z/jJnb+7vp6/qa7ESpZLXi7p3U
TniCZz0gVrykF7NVS1Ospvw/JcEuuFeo4gMnXIupYOqF9jlI1y0HydVreIOCwgq6
fn/4UA9Ku6iGYuPE+k8AQvgbQ3ILaODVanUPTtLQstjfXpHUVGerakclUoxIOawH
ZkzKledtsFzLByPFdI6/Y9+WXDAVtwCKVEFNWc2lHbogy6FC+5ozubRwC5MP8CO4
vicaCl7X+hDwDfxAZEZp1pbGrAd0G7YGgzTSKz5r61t9opv8k5HjWqGFiB2MFob1
jedbWvE40w/d54kuy+1MUAqjH5wM1Rw4hY7glALHiNVs1z5m83YFPYQhcbGdsi31
PxAA5OTVU8hggsbVcq9P5qjYOIQlLH+b2//K2eZK/4QV6u9u0jMuhMq/eeEy93JW
HPb0Uri/TK+LajrJ3fsMTKhWZDS4VkCB+kkJ3BedKDnq1yQMqckJW5WKx4wgaZHA
lxnDbxPjq4nZLI3odsG1eiU7/7yl32tPTy3b038iTacihl5hMdIFZXM5VkoKvdi8
AGidy4ZhsZ920ZAZQqJS
=dqVz
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache/Tomcat vulnerability

[tomcat]

Let me ask you a question : if you have nothing in the logs, and there are no connections 
to your (Apache) server on port 80, then *what exactly* makes you think that you are under 
some kind of attack ?
How do you know that it is not simply your application that is freezing up under normal 
usage, as opposed to an "attack" ?


This is a serious question, and please think before you answer. Provide some real 
information that might help us to help you. We really need some data, otherwise we may as 
well consult a crystal ball.

(It has been a long time since we have needed it, and I cannot remember where 
we left it)

Saying that "before, they DoS-ed bind", is not real information that we can do anything 
with. It may not be the same "people", it may not have anything to do with this issue that 
you are having.
Similarly, someone "scanning your server for Apache vulnerabilities" is not real 
information either. That happens to any webserver on the WWW, constantly, and they do not 
crash for that. And they generally do not scan for "Apache vulnerabilities", they scan for 
application vulnerabilities or Apache bad configuration issues.


If I had to make a guess, I would say that there are probably, today, more than 100,000 
webservers active on the WWW with an Apache httpd front-end, and an Apache tomcat 
back-end.  If 0.1% of those was crashing because of some attack, that would be 100 tomcat 
servers crashed today, and this official tomcat users list would be full of messages about it.
And that is not happening : your messages about this have been the only ones for the last 
week or so.
So there must be something special to your webserver, or your web application, or your 
configuration, to make this happen.
But so far, you have not given us much to really get going in finding out what is really 
happening.



P.S.
Having a number of TCP connections established (open), between your Apache front-end 
webserver and your back-and tomcat server, is normal and expected.
Apache httpd + mod_jk communicate with tomcat, via TCP.  The port on tomcat which is 
listening on those connections is 8009 (as you can see in the AJP Connector of your tomcat 
configuration).  When Apache starts, mod_jk will automatically create a number of 
connections to tomcat, and keep them open for better performance.


This is what I see currently on one of my servers (where httpd and tomcat are currently 
alive and well, and not loaded at all) :


 Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State

tcp0  0 127.0.0.1:43611 127.0.0.1:8009  ESTABLISHED 
20059/apache2
tcp0  0 127.0.0.1:41927 127.0.0.1:8009  ESTABLISHED 
10379/apache2
tcp0  0 127.0.0.1:42416 127.0.0.1:8009  ESTABLISHED 
15167/apache2
tcp0  0 127.0.0.1:40649 127.0.0.1:8009  ESTABLISHED 
5279/apache2
tcp0  0 127.0.0.1:40236 127.0.0.1:8009  ESTABLISHED 
533/apache2
tcp0  0 127.0.0.1:43695 127.0.0.1:8009  ESTABLISHED 
15169/apache2
tcp6   0  0 :::8009 :::*LISTEN  
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:40649 ESTABLISHED 
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:42416 ESTABLISHED 
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:41927 ESTABLISHED 
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:43695 ESTABLISHED 
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:40236 ESTABLISHED 
2940/java
tcp6   0  0 127.0.0.1:8009  127.0.0.1:43611 ESTABLISHED 
2940/java

(The java process # 2940 is tomcat. Each Apache process that you see is an Apache "child" 
process running mod_jk. This server is not configured for a heavy load, and actually doing 
almost nothing right now.)


What do you see when you enter this command now ?
# netstat --tcp -pan | grep ":8009"


On 30.11.2016 20:39, Jaaz Portal wrote:

hi mark,
thanks, i have fixed configuration as you pointed out,
maybe this will mitigate the attack

before there was no connection_timeout in configuration
and this things was occurring too


best,
artur

2016-11-30 20:29 GMT+01:00 Mark Eggers :


Artur,

On 11/30/2016 10:41 AM, Jaaz Portal wrote:

no it looks like dos, its dos

i told you they dosed before bind server until we changed it to other
vendor,
and later was scanning my host for apache vulnerabilities

configuration is standard, the only thing i changed (after your guidance)
is connection_timeout
but this does not work for this exploit

workers.properties
worker.list=ajp13_worker

#
#-- ajp13_worker WORKER DEFINITION --
#-
#

#
# Defining a worker named 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi mark,
thanks, i have fixed configuration as you pointed out,
maybe this will mitigate the attack

before there was no connection_timeout in configuration
and this things was occurring too


best,
artur

2016-11-30 20:29 GMT+01:00 Mark Eggers :

> Artur,
>
> On 11/30/2016 10:41 AM, Jaaz Portal wrote:
> > no it looks like dos, its dos
> >
> > i told you they dosed before bind server until we changed it to other
> > vendor,
> > and later was scanning my host for apache vulnerabilities
> >
> > configuration is standard, the only thing i changed (after your guidance)
> > is connection_timeout
> > but this does not work for this exploit
> >
> > workers.properties
> > worker.list=ajp13_worker
> >
> > #
> > #-- ajp13_worker WORKER DEFINITION --
> > #-
> > #
> >
> > #
> > # Defining a worker named ajp13_worker and of type ajp13
> > # Note that the name and the type do not have to match.
> > #
> > worker.ajp13_worker.port=8009
> > worker.ajp13_worker.host=localhost
> > worker.ajp13_worker.socket_timeout=6
> > worker.ajp13_worker.type=ajp13
> > #
> > # Specifies the load balance factor when used with
> > # a load balancing worker.
> > # Note:
> > #  > lbfactor must be > 0
> > #  > Low lbfactor means less work done by the worker.
> > worker.ajp13_worker.lbfactor=1
> >
> > #
> > # Specify the size of the open connection cache.
> > #worker.ajp13_worker.cachesize
> >
> > #
> > #-- DEFAULT LOAD BALANCER WORKER DEFINITION --
> > #-
> > #
> >
> > #
> > # The loadbalancer (type lb) workers perform wighted round-robin
> > # load balancing with sticky sessions.
> > # Note:
> > #  > If a worker dies, the load balancer will check its state
> > #once in a while. Until then all work is redirected to peer
> > #workers.
> > worker.loadbalancer.type=lb
> > worker.loadbalancer.balance_workers=ajp13_worker
> >
> > 
> > server.xml
> >
> >
> >   > redirectPort="8443" maxConnections="256" keepAliveTimeout="3"/>
> >
> > best,
> > artur
>
> From the following fine documentation (which André has posted before):
>
> http://tomcat.apache.org/connectors-doc/reference/workers.html
>
> connection_pool_timeout (lots of stuff) . . . last paragraph:
>
> You should keep this time interval in sync with the keepAliveTimeout
> attribute (if it is set explicitly) or connectionTimeout attribute of
> your AJP connector in Tomcat's server.xml. Note however, that the value
> for mod_jk is given in seconds, the one in server.xml has to use
> milliseconds.
>
> The last line of the above snippet of the documentation is very important.
>
> Now let's look at your values.
>
> From workers.properties:
> worker.ajp13_worker.socket_timeout=6
>
> From server.xml
> connectionTimeout="6"
>
> So your socket_timeout value from workers.properties is 60,000 seconds
> (16 hours, 40 minutes), while your connectionTimeout value is 60,000
> milliseconds (1 minute).
>
> And your keepAliveTimeout (30,000 = 30 seconds) is not in sync with
> either value.
>
> So . . .
>
> 1. remove keepAliveTimeout from your AJP connector
> 2. change worker.ajp13_worker.socket_timeout to 60
>
> This will at least get you in line with the documentation. You can then
> proceed to diagnose whether you have a DOS (or DDOS) attack, an
> application issue, or if this solved the problem.
>
> . . . just my two cents (if I've done the math right)
> /mde/
>
> >
> > 2016-11-30 19:21 GMT+01:00 Mark Eggers :
> >
> >> Artur,
> >> On 11/30/2016 8:36 AM, Jaaz Portal wrote:
> >>> hi,
> >>> they has tried again with success despite setting connection_timeout
> and
> >>> limiting number of clients by mod_bw
> >>> the tomcat has frozen again.
> >>>
> >>> netstat does not showed any connections on port 80 but plenty of
> >>> connections from apache to localhost:8009
> >>> so it was not an attack that you has described (no slowlaris)
> >>>
> >>> im looking into debug files of mod_jk and forensic for some hints. If
> you
> >>> want i can share them (they have 4mb compressed)
> >>>
> >>> best wishes
> >>> artur
> >>
> >> This is beginning to look like an application or a configuration issue
> >> and not a DOS (or DDOS) attack.
> >>
> >> One the issues that may cause this is a mismatched timeout value between
> >> connection_pool_timeout in workers.properties (mod_jk) and the
> >> connectionTimeout in server.xml (Tomcat) for the AJP connector.
> >>
> >> Also, at least for the mod_jk version that I'm running, there is no
> >> limit for reply_timeout (mod_jk) by default.
> >>
> >> Can you post your workers.properties file and the AJP connector portion
> >> of your server.xml?
> >>
> >> In the conf directory of the mod_jk source code, there is a very nice
> >> workers.properties file that has sensible defaults. If 

Re: Apache/Tomcat vulnerability

[Mark Eggers]

Artur,

On 11/30/2016 10:41 AM, Jaaz Portal wrote:
> no it looks like dos, its dos
> 
> i told you they dosed before bind server until we changed it to other
> vendor,
> and later was scanning my host for apache vulnerabilities
> 
> configuration is standard, the only thing i changed (after your guidance)
> is connection_timeout
> but this does not work for this exploit
> 
> workers.properties
> worker.list=ajp13_worker
> 
> #
> #-- ajp13_worker WORKER DEFINITION --
> #-
> #
> 
> #
> # Defining a worker named ajp13_worker and of type ajp13
> # Note that the name and the type do not have to match.
> #
> worker.ajp13_worker.port=8009
> worker.ajp13_worker.host=localhost
> worker.ajp13_worker.socket_timeout=6
> worker.ajp13_worker.type=ajp13
> #
> # Specifies the load balance factor when used with
> # a load balancing worker.
> # Note:
> #  > lbfactor must be > 0
> #  > Low lbfactor means less work done by the worker.
> worker.ajp13_worker.lbfactor=1
> 
> #
> # Specify the size of the open connection cache.
> #worker.ajp13_worker.cachesize
> 
> #
> #-- DEFAULT LOAD BALANCER WORKER DEFINITION --
> #-
> #
> 
> #
> # The loadbalancer (type lb) workers perform wighted round-robin
> # load balancing with sticky sessions.
> # Note:
> #  > If a worker dies, the load balancer will check its state
> #once in a while. Until then all work is redirected to peer
> #workers.
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=ajp13_worker
> 
> 
> server.xml
> 
> 
>   redirectPort="8443" maxConnections="256" keepAliveTimeout="3"/>
> 
> best,
> artur

From the following fine documentation (which André has posted before):

http://tomcat.apache.org/connectors-doc/reference/workers.html

connection_pool_timeout (lots of stuff) . . . last paragraph:

You should keep this time interval in sync with the keepAliveTimeout
attribute (if it is set explicitly) or connectionTimeout attribute of
your AJP connector in Tomcat's server.xml. Note however, that the value
for mod_jk is given in seconds, the one in server.xml has to use
milliseconds.

The last line of the above snippet of the documentation is very important.

Now let's look at your values.

From workers.properties:
worker.ajp13_worker.socket_timeout=6

From server.xml
connectionTimeout="6"

So your socket_timeout value from workers.properties is 60,000 seconds
(16 hours, 40 minutes), while your connectionTimeout value is 60,000
milliseconds (1 minute).

And your keepAliveTimeout (30,000 = 30 seconds) is not in sync with
either value.

So . . .

1. remove keepAliveTimeout from your AJP connector
2. change worker.ajp13_worker.socket_timeout to 60

This will at least get you in line with the documentation. You can then
proceed to diagnose whether you have a DOS (or DDOS) attack, an
application issue, or if this solved the problem.

. . . just my two cents (if I've done the math right)
/mde/

> 
> 2016-11-30 19:21 GMT+01:00 Mark Eggers :
> 
>> Artur,
>> On 11/30/2016 8:36 AM, Jaaz Portal wrote:
>>> hi,
>>> they has tried again with success despite setting connection_timeout and
>>> limiting number of clients by mod_bw
>>> the tomcat has frozen again.
>>>
>>> netstat does not showed any connections on port 80 but plenty of
>>> connections from apache to localhost:8009
>>> so it was not an attack that you has described (no slowlaris)
>>>
>>> im looking into debug files of mod_jk and forensic for some hints. If you
>>> want i can share them (they have 4mb compressed)
>>>
>>> best wishes
>>> artur
>>
>> This is beginning to look like an application or a configuration issue
>> and not a DOS (or DDOS) attack.
>>
>> One the issues that may cause this is a mismatched timeout value between
>> connection_pool_timeout in workers.properties (mod_jk) and the
>> connectionTimeout in server.xml (Tomcat) for the AJP connector.
>>
>> Also, at least for the mod_jk version that I'm running, there is no
>> limit for reply_timeout (mod_jk) by default.
>>
>> Can you post your workers.properties file and the AJP connector portion
>> of your server.xml?
>>
>> In the conf directory of the mod_jk source code, there is a very nice
>> workers.properties file that has sensible defaults. If you've not done
>> so, I suggest that you start with the values specified in that file, and
>> make sure that the timeout values match (see my comment above).
>>
>> Also, when you used mod-proxy, did you use mod-proxy-ajp or
>> mod-proxy-http? If you used mod-proxy-ajp, then again there could be a
>> timeout mismatch (or no timeout specified at all).
>>
>> . . . just my two cents
>> /mde/
>>
>>>
>>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
>>>
 On 28.11.2016 22:04, Jaaz Portal wrote:

> hi 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

no it looks like dos, its dos

i told you they dosed before bind server until we changed it to other
vendor,
and later was scanning my host for apache vulnerabilities

configuration is standard, the only thing i changed (after your guidance)
is connection_timeout
but this does not work for this exploit

workers.properties
worker.list=ajp13_worker

#
#-- ajp13_worker WORKER DEFINITION --
#-
#

#
# Defining a worker named ajp13_worker and of type ajp13
# Note that the name and the type do not have to match.
#
worker.ajp13_worker.port=8009
worker.ajp13_worker.host=localhost
worker.ajp13_worker.socket_timeout=6
worker.ajp13_worker.type=ajp13
#
# Specifies the load balance factor when used with
# a load balancing worker.
# Note:
#  > lbfactor must be > 0
#  > Low lbfactor means less work done by the worker.
worker.ajp13_worker.lbfactor=1

#
# Specify the size of the open connection cache.
#worker.ajp13_worker.cachesize

#
#-- DEFAULT LOAD BALANCER WORKER DEFINITION --
#-
#

#
# The loadbalancer (type lb) workers perform wighted round-robin
# load balancing with sticky sessions.
# Note:
#  > If a worker dies, the load balancer will check its state
#once in a while. Until then all work is redirected to peer
#workers.
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=ajp13_worker


server.xml


 

best,
artur

2016-11-30 19:21 GMT+01:00 Mark Eggers :

> Artur,
> On 11/30/2016 8:36 AM, Jaaz Portal wrote:
> > hi,
> > they has tried again with success despite setting connection_timeout and
> > limiting number of clients by mod_bw
> > the tomcat has frozen again.
> >
> > netstat does not showed any connections on port 80 but plenty of
> > connections from apache to localhost:8009
> > so it was not an attack that you has described (no slowlaris)
> >
> > im looking into debug files of mod_jk and forensic for some hints. If you
> > want i can share them (they have 4mb compressed)
> >
> > best wishes
> > artur
>
> This is beginning to look like an application or a configuration issue
> and not a DOS (or DDOS) attack.
>
> One the issues that may cause this is a mismatched timeout value between
> connection_pool_timeout in workers.properties (mod_jk) and the
> connectionTimeout in server.xml (Tomcat) for the AJP connector.
>
> Also, at least for the mod_jk version that I'm running, there is no
> limit for reply_timeout (mod_jk) by default.
>
> Can you post your workers.properties file and the AJP connector portion
> of your server.xml?
>
> In the conf directory of the mod_jk source code, there is a very nice
> workers.properties file that has sensible defaults. If you've not done
> so, I suggest that you start with the values specified in that file, and
> make sure that the timeout values match (see my comment above).
>
> Also, when you used mod-proxy, did you use mod-proxy-ajp or
> mod-proxy-http? If you used mod-proxy-ajp, then again there could be a
> timeout mismatch (or no timeout specified at all).
>
> . . . just my two cents
> /mde/
>
> >
> > 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
> >
> >> On 28.11.2016 22:04, Jaaz Portal wrote:
> >>
> >>> hi Andre,
> >>> you are wrong. This vulnerability is not only causing memory leaks, it
> >>> makes also apache workers to hang
> >>>
> >>
> >> Maybe for the last time here :
> >>
> >> - what do you call "apache workers" ?
> >>
> >> , making it easy to exhaust the pool.
> >>
> >> - what do you call "the pool" ?
> >>
> >> what i have in my log files. But it is true also that such exhaustion
> can
> >>> be made by other forms of dos attacks described in this thread.
> >>>
> >>> regarding you suggestion on our application, it does not dos bind
> server
> >>> nether does not scan for various vulnerabilities in apache, what i have
> >>> also in the logs
> >>>
> >>
> >> For your information : I run about 25 Internet-facing Apache webservers
> >> (some with a back-end tomcat, some not).
> >> On every single one of those webservers, there are *hundreds* of such
> >> "scans" every day, as shown by the Apache access logs.  That is just a
> fact
> >> of life on the Internet.
> >> They are annoying, but most of them are harmless (from an "attack" point
> >> of view), because they are scanning for things that we do not run
> >> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless),
> and
> >> thus are responded to by Apache as "404 Not found".
> >> What is annoying with those scans, is
> >> a) that they fill the logfile, and thus make it more difficult to find
> >> really significant things
> >> b) that each of those requires some bandwidth and system resource, if
> only
> >> to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
> for
> >> that bandwidth 

Re: Apache/Tomcat vulnerability

[Mark Eggers]

Artur,
On 11/30/2016 8:36 AM, Jaaz Portal wrote:
> hi,
> they has tried again with success despite setting connection_timeout and
> limiting number of clients by mod_bw
> the tomcat has frozen again.
> 
> netstat does not showed any connections on port 80 but plenty of
> connections from apache to localhost:8009
> so it was not an attack that you has described (no slowlaris)
> 
> im looking into debug files of mod_jk and forensic for some hints. If you
> want i can share them (they have 4mb compressed)
> 
> best wishes
> artur

This is beginning to look like an application or a configuration issue
and not a DOS (or DDOS) attack.

One the issues that may cause this is a mismatched timeout value between
connection_pool_timeout in workers.properties (mod_jk) and the
connectionTimeout in server.xml (Tomcat) for the AJP connector.

Also, at least for the mod_jk version that I'm running, there is no
limit for reply_timeout (mod_jk) by default.

Can you post your workers.properties file and the AJP connector portion
of your server.xml?

In the conf directory of the mod_jk source code, there is a very nice
workers.properties file that has sensible defaults. If you've not done
so, I suggest that you start with the values specified in that file, and
make sure that the timeout values match (see my comment above).

Also, when you used mod-proxy, did you use mod-proxy-ajp or
mod-proxy-http? If you used mod-proxy-ajp, then again there could be a
timeout mismatch (or no timeout specified at all).

. . . just my two cents
/mde/

> 
> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
> 
>> On 28.11.2016 22:04, Jaaz Portal wrote:
>>
>>> hi Andre,
>>> you are wrong. This vulnerability is not only causing memory leaks, it
>>> makes also apache workers to hang
>>>
>>
>> Maybe for the last time here :
>>
>> - what do you call "apache workers" ?
>>
>> , making it easy to exhaust the pool.
>>
>> - what do you call "the pool" ?
>>
>> what i have in my log files. But it is true also that such exhaustion can
>>> be made by other forms of dos attacks described in this thread.
>>>
>>> regarding you suggestion on our application, it does not dos bind server
>>> nether does not scan for various vulnerabilities in apache, what i have
>>> also in the logs
>>>
>>
>> For your information : I run about 25 Internet-facing Apache webservers
>> (some with a back-end tomcat, some not).
>> On every single one of those webservers, there are *hundreds* of such
>> "scans" every day, as shown by the Apache access logs.  That is just a fact
>> of life on the Internet.
>> They are annoying, but most of them are harmless (from an "attack" point
>> of view), because they are scanning for things that we do not run
>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
>> thus are responded to by Apache as "404 Not found".
>> What is annoying with those scans, is
>> a) that they fill the logfile, and thus make it more difficult to find
>> really significant things
>> b) that each of those requires some bandwidth and system resource, if only
>> to return a "404 Not found" (or a "401 Unauthorised"), and that we pay for
>> that bandwidth and resources.
>>
>> If I could find a way to charge 0.1 cent per access to my servers, from
>> the people who wrote or run the programs who are doing this, I could retire
>> in luxury.
>>
>> But they are not a real problem, because they are caught as "invalid" by
>> Apache, and rejected quickly, so they cannot do anything really nasty
>> (except if they were sending several thousand such requests per second to
>> one of my servers for a long time).
>>
>> The ones that are worrying, are the ones
>> - a) which do /not/ end up as a "404 Not found", because they have found
>> an application which responds, and they are not coming from our legitimate
>> customers
>> - b) /the ones which we do not see/, because they either do not send a
>> valid HTTP request, or they have found a way to trigger one of our
>> applications, in such a way that the application misbehaves and, perhaps,
>> even if they do not crash our servers, they may provide the attacker with
>> some entry point to do other things which we do not know and do not control
>>
>> What I am trying to say here, is /do not jump to premature conclusions/.
>> Such "scans" as you mention, happen to everyone, all the time, from
>> ever-changing IP addresses located all over the world. Some of those
>> "scans" may come from the infected PC of your grandmother, and she does not
>> even know about it.
>>
>> There is no guarantee, and no indication or proof so far, that /these/
>> scans are even related to "the other thing" which happens on your
>> webserver, which looked much more focused.
>>
>> So do not just bundle them together as being the same thing, until you
>> have some real data that shows for example that these different things all
>> come from the same IP addresses.
>>
>> And one more thing, also finally until you come back 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

yes,
i was in hurry and pasted wrong ip
but i talked with my mate, this one that
had open connection was his host checking our webpage

so beside our connections there was no open http connections
but plenty of that between apache and tomcat

it was no slowlaris guys and with the forensics logs and debug of mod_jk
we are no step closer to find out the cause

any idea what to do next?

best
artur

2016-11-30 18:58 GMT+01:00 Mark Eggers :

> Artur,
>
> On 11/30/2016 9:02 AM, Jaaz Portal wrote:
> > hi,
> > sorry, there was two open connection on port 80
> > from 194.135.88.32 that is somwhere on epix.net.pl
> > an association of internet traffick exchange (some pirate hub)
> >
> > best,
> > artur
>
> 194.135.88.32 appears to be your web site, no?
>
> . . . just my two cents
> /mde/
>
> >
> > 2016-11-30 17:52 GMT+01:00 Jaaz Portal :
> >
> >> hi,
> >> i looked at the logs but there are no strange things,
> >> traffic as usual, no errors despite this one:
> >> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
> >> 139906329666752] AH00484: server reached MaxRequestWorkers setting,
> >> consider raising the MaxRequestWorkers setting
> >>
> >> any idea what to do next?
> >>
> >> best,
> >> artur
> >>
> >> 2016-11-30 17:36 GMT+01:00 Jaaz Portal :
> >>
> >>> hi,
> >>> they has tried again with success despite setting connection_timeout
> and
> >>> limiting number of clients by mod_bw
> >>> the tomcat has frozen again.
> >>>
> >>> netstat does not showed any connections on port 80 but plenty of
> >>> connections from apache to localhost:8009
> >>> so it was not an attack that you has described (no slowlaris)
> >>>
> >>> im looking into debug files of mod_jk and forensic for some hints. If
> you
> >>> want i can share them (they have 4mb compressed)
> >>>
> >>> best wishes
> >>> artur
> >>>
> >>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
> >>>
>  On 28.11.2016 22:04, Jaaz Portal wrote:
> 
> > hi Andre,
> > you are wrong. This vulnerability is not only causing memory leaks,
> it
> > makes also apache workers to hang
> >
> 
>  Maybe for the last time here :
> 
>  - what do you call "apache workers" ?
> 
>  , making it easy to exhaust the pool.
> 
>  - what do you call "the pool" ?
> 
>  what i have in my log files. But it is true also that such exhaustion
> can
> > be made by other forms of dos attacks described in this thread.
> >
> > regarding you suggestion on our application, it does not dos bind
> server
> > nether does not scan for various vulnerabilities in apache, what i
> have
> > also in the logs
> >
> 
>  For your information : I run about 25 Internet-facing Apache
> webservers
>  (some with a back-end tomcat, some not).
>  On every single one of those webservers, there are *hundreds* of such
>  "scans" every day, as shown by the Apache access logs.  That is just
> a fact
>  of life on the Internet.
>  They are annoying, but most of them are harmless (from an "attack"
> point
>  of view), because they are scanning for things that we do not run
>  (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost
> endless), and
>  thus are responded to by Apache as "404 Not found".
>  What is annoying with those scans, is
>  a) that they fill the logfile, and thus make it more difficult to find
>  really significant things
>  b) that each of those requires some bandwidth and system resource, if
>  only to return a "404 Not found" (or a "401 Unauthorised"), and that
> we pay
>  for that bandwidth and resources.
> 
>  If I could find a way to charge 0.1 cent per access to my servers,
> from
>  the people who wrote or run the programs who are doing this, I could
> retire
>  in luxury.
> 
>  But they are not a real problem, because they are caught as "invalid"
> by
>  Apache, and rejected quickly, so they cannot do anything really nasty
>  (except if they were sending several thousand such requests per
> second to
>  one of my servers for a long time).
> 
>  The ones that are worrying, are the ones
>  - a) which do /not/ end up as a "404 Not found", because they have
> found
>  an application which responds, and they are not coming from our
> legitimate
>  customers
>  - b) /the ones which we do not see/, because they either do not send a
>  valid HTTP request, or they have found a way to trigger one of our
>  applications, in such a way that the application misbehaves and,
> perhaps,
>  even if they do not crash our servers, they may provide the attacker
> with
>  some entry point to do other things which we do not know and do not
> control
> 
>  What I am trying to say here, is /do not jump to premature
> conclusions/.
>  Such "scans" as you mention, happen to 

Re: Apache/Tomcat vulnerability

[Mark Eggers]

Artur,

On 11/30/2016 9:02 AM, Jaaz Portal wrote:
> hi,
> sorry, there was two open connection on port 80
> from 194.135.88.32 that is somwhere on epix.net.pl
> an association of internet traffick exchange (some pirate hub)
> 
> best,
> artur

194.135.88.32 appears to be your web site, no?

. . . just my two cents
/mde/

> 
> 2016-11-30 17:52 GMT+01:00 Jaaz Portal :
> 
>> hi,
>> i looked at the logs but there are no strange things,
>> traffic as usual, no errors despite this one:
>> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
>> 139906329666752] AH00484: server reached MaxRequestWorkers setting,
>> consider raising the MaxRequestWorkers setting
>>
>> any idea what to do next?
>>
>> best,
>> artur
>>
>> 2016-11-30 17:36 GMT+01:00 Jaaz Portal :
>>
>>> hi,
>>> they has tried again with success despite setting connection_timeout and
>>> limiting number of clients by mod_bw
>>> the tomcat has frozen again.
>>>
>>> netstat does not showed any connections on port 80 but plenty of
>>> connections from apache to localhost:8009
>>> so it was not an attack that you has described (no slowlaris)
>>>
>>> im looking into debug files of mod_jk and forensic for some hints. If you
>>> want i can share them (they have 4mb compressed)
>>>
>>> best wishes
>>> artur
>>>
>>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
>>>
 On 28.11.2016 22:04, Jaaz Portal wrote:

> hi Andre,
> you are wrong. This vulnerability is not only causing memory leaks, it
> makes also apache workers to hang
>

 Maybe for the last time here :

 - what do you call "apache workers" ?

 , making it easy to exhaust the pool.

 - what do you call "the pool" ?

 what i have in my log files. But it is true also that such exhaustion can
> be made by other forms of dos attacks described in this thread.
>
> regarding you suggestion on our application, it does not dos bind server
> nether does not scan for various vulnerabilities in apache, what i have
> also in the logs
>

 For your information : I run about 25 Internet-facing Apache webservers
 (some with a back-end tomcat, some not).
 On every single one of those webservers, there are *hundreds* of such
 "scans" every day, as shown by the Apache access logs.  That is just a fact
 of life on the Internet.
 They are annoying, but most of them are harmless (from an "attack" point
 of view), because they are scanning for things that we do not run
 (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
 thus are responded to by Apache as "404 Not found".
 What is annoying with those scans, is
 a) that they fill the logfile, and thus make it more difficult to find
 really significant things
 b) that each of those requires some bandwidth and system resource, if
 only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
 for that bandwidth and resources.

 If I could find a way to charge 0.1 cent per access to my servers, from
 the people who wrote or run the programs who are doing this, I could retire
 in luxury.

 But they are not a real problem, because they are caught as "invalid" by
 Apache, and rejected quickly, so they cannot do anything really nasty
 (except if they were sending several thousand such requests per second to
 one of my servers for a long time).

 The ones that are worrying, are the ones
 - a) which do /not/ end up as a "404 Not found", because they have found
 an application which responds, and they are not coming from our legitimate
 customers
 - b) /the ones which we do not see/, because they either do not send a
 valid HTTP request, or they have found a way to trigger one of our
 applications, in such a way that the application misbehaves and, perhaps,
 even if they do not crash our servers, they may provide the attacker with
 some entry point to do other things which we do not know and do not control

 What I am trying to say here, is /do not jump to premature conclusions/.
 Such "scans" as you mention, happen to everyone, all the time, from
 ever-changing IP addresses located all over the world. Some of those
 "scans" may come from the infected PC of your grandmother, and she does not
 even know about it.

 There is no guarantee, and no indication or proof so far, that /these/
 scans are even related to "the other thing" which happens on your
 webserver, which looked much more focused.

 So do not just bundle them together as being the same thing, until you
 have some real data that shows for example that these different things all
 come from the same IP addresses.

 And one more thing, also finally until you come back with some real data
 : I am not saying that your 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
sorry, there was two open connection on port 80
from 194.135.88.32 that is somwhere on epix.net.pl
an association of internet traffick exchange (some pirate hub)

best,
artur

2016-11-30 17:52 GMT+01:00 Jaaz Portal :

> hi,
> i looked at the logs but there are no strange things,
> traffic as usual, no errors despite this one:
> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
> 139906329666752] AH00484: server reached MaxRequestWorkers setting,
> consider raising the MaxRequestWorkers setting
>
> any idea what to do next?
>
> best,
> artur
>
> 2016-11-30 17:36 GMT+01:00 Jaaz Portal :
>
>> hi,
>> they has tried again with success despite setting connection_timeout and
>> limiting number of clients by mod_bw
>> the tomcat has frozen again.
>>
>> netstat does not showed any connections on port 80 but plenty of
>> connections from apache to localhost:8009
>> so it was not an attack that you has described (no slowlaris)
>>
>> im looking into debug files of mod_jk and forensic for some hints. If you
>> want i can share them (they have 4mb compressed)
>>
>> best wishes
>> artur
>>
>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
>>
>>> On 28.11.2016 22:04, Jaaz Portal wrote:
>>>
 hi Andre,
 you are wrong. This vulnerability is not only causing memory leaks, it
 makes also apache workers to hang

>>>
>>> Maybe for the last time here :
>>>
>>> - what do you call "apache workers" ?
>>>
>>> , making it easy to exhaust the pool.
>>>
>>> - what do you call "the pool" ?
>>>
>>> what i have in my log files. But it is true also that such exhaustion can
 be made by other forms of dos attacks described in this thread.

 regarding you suggestion on our application, it does not dos bind server
 nether does not scan for various vulnerabilities in apache, what i have
 also in the logs

>>>
>>> For your information : I run about 25 Internet-facing Apache webservers
>>> (some with a back-end tomcat, some not).
>>> On every single one of those webservers, there are *hundreds* of such
>>> "scans" every day, as shown by the Apache access logs.  That is just a fact
>>> of life on the Internet.
>>> They are annoying, but most of them are harmless (from an "attack" point
>>> of view), because they are scanning for things that we do not run
>>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
>>> thus are responded to by Apache as "404 Not found".
>>> What is annoying with those scans, is
>>> a) that they fill the logfile, and thus make it more difficult to find
>>> really significant things
>>> b) that each of those requires some bandwidth and system resource, if
>>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
>>> for that bandwidth and resources.
>>>
>>> If I could find a way to charge 0.1 cent per access to my servers, from
>>> the people who wrote or run the programs who are doing this, I could retire
>>> in luxury.
>>>
>>> But they are not a real problem, because they are caught as "invalid" by
>>> Apache, and rejected quickly, so they cannot do anything really nasty
>>> (except if they were sending several thousand such requests per second to
>>> one of my servers for a long time).
>>>
>>> The ones that are worrying, are the ones
>>> - a) which do /not/ end up as a "404 Not found", because they have found
>>> an application which responds, and they are not coming from our legitimate
>>> customers
>>> - b) /the ones which we do not see/, because they either do not send a
>>> valid HTTP request, or they have found a way to trigger one of our
>>> applications, in such a way that the application misbehaves and, perhaps,
>>> even if they do not crash our servers, they may provide the attacker with
>>> some entry point to do other things which we do not know and do not control
>>>
>>> What I am trying to say here, is /do not jump to premature conclusions/.
>>> Such "scans" as you mention, happen to everyone, all the time, from
>>> ever-changing IP addresses located all over the world. Some of those
>>> "scans" may come from the infected PC of your grandmother, and she does not
>>> even know about it.
>>>
>>> There is no guarantee, and no indication or proof so far, that /these/
>>> scans are even related to "the other thing" which happens on your
>>> webserver, which looked much more focused.
>>>
>>> So do not just bundle them together as being the same thing, until you
>>> have some real data that shows for example that these different things all
>>> come from the same IP addresses.
>>>
>>> And one more thing, also finally until you come back with some real data
>>> : I am not saying that your application "scans your server".  What I am
>>> saying is that, maybe, by chance or by design, the attackers have found a
>>> URL which goes to your application, and which causes your application to
>>> keep tomcat and/or Apache busy for a long time.
>>> And 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
i looked at the logs but there are no strange things,
traffic as usual, no errors despite this one:
[Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
139906329666752] AH00484: server reached MaxRequestWorkers setting,
consider raising the MaxRequestWorkers setting

any idea what to do next?

best,
artur

2016-11-30 17:36 GMT+01:00 Jaaz Portal :

> hi,
> they has tried again with success despite setting connection_timeout and
> limiting number of clients by mod_bw
> the tomcat has frozen again.
>
> netstat does not showed any connections on port 80 but plenty of
> connections from apache to localhost:8009
> so it was not an attack that you has described (no slowlaris)
>
> im looking into debug files of mod_jk and forensic for some hints. If you
> want i can share them (they have 4mb compressed)
>
> best wishes
> artur
>
> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :
>
>> On 28.11.2016 22:04, Jaaz Portal wrote:
>>
>>> hi Andre,
>>> you are wrong. This vulnerability is not only causing memory leaks, it
>>> makes also apache workers to hang
>>>
>>
>> Maybe for the last time here :
>>
>> - what do you call "apache workers" ?
>>
>> , making it easy to exhaust the pool.
>>
>> - what do you call "the pool" ?
>>
>> what i have in my log files. But it is true also that such exhaustion can
>>> be made by other forms of dos attacks described in this thread.
>>>
>>> regarding you suggestion on our application, it does not dos bind server
>>> nether does not scan for various vulnerabilities in apache, what i have
>>> also in the logs
>>>
>>
>> For your information : I run about 25 Internet-facing Apache webservers
>> (some with a back-end tomcat, some not).
>> On every single one of those webservers, there are *hundreds* of such
>> "scans" every day, as shown by the Apache access logs.  That is just a fact
>> of life on the Internet.
>> They are annoying, but most of them are harmless (from an "attack" point
>> of view), because they are scanning for things that we do not run
>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
>> thus are responded to by Apache as "404 Not found".
>> What is annoying with those scans, is
>> a) that they fill the logfile, and thus make it more difficult to find
>> really significant things
>> b) that each of those requires some bandwidth and system resource, if
>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
>> for that bandwidth and resources.
>>
>> If I could find a way to charge 0.1 cent per access to my servers, from
>> the people who wrote or run the programs who are doing this, I could retire
>> in luxury.
>>
>> But they are not a real problem, because they are caught as "invalid" by
>> Apache, and rejected quickly, so they cannot do anything really nasty
>> (except if they were sending several thousand such requests per second to
>> one of my servers for a long time).
>>
>> The ones that are worrying, are the ones
>> - a) which do /not/ end up as a "404 Not found", because they have found
>> an application which responds, and they are not coming from our legitimate
>> customers
>> - b) /the ones which we do not see/, because they either do not send a
>> valid HTTP request, or they have found a way to trigger one of our
>> applications, in such a way that the application misbehaves and, perhaps,
>> even if they do not crash our servers, they may provide the attacker with
>> some entry point to do other things which we do not know and do not control
>>
>> What I am trying to say here, is /do not jump to premature conclusions/.
>> Such "scans" as you mention, happen to everyone, all the time, from
>> ever-changing IP addresses located all over the world. Some of those
>> "scans" may come from the infected PC of your grandmother, and she does not
>> even know about it.
>>
>> There is no guarantee, and no indication or proof so far, that /these/
>> scans are even related to "the other thing" which happens on your
>> webserver, which looked much more focused.
>>
>> So do not just bundle them together as being the same thing, until you
>> have some real data that shows for example that these different things all
>> come from the same IP addresses.
>>
>> And one more thing, also finally until you come back with some real data
>> : I am not saying that your application "scans your server".  What I am
>> saying is that, maybe, by chance or by design, the attackers have found a
>> URL which goes to your application, and which causes your application to
>> keep tomcat and/or Apache busy for a long time.
>> And that maybe /that/ is the problem you should be looking for, and not
>> some hypothetical bug in Apache httpd or tomcat.
>>
>>
>>
>>> kindly regards,
>>> artur
>>>
>>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) :
>>>
>>> On 28.11.2016 20:34, Jaaz Portal wrote:

 hi mark,
> yes, i understand now what slowloris attack is.
> 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
they has tried again with success despite setting connection_timeout and
limiting number of clients by mod_bw
the tomcat has frozen again.

netstat does not showed any connections on port 80 but plenty of
connections from apache to localhost:8009
so it was not an attack that you has described (no slowlaris)

im looking into debug files of mod_jk and forensic for some hints. If you
want i can share them (they have 4mb compressed)

best wishes
artur

2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) :

> On 28.11.2016 22:04, Jaaz Portal wrote:
>
>> hi Andre,
>> you are wrong. This vulnerability is not only causing memory leaks, it
>> makes also apache workers to hang
>>
>
> Maybe for the last time here :
>
> - what do you call "apache workers" ?
>
> , making it easy to exhaust the pool.
>
> - what do you call "the pool" ?
>
> what i have in my log files. But it is true also that such exhaustion can
>> be made by other forms of dos attacks described in this thread.
>>
>> regarding you suggestion on our application, it does not dos bind server
>> nether does not scan for various vulnerabilities in apache, what i have
>> also in the logs
>>
>
> For your information : I run about 25 Internet-facing Apache webservers
> (some with a back-end tomcat, some not).
> On every single one of those webservers, there are *hundreds* of such
> "scans" every day, as shown by the Apache access logs.  That is just a fact
> of life on the Internet.
> They are annoying, but most of them are harmless (from an "attack" point
> of view), because they are scanning for things that we do not run
> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
> thus are responded to by Apache as "404 Not found".
> What is annoying with those scans, is
> a) that they fill the logfile, and thus make it more difficult to find
> really significant things
> b) that each of those requires some bandwidth and system resource, if only
> to return a "404 Not found" (or a "401 Unauthorised"), and that we pay for
> that bandwidth and resources.
>
> If I could find a way to charge 0.1 cent per access to my servers, from
> the people who wrote or run the programs who are doing this, I could retire
> in luxury.
>
> But they are not a real problem, because they are caught as "invalid" by
> Apache, and rejected quickly, so they cannot do anything really nasty
> (except if they were sending several thousand such requests per second to
> one of my servers for a long time).
>
> The ones that are worrying, are the ones
> - a) which do /not/ end up as a "404 Not found", because they have found
> an application which responds, and they are not coming from our legitimate
> customers
> - b) /the ones which we do not see/, because they either do not send a
> valid HTTP request, or they have found a way to trigger one of our
> applications, in such a way that the application misbehaves and, perhaps,
> even if they do not crash our servers, they may provide the attacker with
> some entry point to do other things which we do not know and do not control
>
> What I am trying to say here, is /do not jump to premature conclusions/.
> Such "scans" as you mention, happen to everyone, all the time, from
> ever-changing IP addresses located all over the world. Some of those
> "scans" may come from the infected PC of your grandmother, and she does not
> even know about it.
>
> There is no guarantee, and no indication or proof so far, that /these/
> scans are even related to "the other thing" which happens on your
> webserver, which looked much more focused.
>
> So do not just bundle them together as being the same thing, until you
> have some real data that shows for example that these different things all
> come from the same IP addresses.
>
> And one more thing, also finally until you come back with some real data :
> I am not saying that your application "scans your server".  What I am
> saying is that, maybe, by chance or by design, the attackers have found a
> URL which goes to your application, and which causes your application to
> keep tomcat and/or Apache busy for a long time.
> And that maybe /that/ is the problem you should be looking for, and not
> some hypothetical bug in Apache httpd or tomcat.
>
>
>
>> kindly regards,
>> artur
>>
>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) :
>>
>> On 28.11.2016 20:34, Jaaz Portal wrote:
>>>
>>> hi mark,
 yes, i understand now what slowloris attack is.
 maybe it was this maybe *this one based on * * mod_proxy denial of
 service *
 CVE-2014-0117 


>>> You keep on saying this, but the description of that vulnerability of
>>> *Apache httpd*, and the symptoms which you described, *do not match*.
>>> You described the problem as ocurring in Apache tomcat, which in your
>>> case
>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
>>> cured the problem.
>>>
>>> 

Re: Apache/Tomcat vulnerability

[tomcat]

On 28.11.2016 22:04, Jaaz Portal wrote:

hi Andre,
you are wrong. This vulnerability is not only causing memory leaks, it
makes also apache workers to hang


Maybe for the last time here :

- what do you call "apache workers" ?

, making it easy to exhaust the pool.

- what do you call "the pool" ?


what i have in my log files. But it is true also that such exhaustion can
be made by other forms of dos attacks described in this thread.

regarding you suggestion on our application, it does not dos bind server
nether does not scan for various vulnerabilities in apache, what i have
also in the logs


For your information : I run about 25 Internet-facing Apache webservers (some with a 
back-end tomcat, some not).
On every single one of those webservers, there are *hundreds* of such "scans" every day, 
as shown by the Apache access logs.  That is just a fact of life on the Internet.
They are annoying, but most of them are harmless (from an "attack" point of view), because 
they are scanning for things that we do not run (phpmyadmin, xmlrpc, vti_bin, etc., etc., 
the list is almost endless), and thus are responded to by Apache as "404 Not found".

What is annoying with those scans, is
a) that they fill the logfile, and thus make it more difficult to find really significant 
things
b) that each of those requires some bandwidth and system resource, if only to return a 
"404 Not found" (or a "401 Unauthorised"), and that we pay for that bandwidth and resources.


If I could find a way to charge 0.1 cent per access to my servers, from the people who 
wrote or run the programs who are doing this, I could retire in luxury.


But they are not a real problem, because they are caught as "invalid" by Apache, and 
rejected quickly, so they cannot do anything really nasty (except if they were sending 
several thousand such requests per second to one of my servers for a long time).


The ones that are worrying, are the ones
- a) which do /not/ end up as a "404 Not found", because they have found an application 
which responds, and they are not coming from our legitimate customers
- b) /the ones which we do not see/, because they either do not send a valid HTTP request, 
or they have found a way to trigger one of our applications, in such a way that the 
application misbehaves and, perhaps, even if they do not crash our servers, they may 
provide the attacker with some entry point to do other things which we do not know and do 
not control


What I am trying to say here, is /do not jump to premature conclusions/.
Such "scans" as you mention, happen to everyone, all the time, from ever-changing IP 
addresses located all over the world. Some of those "scans" may come from the infected PC 
of your grandmother, and she does not even know about it.


There is no guarantee, and no indication or proof so far, that /these/ scans are even 
related to "the other thing" which happens on your webserver, which looked much more focused.


So do not just bundle them together as being the same thing, until you have some real data 
that shows for example that these different things all come from the same IP addresses.


And one more thing, also finally until you come back with some real data : I am not saying 
that your application "scans your server".  What I am saying is that, maybe, by chance or 
by design, the attackers have found a URL which goes to your application, and which causes 
your application to keep tomcat and/or Apache busy for a long time.
And that maybe /that/ is the problem you should be looking for, and not some hypothetical 
bug in Apache httpd or tomcat.





kindly regards,
artur

2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) :


On 28.11.2016 20:34, Jaaz Portal wrote:


hi mark,
yes, i understand now what slowloris attack is.
maybe it was this maybe *this one based on * * mod_proxy denial of
service *
CVE-2014-0117 



You keep on saying this, but the description of that vulnerability of
*Apache httpd*, and the symptoms which you described, *do not match*.
You described the problem as ocurring in Apache tomcat, which in your case
is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
cured the problem.

The CVE above applies to Apache httpd, and describes how an attacker could
attack Apache httpd and cause *its children* processes to crash (the
children processes of Apache httpd), by leading them to consume a lot of
memory and crash with an out-of-memory error.
Granted, the problem occurred in the mod_proxy module of Apache httpd; but
it was httpd which crashed, not tomcat.
And tomcat processes are not "Apache httpd children processes" in any
understanding of the term.

As far as I remember, you never mentioned Apache httpd crashing. You
mentioned "the pool" getting full or satured or something like that,
without ever describing properly what you meant by "the pool".

As far as I am concerned, according to all the 

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jaaz,

On 11/28/16 2:24 PM, Jaaz Portal wrote:
> hi, i written "exploited some well know vulnerability in mod_proxy"
> not mod_jk.

Yes but then you implied that mod_jk had the same problem:

On 11/27/16 1:03 PM, Jaaz Portal wrote:
> Then they exploited some well know vulnerability in mod_proxy. We 
> have updated apache to the latest but again they has exploited it,
> so we have switched to mod_jk. *And then guess what. They exploited
> it too* so i decided to write to this list looking for help before
>  trying jetty.

(emphasis mine).

> we switched to mod_jk but they exploited it too, rest you can read
> on list

I have read all your posts, and I see no evidence of an exploit.

You have posted a single line from your log file which tells us next
to nothing:

> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
>  139793489638592] AH00484: server reached MaxRequestWorkers
> setting, consider raising the MaxRequestWorkers setting

Okay, you ran out of workers, meaning that you have a large number of
incoming requests. This indicates neither a vulnerability nor an exploit
.

> anyway we have configured apache to mitigate such kind of attack
> (whatever flavor it is)

Great. How did you do that? This is a community of users who support
each other, not a Help Desk. So please help the community by
explaining what the problem was and what the solution was as well.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPLZoAAoJEBzwKT+lPKRYSdkP/2wqolHXAjHGdYiY3qlQVwt5
0ND8fSnZQmSJROFwd1foGAiiAlSHsQV30QML6AgLzlhpTnyb7rOEE1VohZl5QMKv
Z3No4N3LXH/c9OtoYsf3A18h5GyvXzG0abRaI9dbIYLjqrv1XAR0J3cDrvmmirqW
Nh22fC0aYNJk68MbREgO8t6D2TIWEPqLyXAJAxc6nRtCaWaMJ22BnGH4rPmPcV8h
KzuJ6oxjSzSzewJGRDuZsdpLqJl+TvOGRzuv4PTKWnXmUXs1DeBLaUJHOmNeuGKB
WW18GlIwOOhVxdPY8qatiIuInotO6w0v1nL1r9nqV+d/3gA9Rl/myH/Vi5tWa6Io
Ca30NBKEk4A3xXmuJyTwryqJlK4QHP2JjjLCGQvYu8/fsNLzJPBViHRKQvgM+Yjt
YT46Kels0mXGtgLE8D4rhgA+Zu7wrIAtHKWh7pXZ7lBm3RQB31e6elkwfGDaSk2v
yEbXKWDWkfxPbIG9e/fxbZT/pgfE6hOMCL94l7oUGvxLxjEzVBzkuTjHBYV7zGpA
klxWOHrOJWtWn0KDmaU21jeuVzWZjKr/HVDnMGyAo2p9svNTuB/yLx+aToXFe2Sq
hrpZZoa61Jcdq+tU0maDq01PvUfeLW/fxcSjTXChZpuEFv5IUo6NKP5GoxqC3wrv
wDNqKX5IyyKJXVupnsB4
=q1fb
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi Andre,
you are wrong. This vulnerability is not only causing memory leaks, it
makes also apache workers to hang, making it easy to exhaust the pool.
what i have in my log files. But it is true also that such exhaustion can
be made by other forms of dos attacks described in this thread.

regarding you suggestion on our application, it does not dos bind server
nether does not scan for various vulnerabilities in apache, what i have
also in the logs

kindly regards,
artur

2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) :

> On 28.11.2016 20:34, Jaaz Portal wrote:
>
>> hi mark,
>> yes, i understand now what slowloris attack is.
>> maybe it was this maybe *this one based on * * mod_proxy denial of
>> service *
>> CVE-2014-0117 > cvename.cgi?name=CVE-2014-0117>
>>
>
> You keep on saying this, but the description of that vulnerability of
> *Apache httpd*, and the symptoms which you described, *do not match*.
> You described the problem as ocurring in Apache tomcat, which in your case
> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
> cured the problem.
>
> The CVE above applies to Apache httpd, and describes how an attacker could
> attack Apache httpd and cause *its children* processes to crash (the
> children processes of Apache httpd), by leading them to consume a lot of
> memory and crash with an out-of-memory error.
> Granted, the problem occurred in the mod_proxy module of Apache httpd; but
> it was httpd which crashed, not tomcat.
> And tomcat processes are not "Apache httpd children processes" in any
> understanding of the term.
>
> As far as I remember, you never mentioned Apache httpd crashing. You
> mentioned "the pool" getting full or satured or something like that,
> without ever describing properly what you meant by "the pool".
>
> As far as I am concerned, according to all the relatively unspecific
> information which you have previously provided :
> 1) the attack - if that is what it is/was - is definitely NOT related to
> the CVE which you have repeatedly mentioned
> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed at
> your front-end Apache. Instead, it seems that the requests are properly
> received by Apache, properly decoded by Apache, and then whatever Apache
> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is
> properly forwarding these requests to a back-end tomcat; and it is at the
> level of that back-end tomcat that the requests never seem to end, and in
> the end paralyse your tomcat server (and later on maybe your Apache httpd
> server too, because it is also waiting for tomcat to respond).
>
> So your very way of describing the problem, in terms of "first we used
> this proxy module, and then they exploited the vulnerability so and so;
> then we changed the proxy module, and they exploited that too; etc.."
> seems to not have anything to do with the problem per se, and (I believe)
> confuses everyone, including yourself.
>
> It is not that "they" exploited different vulnerabilities of various httpd
> proxy modules, one after the other. Each of these proxy modules was doing
> its job properly, and forwarding valid HTTP requests properly to tomcat.
> When you changed from one proxy module to another, you did not really
> change anything in that respect, because any proxy module would do the same.
>
> But in all cases, what did not change, was the tomcat behind the
> front-end, and the application running on that tomcat.  So the presumed
> attackers did not have to change anything, they just kept on sending the
> same requests, because they were really targeting your back-end tomcat or
> the tomcat application in it, no matter /how/ you were forwarding requests
> from Apache httpd to tomcat.
>
> So either it is tomcat itself, which has a problem with some request URLs
> which do not bother Apache httpd (possible, but statistically unlikely), or
> it is the application which runs in tomcat that has such a problem
> (statistically, much more likely).
>
> we do not know yet
>>
>> we have setup more logging and are waiting for them to attack once again
>>
>
> Yes, that is the right thing to do.  Before deciding what the problem may
> be, and what you can do about it, the first thing you need is *data*.  You
> need to know
> - which request URL(s?) cause that problem
> - which IPs these requests come from (always the same ? multiple IPs that
> change all the time ? how many ? can these IPs be valid/expected clients or
> not ? do these IPs look like some "coordinated group" ?)
> - how many such requests there may be during some period of time (10, 100,
> 1000, more ?)
> - if these URLs result in passing the request to tomcat
> - what tomcat application (if any) they are directed to
> - if so, when that application receives such a request, what is it
> supposed to do ? does it do it properly ? how long does it need, to respond
> to such a request ?
>
> You also need to ask 

Re: Apache/Tomcat vulnerability

[tomcat]

On 28.11.2016 20:34, Jaaz Portal wrote:

hi mark,
yes, i understand now what slowloris attack is.
maybe it was this maybe *this one based on * * mod_proxy denial of service *
CVE-2014-0117 


You keep on saying this, but the description of that vulnerability of *Apache httpd*, and 
the symptoms which you described, *do not match*.
You described the problem as ocurring in Apache tomcat, which in your case is sitting as a 
back-end, *behind* Apache httpd. And restarting tomcat cured the problem.


The CVE above applies to Apache httpd, and describes how an attacker could attack Apache 
httpd and cause *its children* processes to crash (the children processes of Apache 
httpd), by leading them to consume a lot of memory and crash with an out-of-memory error.
Granted, the problem occurred in the mod_proxy module of Apache httpd; but it was httpd 
which crashed, not tomcat.
And tomcat processes are not "Apache httpd children processes" in any understanding of the 
term.


As far as I remember, you never mentioned Apache httpd crashing. You mentioned "the pool" 
getting full or satured or something like that, without ever describing properly what you 
meant by "the pool".


As far as I am concerned, according to all the relatively unspecific information which you 
have previously provided :
1) the attack - if that is what it is/was - is definitely NOT related to the CVE which you 
have repeatedly mentioned
2) it is apparently not a "classical" DoS or "slowloris DoS" directed at your front-end 
Apache. Instead, it seems that the requests are properly received by Apache, properly 
decoded by Apache, and then whatever Apache proxy module you are using (mod_proxy_http, 
mod_proxy_ajp or mod_jk) is properly forwarding these requests to a back-end tomcat; and 
it is at the level of that back-end tomcat that the requests never seem to end, and in the 
end paralyse your tomcat server (and later on maybe your Apache httpd server too, because 
it is also waiting for tomcat to respond).


So your very way of describing the problem, in terms of "first we used this proxy module, 
and then they exploited the vulnerability so and so; then we changed the proxy module, and 
they exploited that too; etc.."
seems to not have anything to do with the problem per se, and (I believe) confuses 
everyone, including yourself.


It is not that "they" exploited different vulnerabilities of various httpd proxy modules, 
one after the other. Each of these proxy modules was doing its job properly, and 
forwarding valid HTTP requests properly to tomcat. When you changed from one proxy module 
to another, you did not really change anything in that respect, because any proxy module 
would do the same.


But in all cases, what did not change, was the tomcat behind the front-end, and the 
application running on that tomcat.  So the presumed attackers did not have to change 
anything, they just kept on sending the same requests, because they were really targeting 
your back-end tomcat or the tomcat application in it, no matter /how/ you were forwarding 
requests from Apache httpd to tomcat.


So either it is tomcat itself, which has a problem with some request URLs which do not 
bother Apache httpd (possible, but statistically unlikely), or it is the application which 
runs in tomcat that has such a problem (statistically, much more likely).



we do not know yet

we have setup more logging and are waiting for them to attack once again


Yes, that is the right thing to do.  Before deciding what the problem may be, and what you 
can do about it, the first thing you need is *data*.  You need to know

- which request URL(s?) cause that problem
- which IPs these requests come from (always the same ? multiple IPs that change all the 
time ? how many ? can these IPs be valid/expected clients or not ? do these IPs look like 
some "coordinated group" ?)

- how many such requests there may be during some period of time (10, 100, 
1000, more ?)
- if these URLs result in passing the request to tomcat
- what tomcat application (if any) they are directed to
- if so, when that application receives such a request, what is it supposed to do ? does 
it do it properly ? how long does it need, to respond to such a request ?


You also need to ask yourself a question : is the application which you run inside tomcat 
something that you designed yourself (and which hackers are unlikely to know well-enough 
to find such a URL which paralyses your server) ? or is it some well-known third-party 
java application which you are running (and for which would-be attackers would be much 
more likely to know exactly such a bug) ?




anyway, thank you for all informations, it was very useful and educational
reading for all of us

best wishes,
artur

2016-11-28 19:46 GMT+01:00 Mark Eggers :


Jaaz,

On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:

On 27.11.2016 19:03, Jaaz Portal 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi mark,
yes, i understand now what slowloris attack is.
maybe it was this maybe *this one based on * * mod_proxy denial of service *
CVE-2014-0117 
we do not know yet

we have setup more logging and are waiting for them to attack once again

anyway, thank you for all informations, it was very useful and educational
reading for all of us

best wishes,
artur

2016-11-28 19:46 GMT+01:00 Mark Eggers :

> Jaaz,
>
> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
> > On 27.11.2016 19:03, Jaaz Portal wrote:
> >> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :
> >>
> >>> On 27.11.2016 14:26, Jaaz Portal wrote:
> >>>
>  hi,
>  everything i know so far is just this single log line that appeared in
>  apache error.log
> 
>  [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
>  1397934896385
>  92] AH00484: server reached MaxRequestWorkers setting, consider
> raising
>  the
>  MaxR
>  equestWorkers setting
> 
>  there was nothing else, just this strange line
> 
> >>>
> >>> This is not a "strange" line. It is telling you something.
> >>> One problem is that you seem convinced in advance, without serious
> >>> proof,
> >>> that there is a "bug" or a vulnerability in httpd or tomcat.
> >>> Read the explanation of the httpd parameter, here :
> >>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
> >>> and try to understand it.
> >>>
> >>
> >> I understand it very well. We are serving no more that 500 clients per
> >> day
> >> so there was no other option that some kind of attack.
> >>
> >>
> >> About the "bug" or "vulnerability" : a webserver is supposed to serve
> >> HTTP
> >>> requests from clients.  That is what you expect of it. It has no
> >>> choice but
> >>> to accept any client connection and request, up to the maximum it can
> >>> handle considering its configuration and the capacity of the system on
> >>> which it runs. That is not a bug, it is a feature.
> >>>
> >>>
> >> We have some weeks ago come under attack from some Polish group. It was
> >> first bind that was DoS'ed, we was running on stable Debian so i updated
> >> bind to latest version. It did not helped. They has dos'ed it so we
> >> switched to other dns provider. That has helped.
> >>
> >> Then they exploited some well know vulnerability in mod_proxy. We have
> >> updated apache to the latest but again they has exploited it, so we have
> >> switched to mod_jk. And then guess what. They exploited it too so i
> >> decided
> >> to write to this list looking for help before trying jetty.
> >>
> >>
> >>>
> >>> The normal Apache httpd access log, will log a request only when it is
> >>> finished.  If the request never finishes, it will not get logged.
> >>> That may be why you do not see these requests in the log.
> >>> But have a look at this Apache httpd module :
> >>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
> >>>
> >>
> >> ok, thanks, will take care
> >>
> >> Note : that is also why I was telling you to enable the mod_jk log,
> >> and to
> >>> examine it.
> >>> Because mod_jk will also log information before the request produces a
> >>> response.
> >>>
> >>>
> >>> and server hanged.
> >>>
> >>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
> >>>
> >>
> >> Apache was accepting connection but not processed it. After restart of
> >> tomcat server it worked again.
> >>
> >>
> >>> Also in
> >>>
>  access logs there are no clues that it was under any heavy load.
> 
>  after around hour after discovering that our server hanged-out we have
>  restarted tomcat server
>  and it worked again
> 
> >>>
> >>> Yes, because that will close all connections between Apache httpd and
> >>> tomcat, and abort all requests that are in the process of being
> >>> processed
> >>> by tomcat. So mod_jk will get an error from tomcat, and will report an
> >>> error to httpd, and httpd will communicate that error to the clients,
> >>> and
> >>> close their connection.
> >>> It still does not tell you what the problem was.
> >>> The only thing that it suggests, is that the "bad" requests actually
> >>> make
> >>> it all the way to tomcat.
> >>>
> >>
> >> correct
> >>
> >> i will enable logs that you has pointed out and we will look what i will
> >> catch
> >> however i think we have only one chance, as if the solution we has found
> >> out (connection_timeout + mod_bn)
> >> will work they will stop exploiting it
> >>
> >> thank you very much for all the help and explanations
> >> i will report to the list new facts once they will attack us again
> >>
> >> best regards,
> >> artur
> >>
> >
> > Ok, but also read this e.g. :
> > https://www.corero.com/blog/695-going-after-the-people-
> behind-ddos-attacks.html
> >
> > Attempts to bring down a site by DoS attacks is a crime, in most places.
> > You can report it, while at the same 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
i written "exploited some well know vulnerability in mod_proxy" not mod_jk.

This one:
*moderate: * * mod_proxy denial of service * CVE-2014-0117


A flaw was found in mod_proxy in httpd versions 2.4.6 to 2.4.9. A remote
attacker could send a carefully crafted request to a server configured as a
reverse proxy, and cause the child process to crash. This could lead to a
denial of service against a threaded MPM.
we have updated apache and this module but they exploited it again with
newest version (sorry, no time to investigate this)
we switched to mod_jk but they exploited it too, rest you can read on list

anyway we have configured apache to mitigate such kind of attack (whatever
flavor it is)

best,
artur

2016-11-28 20:11 GMT+01:00 Christopher Schultz :

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Jaaz,
>
> On 11/27/16 1:03 PM, Jaaz Portal wrote:
> > Then they exploited some well know vulnerability in mod_proxy. We
> > have updated apache to the latest but again they has exploited it,
> > so we have switched to mod_jk. And then guess what. They exploited
> > it too so i decided to write to this list looking for help before
> > trying jetty.
>
> I'm unaware of any vulnerabilities in mod_jk, "well-known" or
> otherwise. What vulnerability are you talking about?
>
> Note that switching to Jetty won't solve anything if you think the
> "well-known vulnerability" is in mod_jk.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYPIFQAAoJEBzwKT+lPKRYUPkP/iV5vbBj/v3bDe4Yp9qHO0dA
> 5p740/FRYFicM6fZf+SZ8dqsnPKZ30Ij6PTFfwNRFkGHiszErY1RZ94s3ZdXYJRF
> QxNiSvToQq750ujwkDL4iZv8h4Ash0TRA/E6e0SvZ9W5yjT/FBSfxsSKe8uixN43
> 6xcXaJcTuipjoyNHnVlUFkZFcmMykg7hktcMeaGp9OI2j/4x+OkkWsTjxvEucDXx
> 9ediRNaldrnU+bvmMTKnI9FZSHmxg1hzkpky3gNOv+CF7gOCLo8/ixj8ZELb+Lwt
> AREr30gtBdupfmGLO0pdFJ2EYHfmfW3Rlf6xwCzSBHZtdX8ewR1dw28SCJDlq/um
> vRArMdG+s3OoLQ04GHRm+shom+X3KuhRf/NjZsF//we6XIpgxWv686HaVXPNa0Ys
> 6TomDIDM1N0o5l0mQ1799P2+/9t+hxUoGUcUIOUadn4QHtuGFONe266qYiguhuFj
> sDVsi7R6jK4VodTA2HznKrdUZf1WqE2bw5zLSi7AOuoSu85jDfN+uPRTD/Q3fw3a
> 4vYiyKSYijvHYlS0mi/mtuCMBIgUXHRKYT4SkR7W6SQlwy10QiRZUGxofnzUicO1
> eUDTNa1/pEecci7RXJ3C54aih0nrUZI5mQZdaFBBX1D7h3DAnTHKcnqzeQ0u/Wun
> 3Gf4MfVVYOfn9yyYmr7Q
> =bdyD
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jaaz,

On 11/27/16 1:03 PM, Jaaz Portal wrote:
> Then they exploited some well know vulnerability in mod_proxy. We
> have updated apache to the latest but again they has exploited it,
> so we have switched to mod_jk. And then guess what. They exploited
> it too so i decided to write to this list looking for help before
> trying jetty.

I'm unaware of any vulnerabilities in mod_jk, "well-known" or
otherwise. What vulnerability are you talking about?

Note that switching to Jetty won't solve anything if you think the
"well-known vulnerability" is in mod_jk.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPIFQAAoJEBzwKT+lPKRYUPkP/iV5vbBj/v3bDe4Yp9qHO0dA
5p740/FRYFicM6fZf+SZ8dqsnPKZ30Ij6PTFfwNRFkGHiszErY1RZ94s3ZdXYJRF
QxNiSvToQq750ujwkDL4iZv8h4Ash0TRA/E6e0SvZ9W5yjT/FBSfxsSKe8uixN43
6xcXaJcTuipjoyNHnVlUFkZFcmMykg7hktcMeaGp9OI2j/4x+OkkWsTjxvEucDXx
9ediRNaldrnU+bvmMTKnI9FZSHmxg1hzkpky3gNOv+CF7gOCLo8/ixj8ZELb+Lwt
AREr30gtBdupfmGLO0pdFJ2EYHfmfW3Rlf6xwCzSBHZtdX8ewR1dw28SCJDlq/um
vRArMdG+s3OoLQ04GHRm+shom+X3KuhRf/NjZsF//we6XIpgxWv686HaVXPNa0Ys
6TomDIDM1N0o5l0mQ1799P2+/9t+hxUoGUcUIOUadn4QHtuGFONe266qYiguhuFj
sDVsi7R6jK4VodTA2HznKrdUZf1WqE2bw5zLSi7AOuoSu85jDfN+uPRTD/Q3fw3a
4vYiyKSYijvHYlS0mi/mtuCMBIgUXHRKYT4SkR7W6SQlwy10QiRZUGxofnzUicO1
eUDTNa1/pEecci7RXJ3C54aih0nrUZI5mQZdaFBBX1D7h3DAnTHKcnqzeQ0u/Wun
3Gf4MfVVYOfn9yyYmr7Q
=bdyD
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache/Tomcat vulnerability

[Mark Eggers]

Jaaz,

On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
> On 27.11.2016 19:03, Jaaz Portal wrote:
>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :
>>
>>> On 27.11.2016 14:26, Jaaz Portal wrote:
>>>
 hi,
 everything i know so far is just this single log line that appeared in
 apache error.log

 [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
 1397934896385
 92] AH00484: server reached MaxRequestWorkers setting, consider raising
 the
 MaxR
 equestWorkers setting

 there was nothing else, just this strange line

>>>
>>> This is not a "strange" line. It is telling you something.
>>> One problem is that you seem convinced in advance, without serious
>>> proof,
>>> that there is a "bug" or a vulnerability in httpd or tomcat.
>>> Read the explanation of the httpd parameter, here :
>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
>>> and try to understand it.
>>>
>>
>> I understand it very well. We are serving no more that 500 clients per
>> day
>> so there was no other option that some kind of attack.
>>
>>
>> About the "bug" or "vulnerability" : a webserver is supposed to serve
>> HTTP
>>> requests from clients.  That is what you expect of it. It has no
>>> choice but
>>> to accept any client connection and request, up to the maximum it can
>>> handle considering its configuration and the capacity of the system on
>>> which it runs. That is not a bug, it is a feature.
>>>
>>>
>> We have some weeks ago come under attack from some Polish group. It was
>> first bind that was DoS'ed, we was running on stable Debian so i updated
>> bind to latest version. It did not helped. They has dos'ed it so we
>> switched to other dns provider. That has helped.
>>
>> Then they exploited some well know vulnerability in mod_proxy. We have
>> updated apache to the latest but again they has exploited it, so we have
>> switched to mod_jk. And then guess what. They exploited it too so i
>> decided
>> to write to this list looking for help before trying jetty.
>>
>>
>>>
>>> The normal Apache httpd access log, will log a request only when it is
>>> finished.  If the request never finishes, it will not get logged.
>>> That may be why you do not see these requests in the log.
>>> But have a look at this Apache httpd module :
>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
>>>
>>
>> ok, thanks, will take care
>>
>> Note : that is also why I was telling you to enable the mod_jk log,
>> and to
>>> examine it.
>>> Because mod_jk will also log information before the request produces a
>>> response.
>>>
>>>
>>> and server hanged.
>>>
>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
>>>
>>
>> Apache was accepting connection but not processed it. After restart of
>> tomcat server it worked again.
>>
>>
>>> Also in
>>>
 access logs there are no clues that it was under any heavy load.

 after around hour after discovering that our server hanged-out we have
 restarted tomcat server
 and it worked again

>>>
>>> Yes, because that will close all connections between Apache httpd and
>>> tomcat, and abort all requests that are in the process of being
>>> processed
>>> by tomcat. So mod_jk will get an error from tomcat, and will report an
>>> error to httpd, and httpd will communicate that error to the clients,
>>> and
>>> close their connection.
>>> It still does not tell you what the problem was.
>>> The only thing that it suggests, is that the "bad" requests actually
>>> make
>>> it all the way to tomcat.
>>>
>>
>> correct
>>
>> i will enable logs that you has pointed out and we will look what i will
>> catch
>> however i think we have only one chance, as if the solution we has found
>> out (connection_timeout + mod_bn)
>> will work they will stop exploiting it
>>
>> thank you very much for all the help and explanations
>> i will report to the list new facts once they will attack us again
>>
>> best regards,
>> artur
>>
> 
> Ok, but also read this e.g. :
> https://www.corero.com/blog/695-going-after-the-people-behind-ddos-attacks.html
> 
> Attempts to bring down a site by DoS attacks is a crime, in most places.
> You can report it, while at the same time trying to defend yourself
> against it.
> 
> It is also relatively easy, and quite inexpensive in terms of system
> resources, to run a small shell script which takes a list every few
> seconds of the connections to the port of your webserver, and which IPs
> they are coming *from*.
> E.g.
> First try the netstat command, to see what it lists, like :
> # netstat -n --tcp | more
> 
> Then you will want to filter this a bit, to only consider established
> connections to your webserver, for example :
> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED"
> 
> Then you will want to send this to a logfile, regularly, like this :
> 
> # date >> some_file.log
> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> some_file.log
> 

Re: Apache/Tomcat vulnerability

[tomcat]

On 27.11.2016 19:03, Jaaz Portal wrote:

2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :


On 27.11.2016 14:26, Jaaz Portal wrote:


hi,
everything i know so far is just this single log line that appeared in
apache error.log

[Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
1397934896385
92] AH00484: server reached MaxRequestWorkers setting, consider raising
the
MaxR
equestWorkers setting

there was nothing else, just this strange line



This is not a "strange" line. It is telling you something.
One problem is that you seem convinced in advance, without serious proof,
that there is a "bug" or a vulnerability in httpd or tomcat.
Read the explanation of the httpd parameter, here :
http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
and try to understand it.



I understand it very well. We are serving no more that 500 clients per day
so there was no other option that some kind of attack.


About the "bug" or "vulnerability" : a webserver is supposed to serve HTTP

requests from clients.  That is what you expect of it. It has no choice but
to accept any client connection and request, up to the maximum it can
handle considering its configuration and the capacity of the system on
which it runs. That is not a bug, it is a feature.



We have some weeks ago come under attack from some Polish group. It was
first bind that was DoS'ed, we was running on stable Debian so i updated
bind to latest version. It did not helped. They has dos'ed it so we
switched to other dns provider. That has helped.

Then they exploited some well know vulnerability in mod_proxy. We have
updated apache to the latest but again they has exploited it, so we have
switched to mod_jk. And then guess what. They exploited it too so i decided
to write to this list looking for help before trying jetty.




The normal Apache httpd access log, will log a request only when it is
finished.  If the request never finishes, it will not get logged.
That may be why you do not see these requests in the log.
But have a look at this Apache httpd module :
http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html



ok, thanks, will take care

Note : that is also why I was telling you to enable the mod_jk log, and to

examine it.
Because mod_jk will also log information before the request produces a
response.


and server hanged.

Again, /what/ is "hanged" ? Apache httpd, or tomcat ?



Apache was accepting connection but not processed it. After restart of
tomcat server it worked again.



Also in


access logs there are no clues that it was under any heavy load.

after around hour after discovering that our server hanged-out we have
restarted tomcat server
and it worked again



Yes, because that will close all connections between Apache httpd and
tomcat, and abort all requests that are in the process of being processed
by tomcat. So mod_jk will get an error from tomcat, and will report an
error to httpd, and httpd will communicate that error to the clients, and
close their connection.
It still does not tell you what the problem was.
The only thing that it suggests, is that the "bad" requests actually make
it all the way to tomcat.



correct

i will enable logs that you has pointed out and we will look what i will
catch
however i think we have only one chance, as if the solution we has found
out (connection_timeout + mod_bn)
will work they will stop exploiting it

thank you very much for all the help and explanations
i will report to the list new facts once they will attack us again

best regards,
artur



Ok, but also read this e.g. :
https://www.corero.com/blog/695-going-after-the-people-behind-ddos-attacks.html
Attempts to bring down a site by DoS attacks is a crime, in most places.
You can report it, while at the same time trying to defend yourself against it.

It is also relatively easy, and quite inexpensive in terms of system resources, to run a 
small shell script which takes a list every few seconds of the connections to the port of 
your webserver, and which IPs they are coming *from*.

E.g.
First try the netstat command, to see what it lists, like :
# netstat -n --tcp | more

Then you will want to filter this a bit, to only consider established connections to your 
webserver, for example :

# netstat -n --tcp | grep ":80" | grep "ESTABLISHED"

Then you will want to send this to a logfile, regularly, like this :

# date >> some_file.log
# netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> some_file.log
(repeat every 3 seconds)

This will not generate GB of logfiles, and it will tell you, when the problem happens, how 
many connections there are exactly to your webserver, and where they are coming from.

Then later you can further analyse this information..





i think that setting connection-timeout and limiting the number of clients
by mod_bd i will
have effect that next time somebody will try this exploit it will block
his
access to the site
for minute or two, pretty good holistic 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :

> On 27.11.2016 14:26, Jaaz Portal wrote:
>
>> hi,
>> everything i know so far is just this single log line that appeared in
>> apache error.log
>>
>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
>> 1397934896385
>> 92] AH00484: server reached MaxRequestWorkers setting, consider raising
>> the
>> MaxR
>> equestWorkers setting
>>
>> there was nothing else, just this strange line
>>
>
> This is not a "strange" line. It is telling you something.
> One problem is that you seem convinced in advance, without serious proof,
> that there is a "bug" or a vulnerability in httpd or tomcat.
> Read the explanation of the httpd parameter, here :
> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
> and try to understand it.
>

I understand it very well. We are serving no more that 500 clients per day
so there was no other option that some kind of attack.


About the "bug" or "vulnerability" : a webserver is supposed to serve HTTP
> requests from clients.  That is what you expect of it. It has no choice but
> to accept any client connection and request, up to the maximum it can
> handle considering its configuration and the capacity of the system on
> which it runs. That is not a bug, it is a feature.
>
>
We have some weeks ago come under attack from some Polish group. It was
first bind that was DoS'ed, we was running on stable Debian so i updated
bind to latest version. It did not helped. They has dos'ed it so we
switched to other dns provider. That has helped.

Then they exploited some well know vulnerability in mod_proxy. We have
updated apache to the latest but again they has exploited it, so we have
switched to mod_jk. And then guess what. They exploited it too so i decided
to write to this list looking for help before trying jetty.


>
> The normal Apache httpd access log, will log a request only when it is
> finished.  If the request never finishes, it will not get logged.
> That may be why you do not see these requests in the log.
> But have a look at this Apache httpd module :
> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
>

ok, thanks, will take care

Note : that is also why I was telling you to enable the mod_jk log, and to
> examine it.
> Because mod_jk will also log information before the request produces a
> response.
>
>
> and server hanged.
>
> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
>

Apache was accepting connection but not processed it. After restart of
tomcat server it worked again.


> Also in
>
>> access logs there are no clues that it was under any heavy load.
>>
>> after around hour after discovering that our server hanged-out we have
>> restarted tomcat server
>> and it worked again
>>
>
> Yes, because that will close all connections between Apache httpd and
> tomcat, and abort all requests that are in the process of being processed
> by tomcat. So mod_jk will get an error from tomcat, and will report an
> error to httpd, and httpd will communicate that error to the clients, and
> close their connection.
> It still does not tell you what the problem was.
> The only thing that it suggests, is that the "bad" requests actually make
> it all the way to tomcat.
>

correct

i will enable logs that you has pointed out and we will look what i will
catch
however i think we have only one chance, as if the solution we has found
out (connection_timeout + mod_bn)
will work they will stop exploiting it

thank you very much for all the help and explanations
i will report to the list new facts once they will attack us again

best regards,
artur



>> i think that setting connection-timeout and limiting the number of clients
>> by mod_bd i will
>> have effect that next time somebody will try this exploit it will block
>> his
>> access to the site
>> for minute or two, pretty good holistic solution i would say
>>
>> still, it seems that there is serious vulnerability somewhere in apache,
>> mod_jk or tomcat
>> i would like to help find it out but need some hints which debug options
>> enable to catch the bad guys
>> when they will try next time
>>
>> best regards,
>> artur
>>
>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) :
>>
>> On 27.11.2016 13:23, Jaaz Portal wrote:
>>>
>>> hi Andre,
 thank you very much this was very educative but in my case it is little
 bit
 different.
 The server is no flooded, there is maybe dozen of very sophisticated
 connections that somehow hangs apache workers threads


>>> Can you be a bit more specific ?
>>> When you say "apache workers threads", do you mean threads in Apache
>>> httpd, or threads in Apache Tomcat ? (both are Apache webservers, so it
>>> is
>>> difficult to tell what you are talking about, unless you are very
>>> precise).
>>>
>>> Let me give you some additional explanations, and maybe you can figure
>>> out
>>> exactly where it "hangs".
>>>
>>>  From the Apache httpd 

Re: Apache/Tomcat vulnerability

[tomcat]

On 27.11.2016 14:26, Jaaz Portal wrote:

hi,
everything i know so far is just this single log line that appeared in
apache error.log

[Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
1397934896385
92] AH00484: server reached MaxRequestWorkers setting, consider raising the
MaxR
equestWorkers setting

there was nothing else, just this strange line


This is not a "strange" line. It is telling you something.
One problem is that you seem convinced in advance, without serious proof, that there is a 
"bug" or a vulnerability in httpd or tomcat.

Read the explanation of the httpd parameter, here :
http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
and try to understand it.

If Apache httpd is writing that line to the log, it means that it has reached the maximum 
number of requests that it is allowed to serve at the same time, considering your 
configuration. That could be normal - if your server is supposed to be heavily loaded - or 
it could be the sign of a "flood" attack. But you seem to be convinced in advance that 
there is no flood. You also do not really answer the questions which you are asked, which 
makes helping you difficult.


About the "bug" or "vulnerability" : a webserver is supposed to serve HTTP requests from 
clients.  That is what you expect of it. It has no choice but to accept any client 
connection and request, up to the maximum it can handle considering its configuration and 
the capacity of the system on which it runs. That is not a bug, it is a feature.


It does not know in advance that these requests are bad. To find out that they are bad, it 
needs to accept the request and read it. For that, there needs to be a process or a thread 
available to do that.
If it accepts the request, and if the URL of the request is in principle valid, it will 
try to process that request.  If there is an application on your server responding to that 
request, then if it takes a long time or freezes the thread which processes the request, 
it may be a bug in your application.


(Note : there /may/ be an unknown vulnerability in Apache httpd or tomcat; but is not 
necessarily so. And there are millions of Apache webservers on the Internet; so if they 
had a bug like that, you would not be the only one to report it.  So be humble and 
realistic; try your best to find out what really happens, before you cry wolf).


The normal Apache httpd access log, will log a request only when it is finished.  If the 
request never finishes, it will not get logged.

That may be why you do not see these requests in the log.
But have a look at this Apache httpd module : 
http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html


Note : that is also why I was telling you to enable the mod_jk log, and to 
examine it.
Because mod_jk will also log information before the request produces a response.


and server hanged.

Again, /what/ is "hanged" ? Apache httpd, or tomcat ?

Also in

access logs there are no clues that it was under any heavy load.

after around hour after discovering that our server hanged-out we have
restarted tomcat server
and it worked again


Yes, because that will close all connections between Apache httpd and tomcat, and abort 
all requests that are in the process of being processed by tomcat. So mod_jk will get an 
error from tomcat, and will report an error to httpd, and httpd will communicate that 
error to the clients, and close their connection.

It still does not tell you what the problem was.
The only thing that it suggests, is that the "bad" requests actually make it all the way 
to tomcat.




i think that setting connection-timeout and limiting the number of clients
by mod_bd i will
have effect that next time somebody will try this exploit it will block his
access to the site
for minute or two, pretty good holistic solution i would say

still, it seems that there is serious vulnerability somewhere in apache,
mod_jk or tomcat
i would like to help find it out but need some hints which debug options
enable to catch the bad guys
when they will try next time

best regards,
artur

2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) :


On 27.11.2016 13:23, Jaaz Portal wrote:


hi Andre,
thank you very much this was very educative but in my case it is little
bit
different.
The server is no flooded, there is maybe dozen of very sophisticated
connections that somehow hangs apache workers threads



Can you be a bit more specific ?
When you say "apache workers threads", do you mean threads in Apache
httpd, or threads in Apache Tomcat ? (both are Apache webservers, so it is
difficult to tell what you are talking about, unless you are very precise).

Let me give you some additional explanations, and maybe you can figure out
exactly where it "hangs".

 From the Apache httpd front-end point of view, mod_jk (the connector to
Apache Tomcat) is basically one among other "response generators".  Apache
httpd does not "know" that behind mod_jk, there is one 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
everything i know so far is just this single log line that appeared in
apache error.log

[Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
1397934896385
92] AH00484: server reached MaxRequestWorkers setting, consider raising the
MaxR
equestWorkers setting

there was nothing else, just this strange line and server hanged. Also in
access logs there are no clues that it was under any heavy load.

after around hour after discovering that our server hanged-out we have
restarted tomcat server
and it worked again

i think that setting connection-timeout and limiting the number of clients
by mod_bd i will
have effect that next time somebody will try this exploit it will block his
access to the site
for minute or two, pretty good holistic solution i would say

still, it seems that there is serious vulnerability somewhere in apache,
mod_jk or tomcat
i would like to help find it out but need some hints which debug options
enable to catch the bad guys
when they will try next time

best regards,
artur

2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) :

> On 27.11.2016 13:23, Jaaz Portal wrote:
>
>> hi Andre,
>> thank you very much this was very educative but in my case it is little
>> bit
>> different.
>> The server is no flooded, there is maybe dozen of very sophisticated
>> connections that somehow hangs apache workers threads
>>
>
> Can you be a bit more specific ?
> When you say "apache workers threads", do you mean threads in Apache
> httpd, or threads in Apache Tomcat ? (both are Apache webservers, so it is
> difficult to tell what you are talking about, unless you are very precise).
>
> Let me give you some additional explanations, and maybe you can figure out
> exactly where it "hangs".
>
> From the Apache httpd front-end point of view, mod_jk (the connector to
> Apache Tomcat) is basically one among other "response generators".  Apache
> httpd does not "know" that behind mod_jk, there is one or more Tomcat
> servers.  Apache httpd receives the original client request, and depending
> on the URL of the request, it will pass it to mod_jk or to another response
> generator, to generate the response to the request.
> That mod_jk in the background is using a Tomcat server to actually
> generate the response, is none of the business of Apache httpd, and it does
> not care. All it cares about, is to actually receive the response from
> mod_jk, and pass it back to the client.
>
> If httpd passes a request to mod_jk, it is because you have specified in
> the Apache configuration, the type of URL that it should pass to mod_jk.
> That happens via your "JkMount (URL pattern)" directives in Apache httpd.
>
> Of course Apache httpd will not pass a request to mod_jk, before it has
> received at least the URL of the request, and analysed it to determine *if*
> it should pass it to mod_jk (*).
>
> If the mod_jk logging is enabled, you can see in it, exactly *which*
> requests are passed to mod_jk and to Tomcat.
> Do you know *which* requests, from which clients, cause this "thread
> hanging" symptom ?
> Once you would know this, maybe you can design a strategy to block
> specifically these requests.
>
> and the effect is permanent. Quickly the pool is exhausted
>>
>
> which pool exactly ?
>
> and the only
>
>> solution that works is to restart tomcat.
>>
>> I think it is a bug similar to this one from mod_proxy:
>> https://tools.cisco.com/security/center/viewAlert.x?alertId=34971
>>
>>
> Maybe, maybe not. As long as we do not know what the requests are, that
> block things, we do not know this.
>
>
> I think also that your solution with setting connectionTimeout will solve
>> the problem, at least partially. THANK YOU.
>>
>
> Same thing, we do not know this yet.  It was only giving this explanation,
> to help you think about where the problem may be.
>
>
>> I would like to help you further investigate this issue, as our server
>> comes under such attack once or twice in a week.
>>
>>
> Other than giving you hints, there is not much I or anyone else can do to
> help. You are the one with control of your servers and logfiles, so you
> have to investigate and provide more precise information.
>
>
> (*) actually, to be precise, Apache httpd passes *all* requests to mod_jk,
> to ask it "if it wants that request". mod_jk then accepts or declines,
> depending on the JkMount instructions. If mod_jk declines, then Apache
> httpd will ask the next response generator if it wants this request, etc..
>
>
>
>
>
> best regards,
>> artur
>>
>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) :
>>
>> Hi.
>>>
>>> Have a look that the indicated parameters in the two pages below.
>>> You may be the target of such a variant of DDoS attack : many clients
>>> open
>>> a TCP connection to your server (front-end), but then never sends a HTTP
>>> request on that connection.  In the meantime, the server accepts the TCP
>>> connection, and passes it on to a "child" process or thread for
>>> 

Re: Apache/Tomcat vulnerability

[tomcat]

On 27.11.2016 13:23, Jaaz Portal wrote:

hi Andre,
thank you very much this was very educative but in my case it is little bit
different.
The server is no flooded, there is maybe dozen of very sophisticated
connections that somehow hangs apache workers threads


Can you be a bit more specific ?
When you say "apache workers threads", do you mean threads in Apache httpd, or threads in 
Apache Tomcat ? (both are Apache webservers, so it is difficult to tell what you are 
talking about, unless you are very precise).


Let me give you some additional explanations, and maybe you can figure out exactly where 
it "hangs".


From the Apache httpd front-end point of view, mod_jk (the connector to Apache Tomcat) is 
basically one among other "response generators".  Apache httpd does not "know" that behind 
mod_jk, there is one or more Tomcat servers.  Apache httpd receives the original client 
request, and depending on the URL of the request, it will pass it to mod_jk or to another 
response generator, to generate the response to the request.
That mod_jk in the background is using a Tomcat server to actually generate the response, 
is none of the business of Apache httpd, and it does not care. All it cares about, is to 
actually receive the response from mod_jk, and pass it back to the client.


If httpd passes a request to mod_jk, it is because you have specified in the Apache 
configuration, the type of URL that it should pass to mod_jk.

That happens via your "JkMount (URL pattern)" directives in Apache httpd.

Of course Apache httpd will not pass a request to mod_jk, before it has received at least 
the URL of the request, and analysed it to determine *if* it should pass it to mod_jk (*).


If the mod_jk logging is enabled, you can see in it, exactly *which* requests are passed 
to mod_jk and to Tomcat.

Do you know *which* requests, from which clients, cause this "thread hanging" 
symptom ?
Once you would know this, maybe you can design a strategy to block specifically these 
requests.



and the effect is permanent. Quickly the pool is exhausted


which pool exactly ?

and the only

solution that works is to restart tomcat.

I think it is a bug similar to this one from mod_proxy:
https://tools.cisco.com/security/center/viewAlert.x?alertId=34971



Maybe, maybe not. As long as we do not know what the requests are, that block things, we 
do not know this.




I think also that your solution with setting connectionTimeout will solve
the problem, at least partially. THANK YOU.


Same thing, we do not know this yet.  It was only giving this explanation, to help you 
think about where the problem may be.




I would like to help you further investigate this issue, as our server
comes under such attack once or twice in a week.



Other than giving you hints, there is not much I or anyone else can do to help. You are 
the one with control of your servers and logfiles, so you have to investigate and provide 
more precise information.



(*) actually, to be precise, Apache httpd passes *all* requests to mod_jk, to ask it "if 
it wants that request". mod_jk then accepts or declines, depending on the JkMount 
instructions. If mod_jk declines, then Apache httpd will ask the next response generator 
if it wants this request, etc..






best regards,
artur

2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) :


Hi.

Have a look that the indicated parameters in the two pages below.
You may be the target of such a variant of DDoS attack : many clients open
a TCP connection to your server (front-end), but then never sends a HTTP
request on that connection.  In the meantime, the server accepts the TCP
connection, and passes it on to a "child" process or thread for
processing.  The child then waits for the HTTP request line to arrive on
the connection (during a certain time), but it never arrives. After a
while, this triggers a timeout (see below), but the standard value of that
timeout may be such that in the meantime, a lot of other connections have
been established by other such nefarious clients, so a lot of resources of
the webserver are tied up, waiting for something that will never come.
Since there is never any real request sent on the connection, you would
(probably) not see this in the logs either.

The above is the basic mechanism of such an attack.  There may be
variations, such as the client not "not sending" a request line, but
sending it extremely slowly, thus achieving perhaps similar kinds of
effects.

As someone pointed out, it is quite difficult to do something about this
at the level of the webserver itself, because by the time you would do
something about it, the resources have already been consumed and your
server is probably already overloaded.
There are specialised front-end devices and software available, to detect
and protect against this kind of attack.

You may want to have a look at the following parameters, but make sure to
read the caveats (side-effects, interlocking timeouts 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi Andre,
thank you very much this was very educative but in my case it is little bit
different.
The server is no flooded, there is maybe dozen of very sophisticated
connections that somehow hangs apache workers threads
and the effect is permanent. Quickly the pool is exhausted and the only
solution that works is to restart tomcat.

I think it is a bug similar to this one from mod_proxy:
https://tools.cisco.com/security/center/viewAlert.x?alertId=34971

I think also that your solution with setting connectionTimeout will solve
the problem, at least partially. THANK YOU.

I would like to help you further investigate this issue, as our server
comes under such attack once or twice in a week.

best regards,
artur

2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) :

> Hi.
>
> Have a look that the indicated parameters in the two pages below.
> You may be the target of such a variant of DDoS attack : many clients open
> a TCP connection to your server (front-end), but then never sends a HTTP
> request on that connection.  In the meantime, the server accepts the TCP
> connection, and passes it on to a "child" process or thread for
> processing.  The child then waits for the HTTP request line to arrive on
> the connection (during a certain time), but it never arrives. After a
> while, this triggers a timeout (see below), but the standard value of that
> timeout may be such that in the meantime, a lot of other connections have
> been established by other such nefarious clients, so a lot of resources of
> the webserver are tied up, waiting for something that will never come.
> Since there is never any real request sent on the connection, you would
> (probably) not see this in the logs either.
>
> The above is the basic mechanism of such an attack.  There may be
> variations, such as the client not "not sending" a request line, but
> sending it extremely slowly, thus achieving perhaps similar kinds of
> effects.
>
> As someone pointed out, it is quite difficult to do something about this
> at the level of the webserver itself, because by the time you would do
> something about it, the resources have already been consumed and your
> server is probably already overloaded.
> There are specialised front-end devices and software available, to detect
> and protect against this kind of attack.
>
> You may want to have a look at the following parameters, but make sure to
> read the caveats (side-effects, interlocking timeouts etc.), otherwise you
> may do more harm than good.
>
> Another thing : the settings below are for Apache Tomcat, which in your
> case is the back-end. It would of course be much better to detect and
> eliminate this at the front-end, or even before.  I had a look at the
> Apache httpd documentation, and could not find a corresponding parameter.
> But I am sure that it must exist. You may want to post this same question
> on the Apache httpd user's list for a better response.
>
> Tomcat configuration settings :
>
> AJP Connector : (http://tomcat.apache.org/tomcat-8.5-doc/config/ajp.html#
> Standard_Implementations)
>
> connectionTimeout
>
> The number of milliseconds this Connector will wait, after accepting a
> connection, for the request URI line to be presented. The default value for
> AJP protocol connectors is -1 (i.e. infinite).
>
> (You could for example try to set this to 3000 (milliseconds) or even
> lower. That should be more than enough for any legitimate client so send
> the HTTP request line.  Note however that by doing this at the Tomcat
> level, you will probably move the problem to the Apache httpd/mod_jk
> level.  But at least it might confirm that this is the problem that you are
> seeing.  The mod_jk logfile at the httpd level may give you some hints
> there.)
>
>
> HTTP Connector : (http://tomcat.apache.org/tomc
> at-8.5-doc/config/http.html#Standard_Implementation)
>
> connectionTimeout
>
> The number of milliseconds this Connector will wait, after accepting a
> connection, for the request URI line to be presented. Use a value of -1 to
> indicate no (i.e. infinite) timeout. The default value is 6 (i.e. 60
> seconds) but note that the standard server.xml that ships with Tomcat sets
> this to 2 (i.e. 20 seconds). Unless disableUploadTimeout is set to
> false, this timeout will also be used when reading the request body (if
> any).
>
>
>
> On 26.11.2016 09:57, Jaaz Portal wrote:
>
>> hi,
>> sorry, its mod_jk no jk2, my typo. All at latest versions. We tried with
>> mod proxy too.
>>
>> There is no flood of the server. Nobody is flooding us, they use some
>> specific connections after which pool of apache workers is exhausted and
>> blocked
>> and we need to restart tomcat server.
>> It is some kind of exploit but do not know how to log it to obtain
>> details.
>>
>> i had put a limit on connections per client with hope that this will help
>> but once again, it is not a flood.
>> They open several connections that are not dropped by apache when they
>> 

Re: Apache/Tomcat vulnerability

[tomcat]

Hi.

Have a look that the indicated parameters in the two pages below.
You may be the target of such a variant of DDoS attack : many clients open a TCP 
connection to your server (front-end), but then never sends a HTTP request on that 
connection.  In the meantime, the server accepts the TCP connection, and passes it on to a 
"child" process or thread for processing.  The child then waits for the HTTP request line 
to arrive on the connection (during a certain time), but it never arrives. After a while, 
this triggers a timeout (see below), but the standard value of that timeout may be such 
that in the meantime, a lot of other connections have been established by other such 
nefarious clients, so a lot of resources of the webserver are tied up, waiting for 
something that will never come.
Since there is never any real request sent on the connection, you would (probably) not see 
this in the logs either.


The above is the basic mechanism of such an attack.  There may be variations, such as the 
client not "not sending" a request line, but sending it extremely slowly, thus achieving 
perhaps similar kinds of effects.


As someone pointed out, it is quite difficult to do something about this at the level of 
the webserver itself, because by the time you would do something about it, the resources 
have already been consumed and your server is probably already overloaded.
There are specialised front-end devices and software available, to detect and protect 
against this kind of attack.


You may want to have a look at the following parameters, but make sure to read the caveats 
(side-effects, interlocking timeouts etc.), otherwise you may do more harm than good.


Another thing : the settings below are for Apache Tomcat, which in your case is the 
back-end. It would of course be much better to detect and eliminate this at the front-end, 
or even before.  I had a look at the Apache httpd documentation, and could not find a 
corresponding parameter. But I am sure that it must exist. You may want to post this same 
question on the Apache httpd user's list for a better response.


Tomcat configuration settings :

AJP Connector : 
(http://tomcat.apache.org/tomcat-8.5-doc/config/ajp.html#Standard_Implementations)


connectionTimeout   

The number of milliseconds this Connector will wait, after accepting a connection, for the 
request URI line to be presented. The default value for AJP protocol connectors is -1 
(i.e. infinite).


(You could for example try to set this to 3000 (milliseconds) or even lower. That should 
be more than enough for any legitimate client so send the HTTP request line.  Note however 
that by doing this at the Tomcat level, you will probably move the problem to the Apache 
httpd/mod_jk level.  But at least it might confirm that this is the problem that you are 
seeing.  The mod_jk logfile at the httpd level may give you some hints there.)



HTTP Connector : 
(http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Standard_Implementation)


connectionTimeout   

The number of milliseconds this Connector will wait, after accepting a connection, for the 
request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) 
timeout. The default value is 6 (i.e. 60 seconds) but note that the standard 
server.xml that ships with Tomcat sets this to 2 (i.e. 20 seconds). Unless 
disableUploadTimeout is set to false, this timeout will also be used when reading the 
request body (if any).



On 26.11.2016 09:57, Jaaz Portal wrote:

hi,
sorry, its mod_jk no jk2, my typo. All at latest versions. We tried with
mod proxy too.

There is no flood of the server. Nobody is flooding us, they use some
specific connections after which pool of apache workers is exhausted and
blocked
and we need to restart tomcat server.
It is some kind of exploit but do not know how to log it to obtain details.

i had put a limit on connections per client with hope that this will help
but once again, it is not a flood.
They open several connections that are not dropped by apache when they
disconnect. This way whole pool is quickly exhausted and the server broken.

i would like to help you to figure details of this attack but this is
production server so it is impossible to much debugging options

best,
artur

2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu :


you can find who is flooding site in apache access.log and block them in
firewall.

ex to find the IP:

cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort -gr



On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal  wrote:


hi,
we are from some weeks struggling with some Polish hackers that are
bringing our server down. After updating apache to latest version

(2.4.23)

and tomcat (8.0.38) available for debian systems we still cannot secure

our

server.

Today it has stopped to respond again and we needed to restart tomcat
process to get it back alive.

There is no too much clues 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
sorry, its mod_jk no jk2, my typo. All at latest versions. We tried with
mod proxy too.

There is no flood of the server. Nobody is flooding us, they use some
specific connections after which pool of apache workers is exhausted and
blocked
and we need to restart tomcat server.
It is some kind of exploit but do not know how to log it to obtain details.

i had put a limit on connections per client with hope that this will help
but once again, it is not a flood.
They open several connections that are not dropped by apache when they
disconnect. This way whole pool is quickly exhausted and the server broken.

i would like to help you to figure details of this attack but this is
production server so it is impossible to much debugging options

best,
artur

2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu :

> you can find who is flooding site in apache access.log and block them in
> firewall.
>
> ex to find the IP:
>
> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort -gr
>
>
>
> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal  wrote:
>
> > hi,
> > we are from some weeks struggling with some Polish hackers that are
> > bringing our server down. After updating apache to latest version
> (2.4.23)
> > and tomcat (8.0.38) available for debian systems we still cannot secure
> our
> > server.
> >
> > Today it has stopped to respond again and we needed to restart tomcat
> > process to get it back alive.
> >
> > There is no too much clues in the logs. The apache error.log gives just
> > this line:
> >
> > [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
> > 1397934896385
> > 92] AH00484: server reached MaxRequestWorkers setting, consider raising
> the
> > MaxR
> > equestWorkers setting
> >
> > seems that somehow tomcat, mod-jk2 or even apache is vulnerable to some
> new
> > exploit, as we certainly does not have such traffic that would block our
> > server otherwise
> >
> > for now we have increased MaxRequestWorkers and we have limited number of
> > connections from one client to 5 by mod_bw and limited number of
> > simultaneous connections from one ip by iptables but does not know if
> this
> > will help
> >
> > best regards,
> > artur
> >
>
>
>
> --
> *Thanks*
> *Niranjan*
>

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Artur,

On 11/25/16 8:42 AM, Jaaz Portal wrote:
> hi, we are from some weeks struggling with some Polish hackers that
> are bringing our server down. After updating apache to latest
> version (2.4.23) and tomcat (8.0.38) available for debian systems
> we still cannot secure our server.
> 
> Today it has stopped to respond again and we needed to restart
> tomcat process to get it back alive.
> 
> There is no too much clues in the logs. The apache error.log gives
> just this line:
> 
> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid 
> 1397934896385 92] AH00484: server reached MaxRequestWorkers
> setting, consider raising the MaxR equestWorkers setting
> 
> seems that somehow tomcat, mod-jk2 or even apache is vulnerable to
> some new exploit, as we certainly does not have such traffic that
> would block our server otherwise
> 
> for now we have increased MaxRequestWorkers and we have limited
> number of connections from one client to 5 by mod_bw and limited
> number of simultaneous connections from one ip by iptables but does
> not know if this will help

1. There is no indication here of any exploit being used against any
vulnerability. This sounds like a DDOS attack, and server software
really can't be used to mitigate these kinds of things. Instead, you
need to work at the network level to drop the malicious traffic before
it gets to your servers.

2. mod-jk2? That project was abandoned over a decade ago. Please
discontinue using it this instant. Seriously, go replace it with
mod_jk or mod_proxy_ajp NOW, and then come back to reply.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYON2rAAoJEBzwKT+lPKRYYnQQAJ3EVA56G5LiINdUXL4Ll1wA
jSk0G6hPbz02r8K4YNRrb4CiItZIYNQtmzMZQN3cGXptu+wp+sZE6XjvKc1fVcsi
0vnkZJt7sXvaU8s2JYT5XEun+XKmoexFRzBOvm0k+S7FuUiiJve0KeSW04jeNDBW
4JwBQ8VSn3sucpP9W2qvM0MieQAvViCCxI4iaCBJZi1Pa+z3sNKvHpUNFfmRAf4e
RfJdf0IAahLIXgWADtpLhPVzdBTpqsccNHHVCHSWGrSn/JAtaAbEbdnsG0J7Ds6d
ep8p+fJEwT5jL4s76e7d4Yr5d0cTrxpCWjZp+4eXxxSDWam+kdeShs1iIBxxe0+N
5WEiHTN4gSdTHGElb6owip5HkNE8HlFnlEyyP/tesDl2faHFrKqShzvbJKqaX7Za
FYnKzccoUL4oXldisl+q1Barvb4kUWocp4XxbbaxjzYNCfrZ01NvqneANtBR1r6w
38IyiWw7U9GGhAwu2ZAYR7pDETVIpgzCfO1loft8bhF3tW0NlY7W0cGTBx/kmkd/
I5jXKTQFbbfyf1ra23a1umLMMyppb38k073MMcji+HngDo+po9GsIOh9ox5puAH6
oFB65IG8XhvRtgC2AVognjt8kZizIb8iK/ShleM/YsQ+zf/HPFUha32Q2XTDLhUU
xvD+NkYL/TAvl8ysJsL8
=WTRu
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache/Tomcat vulnerability

[Niranjan Babu Bommu]

you can find who is flooding site in apache access.log and block them in
firewall.

ex to find the IP:

cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort -gr



On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal  wrote:

> hi,
> we are from some weeks struggling with some Polish hackers that are
> bringing our server down. After updating apache to latest version (2.4.23)
> and tomcat (8.0.38) available for debian systems we still cannot secure our
> server.
>
> Today it has stopped to respond again and we needed to restart tomcat
> process to get it back alive.
>
> There is no too much clues in the logs. The apache error.log gives just
> this line:
>
> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
> 1397934896385
> 92] AH00484: server reached MaxRequestWorkers setting, consider raising the
> MaxR
> equestWorkers setting
>
> seems that somehow tomcat, mod-jk2 or even apache is vulnerable to some new
> exploit, as we certainly does not have such traffic that would block our
> server otherwise
>
> for now we have increased MaxRequestWorkers and we have limited number of
> connections from one client to 5 by mod_bw and limited number of
> simultaneous connections from one ip by iptables but does not know if this
> will help
>
> best regards,
> artur
>



-- 
*Thanks*
*Niranjan*

Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
we are from some weeks struggling with some Polish hackers that are
bringing our server down. After updating apache to latest version (2.4.23)
and tomcat (8.0.38) available for debian systems we still cannot secure our
server.

Today it has stopped to respond again and we needed to restart tomcat
process to get it back alive.

There is no too much clues in the logs. The apache error.log gives just
this line:

[Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
1397934896385
92] AH00484: server reached MaxRequestWorkers setting, consider raising the
MaxR
equestWorkers setting

seems that somehow tomcat, mod-jk2 or even apache is vulnerable to some new
exploit, as we certainly does not have such traffic that would block our
server otherwise

for now we have increased MaxRequestWorkers and we have limited number of
connections from one client to 5 by mod_bw and limited number of
simultaneous connections from one ip by iptables but does not know if this
will help

best regards,
artur