Mitigation Range header (Was: DoS with mod_deflate range requests)
Folks, This issue is now active in the wild. So some unified/simple comms is needed. What is the wisdom on mitigation advise/briefing until a proper fix it out - in order of ease: - Where possible - disable mod_deflate = we sure this covers all cases - or this is a good stopgap ? - Where possible - set LimitRequestFieldSize to a small value - Suggesting of 128 fine ? - Where this is not possible (e.g. long cookies, auth headers of serious size) consider using mod_rewrite to not accept more than a few commas = anyone a config snipped for this ? - Perhaps a stop gap module http://people.apache.org/~dirkx/mod_rangecnt.c (is this kosher??) - Apply patch XXX from the mailing list Any thoughts ? Followed by a - upgrade as soon as a release is made Thanks, Dw
RE: Mitigation Range header (Was: DoS with mod_deflate range requests)
-Original Message-
From: Dirk-Willem van Gulik
Sent: Mittwoch, 24. August 2011 13:33
To: dev@httpd.apache.org
Subject: Mitigation Range header (Was: DoS with mod_deflate
range requests)
Folks,
This issue is now active in the wild. So some unified/simple
comms is needed.
What is the wisdom on mitigation advise/briefing until a
proper fix it out - in order of ease:
-Where possible - disable mod_deflate
= we sure this covers all cases - or this is a good stopgap ?
As said this has *nothing* to do with mod_deflate. This was IMHO just
a guess by the original author of the tool.
-Where possible - set LimitRequestFieldSize to a small value
- Suggesting of 128 fine ?
-Where this is not possible (e.g. long cookies, auth
headers of serious size) consider using
mod_rewrite to not accept more than a few commas
= anyone a config snipped for this ?
How about the following (untested) rewrite rule. It should only allow 5
ranges at max.
RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
Regards
Rüdiger
Re: Mitigation Range header (Was: DoS with mod_deflate range requests)
On Wed, Aug 24, 2011 at 7:57 AM, Plüm, Rüdiger, VF-Group
ruediger.pl...@vodafone.com wrote:
-Original Message-
From: Dirk-Willem van Gulik
Sent: Mittwoch, 24. August 2011 13:33
To: dev@httpd.apache.org
Subject: Mitigation Range header (Was: DoS with mod_deflate
range requests)
Folks,
This issue is now active in the wild. So some unified/simple
comms is needed.
What is the wisdom on mitigation advise/briefing until a
proper fix it out - in order of ease:
- Where possible - disable mod_deflate
= we sure this covers all cases - or this is a good stopgap ?
As said this has *nothing* to do with mod_deflate. This was IMHO just
a guess by the original author of the tool.
- Where possible - set LimitRequestFieldSize to a small value
- Suggesting of 128 fine ?
- Where this is not possible (e.g. long cookies, auth
headers of serious size) consider using
mod_rewrite to not accept more than a few commas
= anyone a config snipped for this ?
How about the following (untested) rewrite rule. It should only allow 5
ranges at max.
RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
Is [E=no-gzip] enough to avoid the downward spiral, for the sake of
false positives?
But your regex matches when there's just a couple of ranges -- maybe
{4} and no $?
Re: Mitigation Range header (Was: DoS with mod_deflate range requests)
On 24 Aug 2011, at 12:57, Plüm, Rüdiger, VF-Group wrote:
- Where possible - disable mod_deflate
= we sure this covers all cases - or this is a good stopgap ?
As said this has *nothing* to do with mod_deflate. This was IMHO just
a guess by the original author of the tool.
Ok - but when I try it on my servers (with the check of the tool removed) - it
seems quite impotent unless mod_deflate is in the wire.
And it seems a bit more potent when there is other 'keep in the air' modules
around.
So I guess mod_deflate is right now the largest 'plug' we have in the server
which can cause this backup ?
Or is that totally wrong. Happy to stand correctede !
- Where possible - set LimitRequestFieldSize to a small value
- Suggesting of 128 fine ?
- Where this is not possible (e.g. long cookies, auth
headers of serious size) consider using
mod_rewrite to not accept more than a few commas
= anyone a config snipped for this ?
How about the following (untested) rewrite rule. It should only allow 5
ranges at max.
RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
Sounds like a plan ! This mail crossed one I just sent out - lemme update that
too.
Dw.
Re: Mitigation Range header
* Plüm, Rüdiger, VF-Group: As said this has *nothing* to do with mod_deflate. This was IMHO just a guess by the original author of the tool. This matches my testing, too. I see a significant peak in RAM usage on a server where apachectl -M does not print anything with the string deflate (so I assume that mod_deflate is not enabled). This is with 2.2.9-10+lenny9 on Debian. If it is more difficult to check if mod_deflate is enabled, the advisory should tell how to check your server. If the method I used is the correct one, I don't think it's reasonable to suggest disabling mod_deflate as a mitigation because it does not seem to make much of a difference. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
RE: Mitigation Range header (Was: DoS with mod_deflate range requests)
-Original Message-
From: Eric Covener [mailto:cove...@gmail.com]
Sent: Mittwoch, 24. August 2011 14:05
To: dev@httpd.apache.org
Subject: Re: Mitigation Range header (Was: DoS with
mod_deflate range requests)
On Wed, Aug 24, 2011 at 7:57 AM, Plüm, Rüdiger, VF-Group
ruediger.pl...@vodafone.com wrote:
-Original Message-
From: Dirk-Willem van Gulik
Sent: Mittwoch, 24. August 2011 13:33
To: dev@httpd.apache.org
Subject: Mitigation Range header (Was: DoS with mod_deflate
range requests)
Folks,
This issue is now active in the wild. So some unified/simple
comms is needed.
What is the wisdom on mitigation advise/briefing until a
proper fix it out - in order of ease:
- Where possible - disable mod_deflate
= we sure this covers all cases - or this is a good
stopgap ?
As said this has *nothing* to do with mod_deflate. This was
IMHO just
a guess by the original author of the tool.
- Where possible - set LimitRequestFieldSize to a small value
- Suggesting of 128 fine ?
- Where this is not possible (e.g. long cookies, auth
headers of serious size) consider using
mod_rewrite to not accept more than a few commas
= anyone a config snipped for this ?
How about the following (untested) rewrite rule. It should
only allow 5
ranges at max.
RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
Is [E=no-gzip] enough to avoid the downward spiral, for the sake of
false positives?
As said it has IMHO nothing to do with mod_deflate. It is an issue of the byte
range filter.
But your regex matches when there's just a couple of ranges -- maybe
{4} and no $?
Of course it should have been:
RewriteCond %{HTTP:range} !^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
As said untested. Thanks for remote eyes.
Regards
Rüdiger
Re: Mitigation Range header
On 24 Aug 2011, at 13:22, Florian Weimer wrote: * Plüm, Rüdiger, VF-Group: As said this has *nothing* to do with mod_deflate. This was IMHO just a guess by the original author of the tool. This matches my testing, too. I see a significant peak in RAM usage on a server where apachectl -M does not print anything with the string deflate (so I assume that mod_deflate is not enabled). This is with 2.2.9-10+lenny9 on Debian. If it is more difficult to check if mod_deflate is enabled, the advisory should tell how to check your server. If the method I used is the correct one, I don't think it's reasonable to suggest disabling mod_deflate as a mitigation because it does not seem to make much of a difference. Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Dw. *: as I cannot reproduce the issue with very small index.html files.
Re: Mitigation Range header
* Dirk-WIllem van Gulik: Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Oh. Have you checked memory usage on the server? -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
RE: Mitigation Range header (Was: DoS with mod_deflate range requests)
-Original Message-
From: Dirk-Willem van Gulik [mailto:di...@webweaving.org]
Sent: Mittwoch, 24. August 2011 14:14
To: dev@httpd.apache.org
Subject: Re: Mitigation Range header (Was: DoS with
mod_deflate range requests)
On 24 Aug 2011, at 12:57, Plüm, Rüdiger, VF-Group wrote:
- Where possible - disable mod_deflate
= we sure this covers all cases - or this is a good stopgap ?
As said this has *nothing* to do with mod_deflate. This was
IMHO just
a guess by the original author of the tool.
Ok - but when I try it on my servers (with the check of the
tool removed) - it seems quite impotent unless mod_deflate
is in the wire.
Hm, weird. I would guess that mod_deflate could even mitigate this attack as
the byterange
filter only does something if it sees the whole response in the brigade the
first time it
is called. Having mod_deflate compressing larger chunks of data causes the
byterange filter
to be called multiple times with only parts of the response in the brigade.
So the byte range filter should only be applied with responses whose compressed
response
fits into the zlibs output filter.
Depending on the size of the input and the number of parallel requests it might
be possible
that a lot of memory is consumed by mod_deflate anyway. But I would expect the
same behviour
without range requests as well.
And it seems a bit more potent when there is other 'keep in
the air' modules around.
So I guess mod_deflate is right now the largest 'plug' we
have in the server which can cause this backup ?
Or is that totally wrong. Happy to stand correctede !
- Where possible - set LimitRequestFieldSize to a small value
- Suggesting of 128 fine ?
- Where this is not possible (e.g. long cookies, auth
headers of serious size) consider using
mod_rewrite to not accept more than a few commas
= anyone a config snipped for this ?
How about the following (untested) rewrite rule. It should
only allow 5
ranges at max.
RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
Sounds like a plan ! This mail crossed one I just sent out -
lemme update that too.
Please see my response to Eric. He detected an error in the above.
Regards
Rüdiger
Re: Mitigation Range header (Was: DoS with mod_deflate range requests)
Of course it should have been:
RewriteCond %{HTTP:range} !^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
The problem with the negation is you need an addl rule to handle an
empty range header, this would forbid normal non-range requests.
--
Eric Covener
cove...@gmail.com
Re: Mitigation Range header
On 24 Aug 2011, at 13:43, Florian Weimer wrote: * Dirk-WIllem van Gulik: Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Oh. Have you checked memory usage on the server? I had not - and you are right - quite high. I also tried it on a Ubuntu machine - and that one dies right out of the gate - regardless as to wether deflate is on- or off. So I guess this is somewhat OS specific - but indeed - not overly deflate specific. Deflate just does something. Ok - let me rewrite advisory draft ! Dw
RE: Mitigation Range header
-Original Message- From: Dirk-WIllem van Gulik [mailto:di...@webweaving.org] Sent: Mittwoch, 24. August 2011 14:40 To: dev@httpd.apache.org Cc: Plüm, Rüdiger, VF-Group Subject: Re: Mitigation Range header On 24 Aug 2011, at 13:22, Florian Weimer wrote: * Plüm, Rüdiger, VF-Group: As said this has *nothing* to do with mod_deflate. This was IMHO just a guess by the original author of the tool. This matches my testing, too. I see a significant peak in RAM usage on a server where apachectl -M does not print anything with the string deflate (so I assume that mod_deflate is not enabled). This is with 2.2.9-10+lenny9 on Debian. If it is more difficult to check if mod_deflate is enabled, the advisory should tell how to check your server. If the method I used is the correct one, I don't think it's reasonable to suggest disabling mod_deflate as a mitigation because it does not seem to make much of a difference. Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Dw. *: as I cannot reproduce the issue with very small index.html files. Have you tried if the same happens with mod_deflate, but with one of the the proposed mitigations in place? As said my guess is that this might be an issue with mod_deflate that is unrelated to the Range request issue. Regards Rüdiger
Re: Mitigation Range header
* Dirk-WIllem van Gulik: On 24 Aug 2011, at 13:43, Florian Weimer wrote: * Dirk-WIllem van Gulik: Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Oh. Have you checked memory usage on the server? I had not - and you are right - quite high. I also tried it on a Ubuntu machine - and that one dies right out of the gate - regardless as to wether deflate is on- or off. It seems that this reflects different approaches to memory overcommitment. I didn't see any crashes with vm.overcommit_memory=2 on Linux, either. I wouldn't mention this in the advisory, though, because even if no critical process is terminated due to the out-of-memory condition, thrashing could still push the system beyond the point of recovery. Including the increased memory usage in the adviosry, as a potential attack indicator, would make sense, IMHO. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
Re: Mitigation Range header
On 24 Aug 2011, at 14:01, Plüm, Rüdiger, VF-Group wrote: Have you tried if the same happens with mod_deflate, but with one of the the proposed mitigations in place? As soon as I reject/block on the range header - all is fine again. As said my guess is that this might be an issue with mod_deflate that is unrelated to the Range request issue. Good point. Dw
Re: Mitigation Range header
On Wed, Aug 24, 2011 at 9:01 AM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Dw. *: as I cannot reproduce the issue with very small index.html files. Have you tried if the same happens with mod_deflate, but with one of the the proposed mitigations in place? As said my guess is that this might be an issue with mod_deflate that is unrelated to the Range request issue. I think mod_deflate is just the tool to convert an O(N^2) data size problem into an O(N^2) CPU usage problem, where N is some function of LimitRequestLine. If the file size is smaller than the largest range end used in the attack, it may reduce the amount of data actually going down the filter chain. Greg
RE: Mitigation Range header (Was: DoS with mod_deflate range requests)
-Original Message-
From: Eric Covener [mailto:cove...@gmail.com]
Sent: Mittwoch, 24. August 2011 14:59
To: dev@httpd.apache.org
Subject: Re: Mitigation Range header (Was: DoS with
mod_deflate range requests)
Of course it should have been:
RewriteCond %{HTTP:range} !^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* - [F]
The problem with the negation is you need an addl rule to handle an
empty range header, this would forbid normal non-range requests.
Damn it. Got me again. How about this:
RewriteCond %{HTTP:range} !(^bytes=[^,]+(,[^,]+){0,4}$|^$)
RewriteRule .* - [F]
Regards
Rüdiger
RE: Mitigation Range header
From: Greg Ames Sent: Mittwoch, 24. August 2011 16:05 To: dev@httpd.apache.org Subject: Re: Mitigation Range header On Wed, Aug 24, 2011 at 9:01 AM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: Hmm - when I remove mod_deflate (i.e. explicitly as it is the default in all our installs) and test on a / entry which is a static file which is large (100k)* - then I cannot get apache on its knees on a freebsd machine - saturating the 1Gbit connection it has (Note: the attack machines *are* getting saturated). The moment i put in mod_deflate, mod_external filter, etc - it is much easier to get deplete enough resources to notice. Dw. *: as I cannot reproduce the issue with very small index.html files. Have you tried if the same happens with mod_deflate, but with one of the the proposed mitigations in place? As said my guess is that this might be an issue with mod_deflate that is unrelated to the Range request issue. I think mod_deflate is just the tool to convert an O(N^2) data size problem into an O(N^2) CPU usage problem, where N is some function of LimitRequestLine. If the file size is smaller than the largest range end used in the attack, it may reduce the amount of data actually going down the filter chain. Greg I don't think so. The compression happens before the byterange filter and the byterange filter just hacks the already compressed brigade into more buckets and rearranges them. mod_deflate does not do more work if it is a range request. It does the same amount of work as for the non range request. Regards Rüdiger
Re: Mitigation Range header
On Wed, Aug 24, 2011 at 10:33 AM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: ** I think mod_deflate is just the tool to convert an O(N^2) data size problem into an O(N^2) CPU usage problem, where N is some function of LimitRequestLine. If the file size is smaller than the largest range end used in the attack, it may reduce the amount of data actually going down the filter chain. Greg I don't think so. The compression happens before the byterange filter and the byterange filter just hacks the already compressed brigade into more buckets and rearranges them. mod_deflate does not do more work if it is a range request. It does the same amount of work as for the non range request. OK, thanks for the clarification, Rüdiger. Then I don't understand why mod_deflate seems to be an important factor in killing the server. If the DEFLATE filter runs first, can you do anything useful with a subrange of its output? i.e., could a client decompress a subrange that starts in the middle of the compressed version and get a subrange of the original uncompressed data? Greg
RE: Mitigation Range header
From: Greg Ames [mailto:ames.g...@gmail.com] Sent: Mittwoch, 24. August 2011 18:20 To: dev@httpd.apache.org Subject: Re: Mitigation Range header On Wed, Aug 24, 2011 at 10:33 AM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: I think mod_deflate is just the tool to convert an O(N^2) data size problem into an O(N^2) CPU usage problem, where N is some function of LimitRequestLine. If the file size is smaller than the largest range end used in the attack, it may reduce the amount of data actually going down the filter chain. Greg I don't think so. The compression happens before the byterange filter and the byterange filter just hacks the already compressed brigade into more buckets and rearranges them. mod_deflate does not do more work if it is a range request. It does the same amount of work as for the non range request. OK, thanks for the clarification, Rüdiger. Then I don't understand why mod_deflate seems to be an important factor in killing the server. If the DEFLATE filter runs first, can you do anything useful with a subrange of its output? i.e., could a client decompress a subrange that starts in the middle of the compressed version and get a subrange of the original uncompressed data? Greg It depends. Think of a download that was broken in the middle and that was compressed by mod_deflate on the fly. A client could just request the missing bits and decompress the whole thing afterwards. I guess in general it only works if the client has all the missing pieces (or at least the ones before the contents of the partial response) that are not sent via the partial response at hand to decompress the whole stream (or decompress at least until the last part of the partial response). Regards Rüdiger
