Re: DoS susceptability via email logging

[Donald Stufft]

On Thursday, December 8, 2011 at 7:33 PM, Russell Keith-Magee wrote:
> On Thu, Dec 8, 2011 at 8:42 PM, Sam Berry  (mailto:samkbe...@googlemail.com)> wrote:
> > Hello there,
> > 
> > I am currently running a number of small sites using logging via email
> > to notify me of 404s and server errors. The volume of email produced
> > is perfectly manageable due to the low level of traffic.
> > 
> 
> 
> First off, I'd point out that the 404 emails aren't sent every time
> your server generates a 404 -- they're only generated as a result of a
> 404 *from an internal link on your site*. If John Q Hacker hits
> http://yoursite.com/this_page_doesnt_exist, you won't get a 404 email.
> 
> 

This isn't exactly accurate. It sends them for any 404 response where the 
HTTP_REFERER
header is not in the ignorable urls setting, is not empty, and does not contain 
a ?

This is also trivial to get around and if someone is attempting to trigger a 
boatload of
404 emails this check is only going to prevent someone who knows nothing about 
Django (in
which case how do they know about 404 email sending?). 
> 
> If you're getting *any* 404 emails, it means there are internal links
> on your site that don't resolve. If this is something that is
> happening systematically, at a rate that has given you cause for
> concern about the potential for DOS, I'd suggest you have a bigger
> problem with your site.
> 
> > It does occur to me that it would be very easy to disrupt the site by
> > simply sending repeated requests which trigger a 404 Not Found. I'm
> > sure 10 404s a second would completely shutdown the process to normal
> > requests due to the time taken to send the log emails. I'm not using a
> > message queueing system.
> > 
> > I would like to keep the option of receiving logging emails. Is there
> > anything I can do to throttle or limit the email sending without
> > writing my own logging handler?
> > 
> 
> 
> Interestingly, 404 emails aren't handled by a logging handler (unlike
> 500 messages) -- they're handled as a direct mail to site managers.
> This is something that is probably worth a ticket in itself; using the
> logging framework for 404 mails would allow for much more flexibility.
> 
> 

Agreed, 404 should be configurable via Logging. 
> 
> So - what you want here isn't a custom logging handler, but a custom
> *mail* handler -- django-mailer [1] is a good candidate here. This
> makes "sending" an email a very short lived operation from the
> perspective of your web server -- all the web request does is queue
> the email for sending. The actual sending of mail is handled out of
> the request-response cycle. This removes the possibility of "DOS
> attack by 404" because the expensive operation is taken out of the
> request-response cycle.
> 
> 

This is True.  You could also move email sending to a celery task
or something similar.
> 
> Of course, you could still end up being flooded with email -- so you
> either need to (a) monitor the size of your mail queue to make sure it
> isn't getting flooded, or (b) put a processor in place to
> merge/throttle the contents of the mail queue. This could also be
> handled with a custom mail handler; however, I can't point you at a
> ready-to-use candidate for this.
> 
> Another option is to use a mechanism other than the 404 emails to log
> and handle errors. Django-sentry [2] and Arecibo [3] are two
> candidates here, both of which are easy to install, and provide much
> richer analytics than a full mailbox :-)
> 
> 

This is, in my opinion, the best option if you wish to track 404's. 
> 
> [1] https://github.com/jtauber/django-mailer/
> [2] https://github.com/dcramer/django-sentry/
> [3] http://areciboapp.com/
> 
> Yours,
> Russ Magee %-)
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com 
> (mailto:django-users@googlegroups.com).
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com 
> (mailto:django-users+unsubscr...@googlegroups.com).
> For more options, visit this group at 
> http://groups.google.com/group/django-users?hl=en.
> 
> 


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: DoS susceptability via email logging

[Russell Keith-Magee]

On Thu, Dec 8, 2011 at 8:42 PM, Sam Berry  wrote:
> Hello there,
>
> I am currently running a number of small sites using logging via email
> to notify me of 404s and server errors. The volume of email produced
> is perfectly manageable due to the low level of traffic.

First off, I'd point out that the 404 emails aren't sent every time
your server generates a 404 -- they're only generated as a result of a
404 *from an internal link on your site*. If John Q Hacker hits
http://yoursite.com/this_page_doesnt_exist, you won't get a 404 email.

If you're getting *any* 404 emails, it means there are internal links
on your site that don't resolve. If this is something that is
happening systematically, at a rate that has given you cause for
concern about the potential for DOS, I'd suggest you have a bigger
problem with your site.

> It does occur to me that it would be very easy to disrupt the site by
> simply sending repeated requests which trigger a 404 Not Found. I'm
> sure 10 404s a second would completely shutdown the process to normal
> requests due to the time taken to send the log emails. I'm not using a
> message queueing system.
>
> I would like to keep the option of receiving logging emails. Is there
> anything I can do to throttle or limit the email sending without
> writing my own logging handler?

Interestingly, 404 emails aren't handled by a logging handler (unlike
500 messages) -- they're handled as a direct mail to site managers.
This is something that is probably worth a ticket in itself; using the
logging framework for 404 mails would allow for much more flexibility.

So - what you want here isn't a custom logging handler, but a custom
*mail* handler -- django-mailer [1] is a good candidate here. This
makes "sending" an email a very short lived operation from the
perspective of your web server -- all the web request does is queue
the email for sending. The actual sending of mail is handled out of
the request-response cycle. This removes the possibility of "DOS
attack by 404" because the expensive operation is taken out of the
request-response cycle.

Of course, you could still end up being flooded with email -- so you
either need to (a) monitor the size of your mail queue to make sure it
isn't getting flooded, or (b) put a processor in place to
merge/throttle the contents of the mail queue. This could also be
handled with a custom mail handler; however, I can't point you at a
ready-to-use candidate for this.

Another option is to use a mechanism other than the 404 emails to log
and handle errors. Django-sentry [2] and Arecibo [3] are two
candidates here, both of which are easy to install, and provide much
richer analytics than a full mailbox :-)

[1] https://github.com/jtauber/django-mailer/
[2] https://github.com/dcramer/django-sentry/
[3] http://areciboapp.com/

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: DoS susceptability via email logging

[creecode]

Hello Sam,

You might want to do something like detect the situation and then at the 
firewall for your router or server, cut off the offending addresses.  No 
specific details to offer.  The advantage to getting a rule onto your 
firewall would be reduced load on the software side of request processing.

Another approach would be to use server monitoring software such as Nagios 
to regularlly check that the server/websites are up and running and give 
you a heads up when there is a problem.  The advantage here is that you can 
then find out what the problem is and take care of it manually.  If there 
is some kind of pattern to the problem then you might be able to automate a 
solution.  Going the monitoring route generally requires one or more other 
computers to be effective.

Toodle-looo
creecode

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/KNm8dSfpUbEJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: DoS susceptability via email logging

[Nikolas Stevenson-Molnar]

I imagine you can use something like Celery ( has been recommended on
this list before: http://celeryproject.org/ ) to queue email tasks and
handle them in a single background process.

_Nik

On 12/8/2011 4:42 AM, Sam Berry wrote:
> Hello there,
>
> I am currently running a number of small sites using logging via email
> to notify me of 404s and server errors. The volume of email produced
> is perfectly manageable due to the low level of traffic.
>
> It does occur to me that it would be very easy to disrupt the site by
> simply sending repeated requests which trigger a 404 Not Found. I'm
> sure 10 404s a second would completely shutdown the process to normal
> requests due to the time taken to send the log emails. I'm not using a
> message queueing system.
>
> I would like to keep the option of receiving logging emails. Is there
> anything I can do to throttle or limit the email sending without
> writing my own logging handler?
>
> Any thoughts would be much appreciated.
>
> Many thanks,
> Sam Berry
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

DoS susceptability via email logging

[Sam Berry]

Hello there,

I am currently running a number of small sites using logging via email
to notify me of 404s and server errors. The volume of email produced
is perfectly manageable due to the low level of traffic.

It does occur to me that it would be very easy to disrupt the site by
simply sending repeated requests which trigger a 404 Not Found. I'm
sure 10 404s a second would completely shutdown the process to normal
requests due to the time taken to send the log emails. I'm not using a
message queueing system.

I would like to keep the option of receiving logging emails. Is there
anything I can do to throttle or limit the email sending without
writing my own logging handler?

Any thoughts would be much appreciated.

Many thanks,
Sam Berry

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.