Re: Exceptions due to attacks

[Ferran Jorba]

Hi Samuele,

[...]
> wait :-) This is already implemented, as Theodoros reported ;-) Just
> have a look in maint-1.1 to:
>
> commit 0aeb9fa7e8a6b6809be5d586bcdcf0e7a9784e05
> Author: Samuele Kaplun 
> Date:   Tue Oct 27 16:48:22 2009 +0100
>
> WebStyle: configurable alerts for HTTP errors
[...]
> Isn’t this commit available in your repo and already doing what you are 
> looking for?

I have it in my 1.1.0 production system!  I've just modified the value
(removed 404r), executed inveniocfg --update-config-py and no more
mailbombs!

Thanks to all,

Ferran

Re: Exceptions due to attacks

[Samuele Kaplun]

Hi All,

In data giovedì 25 aprile 2013 11:59:53, Ferran Jorba ha scritto:
> > On 25/4/2013 12:37 μμ, Ferran Jorba wrote:
> >> but try an index.php or any other missing hit at http://cds.cern.ch.  It
> >> is effectively handled by Invenio.
> > 
> > My point exactly. I see that both my installations and CERN's
> > correctly handle those 'attacks'. I even tried with .php and .py files
> > and there is no exception raised and sent to the admin even if you set
> > CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 2 in invenio(-local).conf
> 
> To tell you the truth, I wasn't aware of this variable.  I have the
> default value:
> 
>  ## CFG_SITE_ADMIN_EMAIL_EXCEPTIONS -- set this to 0 if you do not want
>  ## to receive any captured exception via email to CFG_SITE_ADMIN_EMAIL
>  ## address.  Captured exceptions will still be available in
>  ## var/log/invenio.err file.  Set this to 1 if you want to receive
>  ## some of the captured exceptions (this depends on the actual place
>  ## where the exception is captured).  Set this to 2 if you want to
>  ## receive all captured exceptions.
>  CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 1
> 
> > Unless I'm missing something here, I suspect something weird happening
> > only with your installation...
> 
> Watching more carefully (thanks!) my installation, I see that it returns
> a 404 Not found (http://traces.uab.cat/abc), even in the HTTP headers.
> 
> So, now, with a better understanding of what is happening, what I'd like
> is a value for CFG_SITE_ADMIN_EMAIL_EXCEPTIONS that doesn't send me an
> email for a 404 status.
> 
> I'll try to do a local fix and provide it upstream if there is interest.

wait :-) This is already implemented, as Theodoros reported ;-) Just have a 
look in maint-1.1 to:

commit 0aeb9fa7e8a6b6809be5d586bcdcf0e7a9784e05
Author: Samuele Kaplun 
Date:   Tue Oct 27 16:48:22 2009 +0100

WebStyle: configurable alerts for HTTP errors

* When a 404 error is raised, the admin may want to be alerted
  only when there is a known referer for the given URL, so that
  the admin can distinguish between a broken link to the site
  and a user making a typo in the URL location bar.  This patch
  makes this distinction possible by providing a new config variable
  CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST.

This is the implementation to catch any bad request (so not only the 
/record/123/files/* one).

Isn’t this commit available in your repo and already doing what you are 
looking for?

Cheers!
Sam

-- 
Samuele Kaplun
Invenio Developer ** 
INSPIRE Service Manager ** 

Re: Exceptions due to attacks

[Ferran Jorba]

Hello Theodoros,
 
> On 25/4/2013 12:37 μμ, Ferran Jorba wrote:
>> but try an index.php or any other missing hit at http://cds.cern.ch.  It
>> is effectively handled by Invenio.
> My point exactly. I see that both my installations and CERN's
> correctly handle those 'attacks'. I even tried with .php and .py files
> and there is no exception raised and sent to the admin even if you set
> CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 2 in invenio(-local).conf

To tell you the truth, I wasn't aware of this variable.  I have the
default value:

 ## CFG_SITE_ADMIN_EMAIL_EXCEPTIONS -- set this to 0 if you do not want
 ## to receive any captured exception via email to CFG_SITE_ADMIN_EMAIL
 ## address.  Captured exceptions will still be available in
 ## var/log/invenio.err file.  Set this to 1 if you want to receive
 ## some of the captured exceptions (this depends on the actual place
 ## where the exception is captured).  Set this to 2 if you want to
 ## receive all captured exceptions.
 CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 1

> Unless I'm missing something here, I suspect something weird happening
> only with your installation...

Watching more carefully (thanks!) my installation, I see that it returns
a 404 Not found (http://traces.uab.cat/abc), even in the HTTP headers.

So, now, with a better understanding of what is happening, what I'd like
is a value for CFG_SITE_ADMIN_EMAIL_EXCEPTIONS that doesn't send me an
email for a 404 status.

I'll try to do a local fix and provide it upstream if there is interest.

Thanks for your feedback,

Ferran

Re: Exceptions due to attacks

[Theodoros Theodoropoulos]

On 25/4/2013 12:37 μμ, Ferran Jorba wrote:

but try an index.php or any other missing hit at http://cds.cern.ch.  It
is effectively handled by Invenio.
My point exactly. I see that both my installations and CERN's correctly 
handle those 'attacks'. I even tried with .php and .py files and there 
is no exception raised and sent to the admin even if you set 
CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 2 in invenio(-local).conf


Unless I'm missing something here, I suspect something weird happening 
only with your installation...


Theodoros

Re: Exceptions due to attacks

[Ferran Jorba]

Hi Theodoros,
 
> Hello Ferran,
>
> My dev 1.0.1.1218 and latest maint-1.1 sites correctly display a 404
> not found page for either
> /record/xxx/files/wp-whatever
> /record/xxx/wp-whatever
> /record/wp-whatever
>
> without sending me an exception error
> The same applies if wp-whatever is replaced by "../../etc/passwd" and
> the likes.
>
> I tried the same with cds.lib.auth.gr and it also displays a 404 error
> (i don't know if an error is logged)

but try an index.php or any other missing hit at http://cds.cern.ch.  It
is effectively handled by Invenio.

>> So, I understand that we need a general solution to provide an (a)
>> 404 not found to the attacker, and/or (b) a digested summary to the
>> admin.
>>
>> Aren't the other sites having this flood of attacks?  I doubt we are
>> the only ones.

Ferran

Re: Exceptions due to attacks

[Theodoros Theodoropoulos]

On 25/4/2013 12:29 μμ, Theodoros Theodoropoulos wrote:

I tried the same with cds.lib.auth.gr and it also displays a 404 error
(i don't know if an error is logged)

Correction: I meant cds.cern.ch :)

Re: Exceptions due to attacks

[Theodoros Theodoropoulos]

Hello Ferran,

My dev 1.0.1.1218 and latest maint-1.1 sites correctly display a 404 not 
found page for either

/record/xxx/files/wp-whatever
/record/xxx/wp-whatever
/record/wp-whatever

without sending me an exception error
The same applies if wp-whatever is replaced by "../../etc/passwd" and 
the likes.


I tried the same with cds.lib.auth.gr and it also displays a 404 error 
(i don't know if an error is logged)


Hmmm...

Best regards,
Theodoros

On 25/4/2013 12:13 μμ, Ferran Jorba wrote:

Hello Samuele,


Is there any progress on this issue?  Under 1.1 the missing pages
produce much more noise than the old mod_python.

sorry to come back to this issue only now. Indeed a fix for this has been
provided for maint-1.0 in:

commit 22f4e36755d7103e420da10968f60430ed797c26
Author: Samuele Kaplun 
Date:   Fri Dec 7 15:06:46 2012 +0100

 bibdocfile: better error report for unknown format

I've taken a look and it doesn't seem to me that it provides a fix for a
/index.php, /phpmyadmin.php, ../../../etc/passwd or /wp-whatever hits,
exceptions and subsequent mails that I'm constantly getting from our
Traces sistem since we are at 1.1.  Under 0.99 old mod_python Invenio,
Apache handled those not-found, but now they are caught by wsgi Invenio,
causing those mail floods on my inbox.


commit 6d6e985c9abcf02bd85f9eb442e116547eb1f531
Merge: 35fae49 22f4e36
Author: Tibor Simko 
Date:   Thu Dec 20 10:53:36 2012 +0100

 Merge branch 'maint-1.0' into maint-1.1
 
 * maint-1.0:

   bibdocfile: better error report for unknown format

I think you should be able to safely update to latest maint-1.1 in order to
benefit from this fix.

Again, after reading the patch I see it as if it only hanles
/record/x/file/whatever attacks, but not the others.  Maybe I'm wrong.

So, I understand that we need a general solution to provide an (a) 404
not found to the attacker, and/or (b) a digested summary to the admin.

Aren't the other sites having this flood of attacks?  I doubt we are the
only ones.

Thanks,

Ferran

Re: Exceptions due to attacks

[Ferran Jorba]

Hello Samuele,

>> Is there any progress on this issue?  Under 1.1 the missing pages
>> produce much more noise than the old mod_python.
>
> sorry to come back to this issue only now. Indeed a fix for this has been 
> provided for maint-1.0 in:
>
> commit 22f4e36755d7103e420da10968f60430ed797c26
> Author: Samuele Kaplun 
> Date:   Fri Dec 7 15:06:46 2012 +0100
>
> bibdocfile: better error report for unknown format

I've taken a look and it doesn't seem to me that it provides a fix for a
/index.php, /phpmyadmin.php, ../../../etc/passwd or /wp-whatever hits,
exceptions and subsequent mails that I'm constantly getting from our
Traces sistem since we are at 1.1.  Under 0.99 old mod_python Invenio,
Apache handled those not-found, but now they are caught by wsgi Invenio,
causing those mail floods on my inbox.

> commit 6d6e985c9abcf02bd85f9eb442e116547eb1f531
> Merge: 35fae49 22f4e36
> Author: Tibor Simko 
> Date:   Thu Dec 20 10:53:36 2012 +0100
>
> Merge branch 'maint-1.0' into maint-1.1
> 
> * maint-1.0:
>   bibdocfile: better error report for unknown format
>
> I think you should be able to safely update to latest maint-1.1 in order to 
> benefit from this fix.

Again, after reading the patch I see it as if it only hanles
/record/x/file/whatever attacks, but not the others.  Maybe I'm wrong.

So, I understand that we need a general solution to provide an (a) 404
not found to the attacker, and/or (b) a digested summary to the admin.

Aren't the other sites having this flood of attacks?  I doubt we are the
only ones.

Thanks,

Ferran

Re: Exceptions due to attacks

[Samuele Kaplun]

Hi!

In data mercoledì 24 aprile 2013 16:46:48, Ferran Jorba ha scritto:
> >> In data mercoledì 20 marzo 2013 08:19:27, Johnny Mariéthoz ha scritto:
> >>> every day I have some exceptions due to attacks such as: IOError:
> >>> request data read error (webinterface_handler_wsgi.py:377:readline)
> >>> an example of request is:
> >>> /record/17041/files/wp-content/plugins/mm-forms-community/includes/doaja
> >>> xfil eupload.php
> >>> 
> >>> Is it possible to return a 404 status for such as request?
> >> 
> >> which version of Invenio are you running? Depending on it this is
> >> indeed the default configuration. I will check the commit log, and
> >> point you out the missing patches...
> 
> Is there any progress on this issue?  Under 1.1 the missing pages
> produce much more noise than the old mod_python.

sorry to come back to this issue only now. Indeed a fix for this has been 
provided for maint-1.0 in:

commit 22f4e36755d7103e420da10968f60430ed797c26
Author: Samuele Kaplun 
Date:   Fri Dec 7 15:06:46 2012 +0100

bibdocfile: better error report for unknown format

* When an unknown format for a bibdocfile is requested an exception
  was registered (and an email was sent to the admins), even when
  no referer was specified in the HTTP headers, thus cluttering
  error reports in case of malicious users playing with the system.
  This commit improves the check for unknown formats earlier avoiding
  the exception to be registered, when no referer is set.

and subsequently merged into maint-1.1 in:

commit 6d6e985c9abcf02bd85f9eb442e116547eb1f531
Merge: 35fae49 22f4e36
Author: Tibor Simko 
Date:   Thu Dec 20 10:53:36 2012 +0100

Merge branch 'maint-1.0' into maint-1.1

* maint-1.0:
  bibdocfile: better error report for unknown format

I think you should be able to safely update to latest maint-1.1 in order to 
benefit from this fix.

Cheers!
Sam

-- 
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>
INSPIRE Service Manager ** <http://inspirehep.net/>

Re: Exceptions due to attacks

[Ferran Jorba]

Hello,

>> In data mercoledì 20 marzo 2013 08:19:27, Johnny Mariéthoz ha scritto:
>>> every day I have some exceptions due to attacks such as: IOError:
>>> request data read error (webinterface_handler_wsgi.py:377:readline)
>>> an example of request is:
>>> /record/17041/files/wp-content/plugins/mm-forms-community/includes/doajaxfil
>>> eupload.php
>>> 
>>> Is it possible to return a 404 status for such as request?
>> 
>> which version of Invenio are you running? Depending on it this is
>> indeed the default configuration. I will check the commit log, and
>> point you out the missing patches...

Is there any progress on this issue?  Under 1.1 the missing pages
produce much more noise than the old mod_python.

Thanks,

Ferran

Re: Exceptions due to attacks

[Johnny Mariéthoz]

Hello Samuele,

sorry I had to mention it.

Git says me:
* maint-1.1 ea6cef9 [behind 9] global: test suite nosification

Regards,
--
Johnny Mariéthoz
RERO, Av. de la Gare 45, CH - 1920 MARTIGNY
Téléphone:  +41(0)27 721 8579
Fax  : +41(0)27 721 8586
Web: http://www.rero.ch
ReroDoc: http://doc.rero.ch, doc.supp...@rero.ch
--


Le 20 mars 2013 à 09:06, Samuele Kaplun a écrit :

> Hi Johnny,
> 
> In data mercoledì 20 marzo 2013 08:19:27, Johnny Mariéthoz ha scritto:
>> every day I have some exceptions due to attacks such as:
>> IOError: request data read error
>> (webinterface_handler_wsgi.py:377:readline) an example of request is:
>> /record/17041/files/wp-content/plugins/mm-forms-community/includes/doajaxfil
>> eupload.php
>> 
>> Is it possible to return a 404 status for such as request?
> 
> which version of Invenio are you running? Depending on it this is indeed the 
> default configuration. I will check the commit log, and point you out the 
> missing patches...
> 
> Cheers!
>   Sam
> 
> 
> 
> -- 
> Samuele Kaplun
> Invenio Developer ** <http://invenio-software.org/>

Re: Exceptions due to attacks

[Samuele Kaplun]

Hi Johnny,

In data mercoledì 20 marzo 2013 08:19:27, Johnny Mariéthoz ha scritto:
> every day I have some exceptions due to attacks such as:
>  IOError: request data read error
> (webinterface_handler_wsgi.py:377:readline) an example of request is:
> /record/17041/files/wp-content/plugins/mm-forms-community/includes/doajaxfil
> eupload.php
> 
> Is it possible to return a 404 status for such as request?

which version of Invenio are you running? Depending on it this is indeed the 
default configuration. I will check the commit log, and point you out the 
missing patches...

Cheers!
Sam



-- 
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>

Exceptions due to attacks

[Johnny Mariéthoz]

Dear all,

every day I have some exceptions due to attacks such as:
 IOError: request data read error (webinterface_handler_wsgi.py:377:readline)
an example of request is:
/record/17041/files/wp-content/plugins/mm-forms-community/includes/doajaxfileupload.php

Is it possible to return a 404 status for such as request?

Many thanks in advance.

Regards,
--
Johnny Mariéthoz
RERO, Av. de la Gare 45, CH - 1920 MARTIGNY
Téléphone:  +41(0)27 721 8579
Fax  : +41(0)27 721 8586
Web: http://www.rero.ch
ReroDoc: http://doc.rero.ch, doc.supp...@rero.ch
--