Re: New plugin: DecodeShortURLs

2011-01-06 Thread Warren Togami Jr.
On Thu, Jan 6, 2011 at 7:23 AM, Henrik K wrote: > > There are lots of plugins out there that aren't part of the core for one > reason or another. If you ask me, this is one of them. It just asks trouble > widely used. It's not the only way to solve the problem anyway. And the > problem itself is

Re: New plugin: DecodeShortURLs

2011-01-06 Thread Henrik K
On Thu, Jan 06, 2011 at 07:05:05AM -1000, Warren Togami Jr. wrote: > On Wed, Jan 5, 2011 at 2:41 AM, Warren Togami Jr. wrote: > > > The only trouble here is HTTP's TCP handshake and teardown is significantly > > slower than DNSBL and URIBL lookups already used in spamassassin. My > > average sca

Re: New plugin: DecodeShortURLs

2011-01-06 Thread Warren Togami Jr.
On Wed, Jan 5, 2011 at 2:41 AM, Warren Togami Jr. wrote: > The only trouble here is HTTP's TCP handshake and teardown is significantly > slower than DNSBL and URIBL lookups already used in spamassassin. My > average scan time is less than one second. A plugin that catches the 1% of > URL shorte

Re: New plugin: DecodeShortURLs

2011-01-05 Thread Warren Togami Jr.
On Sat, Jan 1, 2011 at 7:19 AM, Steve Freegard wrote: > On 01/01/11 11:51, Warren Togami Jr. wrote: > > I'll help you start the process with a Bugzilla ticket. I also hope you > could get it into some sort of public source control mechanism soon so we > can see the changes that go into it befor

Re: New plugin: DecodeShortURLs

2011-01-02 Thread Raymond Dijkxhoorn
Warren, It appears that under 1% of spam is abusing shortening redirectors.  ~40% of the shortening redirector spam has local-only spamassassin scores below the 5 point threshold.  We'll see next Saturday how it scores with all network rules. Could you please quote the old messages and not p

Re: New plugin: DecodeShortURLs

2011-01-02 Thread Warren Togami Jr.
http://ruleqa.spamassassin.org/20110102-r1054364-n/T_URL_SHORTENER/detail I inserted a giant uri regex into the nightly masscheck in order to get a rough measure the true extent of the URL shortener problem. It appears that under 1% of spam is abusing shortening redirectors. ~40% of the shortenin

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Jason Haar
On 01/02/2011 07:52 AM, Michael Scheidell wrote: >> Currently the default used by the LWP module. Could easily set it to >> use an identical string to Firefox or IE. > > and, on occasion, our IPS will tarpit, or delay, or totally block > anything that hits the web servers more than a couple of tim

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Warren Togami Jr.
On Sat, Jan 1, 2011 at 7:19 AM, Steve Freegard wrote: > 7) How fast are typical URL shortening responses? What is the timeout? We > want to avoid degrading the scan time and delivery performance of > spamassassin, but in a way that cannot be abused by the spammer to evade > detection. > > > This

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Michael Scheidell
On 1/1/11 12:19 PM, Steve Freegard wrote: 8) What UserAgent is used in the HTTP request? If they can easily detect that the request is not a real browser, then they can avoid detection by using a safe looking fake response, while browser-based redirects go to the intended spam target. Curren

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Steve Freegard
On 01/01/11 11:51, Warren Togami Jr. wrote: I'll help you start the process with a Bugzilla ticket. I also hope you could get it into some sort of public source control mechanism soon so we can see the changes that go into it before inclusion in upstream. I feel uncomfortable using something

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Steve Freegard
On 01/01/11 12:02, Warren Togami Jr. wrote: http://www.surbl.org/faqs#redirect BTW, this page mentions SpamCopURI and urirhdbl as existing tools that handle redirection to some degree. Have you confirmed that you are not needlessly reinventing the wheel? It is entirely possible that your de

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Warren Togami Jr.
http://www.surbl.org/faqs#redirect BTW, this page mentions SpamCopURI and urirhdbl as existing tools that handle redirection to some degree. Have you confirmed that you are not needlessly reinventing the wheel? It is entirely possible that your design with suggestions here could be better than th

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Warren Togami Jr.
On Fri, Dec 31, 2010 at 11:46 PM, Steve Freegard wrote: > > I notice that there is no Bugzilla ticket for this plugin. Do you intend >> on submitting it for inclusion in future spamassassin upstream? >> >> > > I hadn't really thought about it TBH and wasn't sure what the procedure was > for this

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Steve Freegard
Hi Warren, On 01/01/11 09:17, Warren Togami Jr. wrote: What is the status of this plugin? As far as I'm concerned - I'm actively maintaining it and have been using it in production on several sites; I've been planning to push out an update as I've recently been contributed a massive list o

Re: New plugin: DecodeShortURLs

2011-01-01 Thread Warren Togami Jr.
What is the status of this plugin? I notice that there is no Bugzilla ticket for this plugin. Do you intend on submitting it for inclusion in future spamassassin upstream? Would a DoS happen if the scanned e-mail contains 10,000 short URL's, and your mail server is hit by many such mail? (Eithe

Re: New plugin: DecodeShortURLs

2010-10-15 Thread Alex
Hi, > Recently I've been getting a bit of filter-bleed from a bunch of spams > injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo > that upon closer inspection would have been rejected with a high score > if the real URL had been used. Can this be made to work with v3.2.5, or

Re: New plugin: DecodeShortURLs

2010-10-05 Thread Brent Gardner
René Berber wrote: On 10/5/2010 3:42 PM, Yet Another Ninja wrote: On 2010-10-05 22:35, Brent Gardner wrote: [snip] Using URLs like these: http://goo.gl/foo http://bit.ly/foo http://2chap.it/foo I consistently hit on these rules: HAS_SHORT_URL SHORT_URL_404 SHORT_URL_CHAINED SHO

Re: New plugin: DecodeShortURLs

2010-10-05 Thread René Berber
On 10/5/2010 3:42 PM, Yet Another Ninja wrote: > On 2010-10-05 22:35, Brent Gardner wrote: [snip] >> Using URLs like these: >> >> http://goo.gl/foo >> http://bit.ly/foo >> http://2chap.it/foo >> >> I consistently hit on these rules: >> >> HAS_SHORT_URL >> SHORT_URL_404 >> SHORT_URL_CHAINED >> SHO

Re: New plugin: DecodeShortURLs

2010-10-05 Thread Yet Another Ninja
On 2010-10-05 22:35, Brent Gardner wrote: Steve Freegard wrote: Hi All, On 17/09/10 14:11, Steve Freegard wrote: Hi All, Recently I've been getting a bit of filter-bleed from a bunch of spams injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo that upon closer inspection wo

Re: New plugin: DecodeShortURLs

2010-10-05 Thread Brent Gardner
Steve Freegard wrote: Hi All, On 17/09/10 14:11, Steve Freegard wrote: Hi All, Recently I've been getting a bit of filter-bleed from a bunch of spams injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo that upon closer inspection would have been rejected with a high score if

Re: New plugin: DecodeShortURLs

2010-10-05 Thread Jason Bertoch
On 2010/10/04 6:35 PM, Martin Gregorie wrote: Just a data point for you. I'm running DecodeShortURLs with the as-issued .cf file (log,cache,syslog options commented out). I initially tried running the plugin with these options commented out, but it just doesn't work. It needs those defined.

Re: New plugin: DecodeShortURLs

2010-10-05 Thread John Horne
On Mon, 2010-10-04 at 22:55 +0100, John Horne wrote: > > I grabbed a copy of the above plugin and tried it this afternoon (on a > CentOS 5.5 system). We log all our spamd messages to /var/log/maillog > via syslog. For the plugin I disabled all the options except > 'url_shortener_syslog' which was s

Re: New plugin: DecodeShortURLs

2010-10-05 Thread David Touzeau
Many thanks ADDED in Artica web Open Source Interface !! http://www.artica.fr/index.php/menudocmessaging/39-manage-filters-anti-spam-content-filters/391--shorturls-spam-checking-plugin-with-spamassassin On 17/09/2010 15:11, Steve Freegard wrote: Hi All, Recently I've been getting a bit of f

Re: New plugin: DecodeShortURLs

2010-10-04 Thread Martin Gregorie
On Mon, 2010-10-04 at 22:55 +0100, John Horne wrote: > I grabbed a copy of the [DecodeShortURLs] plugin and tried it this afternoon > (on a > CentOS 5.5 system). We log all our spamd messages to /var/log/maillog > via syslog. For the plugin I disabled all the options except > 'url_shortener_syslog

Re: New plugin: DecodeShortURLs

2010-10-04 Thread John Horne
On Thu, 2010-09-23 at 11:30 +0100, Steve Freegard wrote: > > > > Hopefully it will be useful to others; you can grab it from: > > > > http://www.fsl.com/support/DecodeShortURLs.pm > > http://www.fsl.com/support/DecodeShortURLs.cf > > > ... > > - Added option to allow logging to syslog (mail.info)

Re: New plugin: DecodeShortURLs

2010-09-23 Thread Steve Freegard
Hi All, On 17/09/10 14:11, Steve Freegard wrote: Hi All, Recently I've been getting a bit of filter-bleed from a bunch of spams injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo that upon closer inspection would have been rejected with a high score if the real URL had been

Re: New plugin: DecodeShortURLs

2010-09-22 Thread Steve Freegard
On 22/09/10 13:44, Michael Scheidell wrote: one more: if # url_shortener_cache /tmp/DecodeShortURLs.sq3 you should not try to load SQLLite.pm. ent host [79.98.90.156] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=79.98.90.156; from= to= proto=ESMTP helo= Sep 22 08:38:40 sns

Re: New plugin: DecodeShortURLs

2010-09-22 Thread Michael Scheidell
On 9/20/10 11:33 AM, Steve Freegard wrote: On 20/09/10 15:28, Bowie Bailey wrote: You can get rid of the 'backslashitis' by using a different delimiter. uri URI_BITLY_BLOCKED m~^http://bit\.ly/a/warning~i You still need to escape the period, but since the tilde (~) is now the delimiter rat

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Steve Freegard
On 20/09/10 15:28, Bowie Bailey wrote: You can get rid of the 'backslashitis' by using a different delimiter. uri URI_BITLY_BLOCKED m~^http://bit\.ly/a/warning~i You still need to escape the period, but since the tilde (~) is now the delimiter rather than the slash, you don't need to escape

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Steve Freegard
On 20/09/10 16:17, Michael Scheidell wrote: On 9/20/10 8:15 AM, Steve Freegard wrote: Caching; if desired it will now cache URLs to a SQLite database for additional speed-up and to prevent DoS of the shortener services. any anticipated write lock problems with this due to sqlite not handling

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Michael Scheidell
On 9/20/10 8:15 AM, Steve Freegard wrote: Caching; if desired it will now cache URLs to a SQLite database for additional speed-up and to prevent DoS of the shortener services. any anticipated write lock problems with this due to sqlite not handling multi-threaded reads/writes? most (many?) SA i

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Bowie Bailey
On 9/20/2010 8:15 AM, Steve Freegard wrote: > On 17/09/10 14:48, RW wrote: >> >> I think it might be better to take the "blocked page" handling out of >> the perl and turn it into an ordinary uri rule. >> > > Yeah; really don't know why I did it like that in the first place. > > I've just uploaded

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Steve Freegard
On 17/09/10 14:48, RW wrote: I think it might be better to take the "blocked page" handling out of the perl and turn it into an ordinary uri rule. Yeah; really don't know why I did it like that in the first place. I've just uploaded version 0.2 which does it this way instead and adds the fo

Re: New plugin: DecodeShortURLs

2010-09-20 Thread Chip M.
Steve Freegard wrote: >Hopefully it will be useful to others; you can grab it from: Thanks Steve! Suggestions (for future enhancements): 1. Consider splitting the list of shorteners between those that are well established and KNOWN to be reasonably diligent, and "all others" (e.g. the anti-patte

Re: New plugin: DecodeShortURLs

2010-09-17 Thread RW
On Fri, 17 Sep 2010 14:11:41 +0100 Steve Freegard wrote: > Hi All, > > Recently I've been getting a bit of filter-bleed from a bunch of > spams injected via Hotmail/Yahoo that contain shortened URLs e.g. > bit.ly/foo that upon closer inspection would have been rejected with > a high score if the

Re: New plugin: DecodeShortURLs

2010-09-17 Thread Steve Freegard
On 17/09/10 14:33, Jari Fredriksson wrote: It has a typo. describe URIBL_SHORT... The rule name is wrong, should be SHORT_URIBL Didn't you --lint it? ;) Doh! - fixed. Regards, Steve.

Re: New plugin: DecodeShortURLs

2010-09-17 Thread Jari Fredriksson
On 17.9.2010 16:11, Steve Freegard wrote: > Hi All, > > Recently I've been getting a bit of filter-bleed from a bunch of spams > injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo > that upon closer inspection would have been rejected with a high score > if the real URL had bee

Re: New plugin: DecodeShortURLs

2010-09-17 Thread Eduardo Casarero
2010/9/17 Steve Freegard > Hi All, > > Recently I've been getting a bit of filter-bleed from a bunch of spams > injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foothat > upon closer inspection would have been rejected with a high score if > the real URL had been used. > > To t

New plugin: DecodeShortURLs

2010-09-17 Thread Steve Freegard
Hi All, Recently I've been getting a bit of filter-bleed from a bunch of spams injected via Hotmail/Yahoo that contain shortened URLs e.g. bit.ly/foo that upon closer inspection would have been rejected with a high score if the real URL had been used. To that end - it annoyed me enough to wr