Re: Plugin for URL shorteners / redirects
On Tue, May 26, 2009 at 06:10:34PM -0700, John Hardin wrote: On Wed, 27 May 2009, Jason Haar wrote: Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? I'd suggest hex- or base64-encoding the MD5 hash of the URI, as is being done in Email BL. You won't see paths on URIBL or SURBL, as they even haven't adopted wildcards (try googling some discussions years ago). Better leave the path cases to publicly rsyncable stuff like Sanesecurity.
Re: Plugin for URL shorteners / redirects
Yes. it immediately exposes a backchannel from the spam to the spammer, thereby enabling a number of interesting security holes. --j. On Wed, May 27, 2009 at 05:25, Rob McEwen r...@invaluement.com wrote: Jason Haar wrote: Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? Because spammers could easily generate a unique URL for each individual spam. They could then map this back to listings in URI blacklists and use that as a very cheap and effective way to listwash. And they only need to add a single astricked hostname in their DNS server to accomplish this. As a result of this and similar tactics, URI lists would bloat exponentially and this would slow down the propagation of the data to rsync users and to DNS mirrors, as well as bringing the backend processing to its knees. Finally, there is some amount of reputation and registration (even if hidden) associated with a domain due to the fact that a domain *requires* ownership. URLs and subdomains are more ambiguous, which then also makes removal requests extremely subjective and murky process. -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
Re: Plugin for URL shorteners / redirects
On Wednesday, May 27, 2009, 1:39:11 AM, Justin Mason wrote: Yes. it immediately exposes a backchannel from the spam to the spammer, thereby enabling a number of interesting security holes. --j. Yes, it's impractical for some of the reasons Rob mentions, and it would also allow any of the following: 1. Listwashing 2. Mapping out of spam traps 3. Poisoning of spam traps 4. Confirming delivery of spams and email addresses etc. Jeff C. On Wed, May 27, 2009 at 05:25, Rob McEwen r...@invaluement.com wrote: Jason Haar wrote: Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? Because spammers could easily generate a unique URL for each individual spam. They could then map this back to listings in URI blacklists and use that as a very cheap and effective way to listwash. And they only need to add a single astricked hostname in their DNS server to accomplish this. As a result of this and similar tactics, URI lists would bloat exponentially and this would slow down the propagation of the data to rsync users and to DNS mirrors, as well as bringing the backend processing to its knees. Finally, there is some amount of reputation and registration (even if hidden) associated with a domain due to the fact that a domain *requires* ownership. URLs and subdomains are more ambiguous, which then also makes removal requests extremely subjective and murky process.
Re: Plugin for URL shorteners / redirects
On Tuesday, May 26, 2009, 6:20:13 PM, Jason Haar wrote: John Hardin wrote: Better still, the tinyurl-esque services should vet the URLs people submit against SURBL... They actually do. When I was trying to test Jonas URLredirect plugin, it was actually hard to get tinyurl.com to generate a link for some known spam URLs. I suspect they are indeed doing SURBL lookups. Hope I didn't end up blacklisting myself :-} Yes, tinyurl and several other URL shortening services use SURBL data to fight abuse of their services: http://www.surbl.org/redirect.html Jeff C. -- Jeff Chan mailto:je...@surbl.org http://www.surbl.org/
Re: Plugin for URL shorteners / redirects
On 5/27/2009 11:00 AM, Jeff Chan wrote: On Tuesday, May 26, 2009, 6:20:13 PM, Jason Haar wrote: John Hardin wrote: Better still, the tinyurl-esque services should vet the URLs people submit against SURBL... They actually do. When I was trying to test Jonas URLredirect plugin, it was actually hard to get tinyurl.com to generate a link for some known spam URLs. I suspect they are indeed doing SURBL lookups. Hope I didn't end up blacklisting myself :-} Yes, tinyurl and several other URL shortening services use SURBL data to fight abuse of their services: http://www.surbl.org/redirect.html http://bit.ly/hnds8 This resource has permanently moved to a href='http://alvinabeate.narod.ru/pages.html'http://alvinabeate.narod.ru/pages.html/a. even if SURBL saw that URL, as it doesn't list 2ltlds, alvinabeate.narod.ru will not be listed. other URI blacklists won't ever see alvinabeate.narod.ru either as its not in the mail flow.
Plugin for URL shorteners / redirects
Hi! I just threw together a plugin that can check URLs for redirections, and add whatever they redirect to to the message meta-data so that the true destinations are checked by URIBLs etc. It doesn't do this for all URLs in a message. I will only follow those URLs it is specifically told to follow. Also, it only asks for HEAD rather than pages in order to keep the traffic down. I'm not sure wether this is really worthwhile or if it is just a waste of time and resources, but the idea is to use it for URL shorteners that are beeing abused by spammers. To be really useful it needs a list of abused URL shorteners. I don't know wich shorteners are most abused, so I don't know what the list should contain. (The three example shorteners are in the POD because I knew about them, not because they are beeing used by spammers.) If anyone thinks this is a good idea you can check the plugin at http://whatever.frukt.org/spamassassin.text.shtml?accessibler#URLRedirect.pm. Suggestions and criticism are very welcome. URL shortener addresses (with formats) even more welcome. Notes: This is not extensively tested. It may well contain bugs. It's not a finished thing. If this plugin is a good idea, making it do it's HEAD requests in paralell would be a good idea, but I don't know what the best way to do that in perl for SA would be. (Currently it has a hardcoded timeout of 10 seconds around it's requesting stage, but no other time saving stuff.) Using a cache should also be implemented so that repeatedly seen URLs aren't followed over and over again. This should be pretty simple. Since it needs URL meta-data to be checked before it runs, and needs to add it's own meta-data before the rest of the scan run, it can't really work asyncronoulsy AFAICS. Currently it uses a parsed_metadata at priority -1 in order to add it's own meta-data. Maybe this isn't the right way to do this. Regards /Jonas -- Jonas Eckerman Fruktträdet Förbundet Sveriges Dövblinda http://www.fsdb.org/ http://www.frukt.org/ http://whatever.frukt.org/
Re: Plugin for URL shorteners / redirects
On Tue, May 26, 2009 17:37, Jonas Eckerman wrote: I just threw together a plugin that can check URLs for redirections, and add whatever they redirect to to the message meta-data so that the true destinations are checked by URIBLs etc. http://wiki.apache.org/spamassassin/WebRedirectPlugin know this plugin ? what is the diff in the testing ? -- http://localhost/ 100% uptime and 100% mirrored :)
Re: Plugin for URL shorteners / redirects
Benny Pedersen wrote: http://wiki.apache.org/spamassassin/WebRedirectPlugin know this plugin ? Yes. Though I hade forgotten it's name. what is the diff in the testing ? Reading the descriptions of the two plugins would have given you some good hints. Reading the documentation (both have PODs) would have given you the answer. They are very different. The WebRedirect plugin fetches pages. My plugin only fetches headers. The WebRedirect plugin adds the content of pages as pseudoheaders. My plugin adds the location for a redirect to the existing canonicalized list of URIs (so that existing URI checkers sees them). The WebRedirect plugin provides an eval test to check the status code for queried links. My plugin doesn't. /Jonas -- Jonas Eckerman Fruktträdet Förbundet Sveriges Dövblinda http://www.fsdb.org/ http://www.frukt.org/ http://whatever.frukt.org/
Re: Plugin for URL shorteners / redirects
Jonas Eckerman wrote: Hi! I just threw together a plugin that can check URLs for redirections, and add whatever they redirect to to the message meta-data so that the true destinations are checked by URIBLs etc... I'm giving this a go and it looks good. I do wonder how companies like tinyurl are going to respond if 500,000 mail servers suddenly turned around and hit it for the same URL within a 2 minute period. Techniques that perform callback do tend to get nasty. However (and I've bet this has been talked about before), this seems to be solving a problem that should have been solved by SURBL? Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? Isn't it true that antispam systems want to check email for known bad websites? As such that is defined as bad FQDN and bad URLs that would redirect users to the bad FQDN (ie redirectors). Just asking :-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Plugin for URL shorteners / redirects
On Wed, 27 May 2009, Jason Haar wrote: Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? I'd suggest hex- or base64-encoding the MD5 hash of the URI, as is being done in Email BL. Isn't it true that antispam systems want to check email for known bad websites? As such that is defined as bad FQDN and bad URLs that would redirect users to the bad FQDN (ie redirectors). Just asking :-) Sounds like a somewhat reasonable proposal to me. Better still, the tinyurl-esque services should vet the URLs people submit against SURBL... -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- News flash: Lowest Common Denominator down 50 points --- 36 days since 9th Circuit incorporated 2nd Amdt - MSM still silent
Re: Plugin for URL shorteners / redirects
John Hardin wrote: Sounds like a somewhat reasonable proposal to me. Oh? I assumed someone else had thought of it before, and I was just ignorant of the reasons against ;-) Better still, the tinyurl-esque services should vet the URLs people submit against SURBL... They actually do. When I was trying to test Jonas URLredirect plugin, it was actually hard to get tinyurl.com to generate a link for some known spam URLs. I suspect they are indeed doing SURBL lookups. Hope I didn't end up blacklisting myself :-} -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Plugin for URL shorteners / redirects
Jason Haar wrote: Why can't SURBL be expanded to support full URLs instead of just the hostname? That way you could blacklist a.bad.domain as well as xttx://tinyurl . com/redirect-to-bad-domain? Some form of BASE64 encoding would be needed of course, but why not? Because spammers could easily generate a unique URL for each individual spam. They could then map this back to listings in URI blacklists and use that as a very cheap and effective way to listwash. And they only need to add a single astricked hostname in their DNS server to accomplish this. As a result of this and similar tactics, URI lists would bloat exponentially and this would slow down the propagation of the data to rsync users and to DNS mirrors, as well as bringing the backend processing to its knees. Finally, there is some amount of reputation and registration (even if hidden) associated with a domain due to the fact that a domain *requires* ownership. URLs and subdomains are more ambiguous, which then also makes removal requests extremely subjective and murky process. -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
