Re: OT: RE: URI with spaces are not recognized

[Kenneth Porter]

--On Monday, February 16, 2009 8:57 AM +1300 Michael Hutchinson 
 wrote:



"plenty of people are greedy, gullible, uninformed, overly trusting,
stupid, or some combination of the above"

This also means: "Anyone that doesn't use a computer as much as an
E-Mail administrator"


Coincidentally, this blog entry was referenced on Slashdot today:



Slashdot coverage:



Comment there suggesting that it might really be the users:




You can't expect everyone to know enough about Spam to not be fooled by
it. The reason people do get fooled is because they aren't all computer
technicians. Everyone is good at something, lets not get carried away
and blame joe bloggs for being.. joe bloggs.. after all, he might be the
next automotive technician to fix your car.


I agree that everyone is ignorant about something, and often an expert at 
something else. But I don't think that you need to be an email/spam expert 
to recognize a con. The same principles used in these things are also 
successfully used in telephone and paper mail scams, where knowledge of the 
communicating technology is irrelevant.


I think our culture suffers from too little skepticism.

Re: OT: RE: URI with spaces are not recognized

[Matus UHLAR - fantomas]

On 16.02.09 08:57, Michael Hutchinson wrote:
> "plenty of people are greedy, gullible, uninformed, overly trusting,
> stupid, or some combination of the above" 
> 
> This also means: "Anyone that doesn't use a computer as much as an
> E-Mail administrator" 
> 
> You can't expect everyone to know enough about Spam to not be fooled by
> it. The reason people do get fooled is because they aren't all computer
> technicians. Everyone is good at something, lets not get carried away
> and blame joe bloggs for being.. joe bloggs.. after all, he might be the
> next automotive technician to fix your car.
> 
> Why write off topic? Well, we do need to understand the end-user, they
> are the ones who see the benefit of our work, no? Are they not the ones
> who pay many of our wages?

While I agree with you in the above statement(s), I must also agree with
Kevin with his statements below. and I think that the first one (AI vs
natural stupidity) should be taken as a premise. 

There are no foolproof things, because fools are very inventive.

> -Original Message-
> From: Kevin Parris [mailto:kpar...@ed.sc.gov] 
> Sent: Saturday, 14 February 2009 9:43 a.m.
> To: users@spamassassin.apache.org
> Subject: Re: URI with spaces are not recognized
> 
> Artificial intelligence will never overcome natural stupidity (or the
> clever ingenuity of criminals) ... if people actually DO that (copy the
> "url" and remove the spaces) there is some temptation to say they get
> what they deserve ... but on the other hand most of the spam/scam stuff
> out there is based on the premise that plenty of people are greedy,
> gullible, uninformed, overly trusting, stupid, or some combination of
> the above.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
M$ Win's are shit, do not use it !

OT: RE: URI with spaces are not recognized

[Michael Hutchinson]

"plenty of people are greedy, gullible, uninformed, overly trusting,
stupid, or some combination of the above" 

This also means: "Anyone that doesn't use a computer as much as an
E-Mail administrator" 

You can't expect everyone to know enough about Spam to not be fooled by
it. The reason people do get fooled is because they aren't all computer
technicians. Everyone is good at something, lets not get carried away
and blame joe bloggs for being.. joe bloggs.. after all, he might be the
next automotive technician to fix your car.

Why write off topic? Well, we do need to understand the end-user, they
are the ones who see the benefit of our work, no? Are they not the ones
who pay many of our wages?

2c/Cheers.


-Original Message-
From: Kevin Parris [mailto:kpar...@ed.sc.gov] 
Sent: Saturday, 14 February 2009 9:43 a.m.
To: users@spamassassin.apache.org
Subject: Re: URI with spaces are not recognized

Artificial intelligence will never overcome natural stupidity (or the
clever ingenuity of criminals) ... if people actually DO that (copy the
"url" and remove the spaces) there is some temptation to say they get
what they deserve ... but on the other hand most of the spam/scam stuff
out there is based on the premise that plenty of people are greedy,
gullible, uninformed, overly trusting, stupid, or some combination of
the above.

>>> Franz Schwartau  02/13/09 2:18 PM >>>
C'mon...

Patient: "Doctor, if I press down here it really hurts..."
Doctor: "Don't press there then."

You won't solve a problem by defining there is no problem.

In these spams people are requested to remove the spaces when entering
the given string ("url") in their browser.

Benny Pedersen wrote:
> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>> www . abcdef .  net
>>
>> After reading the source for a while I found that $schemelessRE in
>> line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be
>> responsible for that. Unfortunally this regexp doesn't care
>> about whitespaces.
> 
> give me a url to a browser that can show above url is simple :)
> 
> even my firefox in my nokia phone wont show this, did i miss another
> one ?
> 
>> Has anyone a solution?
> 
> none so far have a problem ?
> 
>> Would be fine if I could use the "uri" directive
>> or even some uribl on this kind of "urls".
> 
> it will if there was a problem

Re: URI with spaces are not recognized

[mouss]

Wolfgang Zeikat a écrit :
> I think the discussion is getting carried in a direction where we are
> missing a point: spam detection.
> 

exactly.

otherwise, there's no point to waste resources running SA. after all,
nobody would die for visiting a porn/casino/pharma/... site ;-p

and there's also another case: in a school or at home, you don't want
"these" messages to reach children mailboxes, and if an adult
(moderator) checks the delivered messages, you want to reduce his work
by filtering out as much junk as you can.

Re: URI with spaces are not recognized

[Franz Schwartau]

Hi John!

John Hardin wrote:
> On Fri, 13 Feb 2009, Benny Pedersen wrote:
> 
>> On Fri, February 13, 2009 18:12, John Hardin wrote:
>>> If a URI rule works, what's wrong with a body rule?
>>
>> nothing wroung making bad rules either, point is that if bad rules
>> is needed one have also bad behaving browser problem
> 
> Why should the fact that a mail client won't render that URI as a
> clickable link mean there shouldn't be a rule for it? Spammers have been
> obfuscating URIs in this manner for a long time. There's nothing wrong
> with rules for obfuscated URIs.

Thanks for pointing out! :-) Our primary goal is to identify spam, not
to prevent people from typing these obfuscated URLs in their browser...

> OT: Benny, could you refrain from setting your Reply-To to the email
> address of the original poster? Setting it to the mailing list address
> is fine, but setting it to the original poster is just
> passive-aggressive rudeness.
> 
> On Fri, 13 Feb 2009, Franz Schwartau wrote:
> 
>> So, does anyone know a more general solution for this kind of spam
>> instead of individual body rules?
> 
> You might try a rule like:
> 
>  body URI_SPC_OBFU_SPC
> /\bwww\s{1,20}\.\s{1,20}\w{5,20}\s{1,20}\.\s{1,20}net\b/i
> 
> I think it would be risky to make the URI parser attempt too much
> deobfuscation; however, accepting \s+\.\s+ as \. might be justified.
> Perhaps \s+dot\s+ as well.
> 
> If the spammer uses something more complex they're reducing the
> likelihood the recipient will bother to deobfuscate the URI, and it's
> more likely to be caught by bayes, so I'd suggest the ROI to SA for
> making it more aggressive isn't large enough.

I thought about this generic body rule, too. Unfortunally this rule
catches also legitimate mistyped URLs containing spaces. Think of users
typing URLs fast and hitting the space bar accidentally while typing. ;-)

After reading PerMsgStatus.pm again another idea came up. Instead of
modifying $schemelessRE (which wouldn't help anyway) the URLs containing
spaces are replaced by URLs without spaces before spamassassin gathers
URIs. Thus all URI specific rules can be applied (e. g. uri directive
and URI blacklists).

The regexp is kept simple intentionally and matches legitimate (without
spaces) URLs as well but this doesn't hurt much.

This patch works for me and perhaps someone else finds it useful.
Comments are welcome, too. :-)

Best regards
Franz
--- PerMsgStatus.pm.new.orig2009-02-14 11:21:20.0 +0100
+++ PerMsgStatus.pm.new 2009-02-14 11:20:54.0 +0100
@@ -1417,7 +1417,13 @@
 =cut
 
 sub get_decoded_stripped_body_text_array {
-  return $_[0]->{msg}->get_rendered_body_text_array();
+  my $textary = $_[0]->{msg}->get_rendered_body_text_array();
+
+  for (@$textary) {
+
s/(www)\s{0,2}\.\s{0,2}([a-z\d._-]{10,32})\s{0,2}\.\s{0,2}((net|org))/$1.$2.$3/i;
+  }
+
+  return $textary;
 }
 
 ###

Re: URI with spaces are not recognized

[McDonald, Dan]

On Fri, 2009-02-13 at 15:43 -0500, Kevin Parris wrote:
> Artificial intelligence will never overcome natural stupidity (or the
> clever ingenuity of criminals) ... if people actually DO that (copy
> the "url" and remove the spaces) there is some temptation to say they
> get what they deserve ... but on the other hand most of the spam/scam
> stuff out there is based on the premise that plenty of people are
> greedy, gullible, uninformed, overly trusting, stupid, or some
> combination of the above.


Whether they are clickable or not, they are still annoying. My hands are
only good for about 50,000 clicks per day, I don't want to waste any of
those on individual spams


-- 
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com



signature.asc
Description: This is a digitally signed message part

Re: URI with spaces are not recognized

[Wolfgang Zeikat]

I think the discussion is getting carried in a direction where we are 
missing a point: spam detection.


Kevin Parris wrote:

Artificial intelligence will never overcome natural stupidity (or the
clever ingenuity of criminals) ... if people actually DO that (copy
the "url" and remove the spaces) there is some temptation to say they
get what they deserve ... but on the other hand most of the spam/scam
stuff out there is based on the premise that plenty of people are
greedy, gullible, uninformed, overly trusting, stupid, or some
combination of the above.


Franz Schwartau  02/13/09 2:18 PM >>>

You won't solve a problem by defining there is no problem.

In these spams people are requested to remove the spaces when
entering the given string ("url") in their browser.


IMHO, the point here is:
how can these obfuscated URI be detected as such and be submitted to 
URI(BL) rules, so that those mails can more easily be classified as what 
they are: spam - no matter what final recipients might "deserve" or do 
with them (or not).


Regards,

wolfgang

Re: URI with spaces are not recognized

[Kevin Parris]

Artificial intelligence will never overcome natural stupidity (or the clever 
ingenuity of criminals) ... if people actually DO that (copy the "url" and 
remove the spaces) there is some temptation to say they get what they deserve 
... but on the other hand most of the spam/scam stuff out there is based on the 
premise that plenty of people are greedy, gullible, uninformed, overly 
trusting, stupid, or some combination of the above.

>>> Franz Schwartau  02/13/09 2:18 PM >>>
C'mon...

Patient: "Doctor, if I press down here it really hurts..."
Doctor: "Don't press there then."

You won't solve a problem by defining there is no problem.

In these spams people are requested to remove the spaces when entering the 
given string ("url") in their browser.

Benny Pedersen wrote:
> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>> www . abcdef .  net
>>
>> After reading the source for a while I found that $schemelessRE in
>> line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be
>> responsible for that. Unfortunally this regexp doesn't care
>> about whitespaces.
> 
> give me a url to a browser that can show above url is simple :)
> 
> even my firefox in my nokia phone wont show this, did i miss another
> one ?
> 
>> Has anyone a solution?
> 
> none so far have a problem ?
> 
>> Would be fine if I could use the "uri" directive
>> or even some uribl on this kind of "urls".
> 
> it will if there was a problem

Re: URI with spaces are not recognized

[John Hardin]

On Fri, 13 Feb 2009, McDonald, Dan wrote:


On Fri, 2009-02-13 at 11:55 -0800, John Hardin wrote:

On Fri, 13 Feb 2009, Franz Schwartau wrote:


So, does anyone know a more general solution for this kind of spam
instead of individual body rules?


You might try a rule like:

  body URI_SPC_OBFU_SPC 
/\bwww\s{1,20}\.\s{1,20}\w{5,20}\s{1,20}\.\s{1,20}net\b/i


I'd go a little further:

/\bwww\s{1,10}\.\s{1,10}\w{5,20}\s{1,10}\.\s{1,10}(?:com|net|org)\b/i


Well, yeah, that's of course possible. Only Franz knows the character of 
the domain TLDs he's seeing, though.


info and biz are two other much-abused TLDs.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Public Education: the bureaucratic process of replacing
  an empty mind with a closed one.  -- Thorax
---
 9 days until George Washington's 277th Birthday

Re: URI with spaces are not recognized

[McDonald, Dan]

On Fri, 2009-02-13 at 11:55 -0800, John Hardin wrote:
> On Fri, 13 Feb 2009, Benny Pedersen wrote:
> 
> > So, does anyone know a more general solution for this kind of spam
> > instead of individual body rules?
> 
> You might try a rule like:
> 
>   body URI_SPC_OBFU_SPC 
> /\bwww\s{1,20}\.\s{1,20}\w{5,20}\s{1,20}\.\s{1,20}net\b/i

I'd go a little further:

/\bwww\s{1,10}\.\s{1,10}\w{5,20}\s{1,10}\.\s{1,10}(?:com|net|org)\b/i

-- 
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com



signature.asc
Description: This is a digitally signed message part

Re: URI with spaces are not recognized

[John Hardin]

On Fri, 13 Feb 2009, Benny Pedersen wrote:


On Fri, February 13, 2009 18:12, John Hardin wrote:

If a URI rule works, what's wrong with a body rule?


nothing wroung making bad rules either, point is that if bad rules
is needed one have also bad behaving browser problem


Why should the fact that a mail client won't render that URI as a 
clickable link mean there shouldn't be a rule for it? Spammers have been 
obfuscating URIs in this manner for a long time. There's nothing wrong 
with rules for obfuscated URIs.


OT: Benny, could you refrain from setting your Reply-To to the email 
address of the original poster? Setting it to the mailing list address is 
fine, but setting it to the original poster is just passive-aggressive 
rudeness.


On Fri, 13 Feb 2009, Franz Schwartau wrote:


So, does anyone know a more general solution for this kind of spam
instead of individual body rules?


You might try a rule like:

 body URI_SPC_OBFU_SPC /\bwww\s{1,20}\.\s{1,20}\w{5,20}\s{1,20}\.\s{1,20}net\b/i

I think it would be risky to make the URI parser attempt too much 
deobfuscation; however, accepting \s+\.\s+ as \. might be justified. 
Perhaps \s+dot\s+ as well.


If the spammer uses something more complex they're reducing the likelihood 
the recipient will bother to deobfuscate the URI, and it's more likely to 
be caught by bayes, so I'd suggest the ROI to SA for making it more 
aggressive isn't large enough.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Windows Vista: Windows ME for the XP generation.
---
 9 days until George Washington's 277th Birthday

Re: URI with spaces are not recognized

[Benny Pedersen]

On Fri, February 13, 2009 20:18, Franz Schwartau wrote:
> C'mon...

france

> Patient: "Doctor, if I press down here it really hurts..."
> Doctor: "Don't press there then."

thats real life, not email

> You won't solve a problem by defining there is no problem.

where is the problem ?, 40 cm from the screen or so ?

> In these spams people are requested to remove the spaces when
> entering the given string ("url") in their browser.

such users ask for problems :)

-- 
http://localhost/ 100% uptime and 100% mirrored :)

Re: URI with spaces are not recognized

[Franz Schwartau]

C'mon...

Patient: "Doctor, if I press down here it really hurts..."
Doctor: "Don't press there then."

You won't solve a problem by defining there is no problem.

In these spams people are requested to remove the spaces when entering
the given string ("url") in their browser.

Benny Pedersen wrote:
> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>> www . abcdef .  net
>>
>> After reading the source for a while I found that $schemelessRE in
>> line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be
>> responsible for that. Unfortunally this regexp doesn't care
>> about whitespaces.
> 
> give me a url to a browser that can show above url is simple :)
> 
> even my firefox in my nokia phone wont show this, did i miss another
> one ?
> 
>> Has anyone a solution?
> 
> none so far have a problem ?
> 
>> Would be fine if I could use the "uri" directive
>> or even some uribl on this kind of "urls".
> 
> it will if there was a problem

Re: URI with spaces are not recognized

[Franz Schwartau]

Hi John,

thanks for your answer. Probably I should have written more about my
problem.

We're getting a lot of spam with obfuscated urls in the form

www . domain .  net

The domain part changes quite often (about daily). The number of domains
is nearly 100 by now. Of course we have body rules for each domain/url
similar to your rule but our time to detect new domains/urls is too slow
(actually our customer has to tell us, that spam got through, which is
quite bad). All these "urls" point to the same content, resolve to the
same ip and are listed in some url black lists. Since spamassassin
doesn't recognize these obfuscated urls, url specific rules don't match.

So, does anyone know a more general solution for this kind of spam
instead of individual body rules?

Best regards
Franz

John Hardin wrote:
> On Fri, 13 Feb 2009, Benny Pedersen wrote:
> 
>> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>>> www . abcdef .  net
>>> Would be fine if I could use the "uri" directive
> 
> If a URI rule works, what's wrong with a body rule?
> 
> body URI_SPC_OBFU_nn
> /\bwww\s{1,20}\.\s{1,20}abcdef\s{1,20}\.\s{1,20}net\b/i

Re: URI with spaces are not recognized

[Benny Pedersen]

On Fri, February 13, 2009 18:12, John Hardin wrote:
> If a URI rule works, what's wrong with a body rule?

nothing wroung making bad rules either, point is that if bad rules
is needed one have also bad behaving browser problem

-- 
http://localhost/ 100% uptime and 100% mirrored :)

Re: URI with spaces are not recognized

[John Hardin]

On Fri, 13 Feb 2009, Benny Pedersen wrote:


On Thu, February 12, 2009 18:26, Franz Schwartau wrote:

www . abcdef .  net
Would be fine if I could use the "uri" directive


If a URI rule works, what's wrong with a body rule?

body URI_SPC_OBFU_nn /\bwww\s{1,20}\.\s{1,20}abcdef\s{1,20}\.\s{1,20}net\b/i

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The fetters imposed on liberty at home have ever been forged out
  of the weapons provided for defense against real, pretended, or
  imaginary dangers from abroad.   -- James Madison, 1799
---
 9 days until George Washington's 277th Birthday

Re: URI with spaces are not recognized

[Benny Pedersen]

On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
> www . abcdef .  net
>
> After reading the source for a while I found that $schemelessRE in
> line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be
> responsible for that. Unfortunally this regexp doesn't care
> about whitespaces.

give me a url to a browser that can show above url is simple :)

even my firefox in my nokia phone wont show this, did i miss another
one ?

> Has anyone a solution?

none so far have a problem ?

> Would be fine if I could use the "uri" directive
> or even some uribl on this kind of "urls".

it will if there was a problem

-- 
http://localhost/ 100% uptime and 100% mirrored :)

URI with spaces are not recognized

[Franz Schwartau]

Hi!

A lot of spams arrives here with URI like strings containing spaces, e.g.:

www . abcdef .  net

After reading the source for a while I found that $schemelessRE in line
1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be responsible for
that. Unfortunally this regexp doesn't care about whitespaces.

Has anyone a solution? Would be fine if I could use the "uri" directive
or even some uribl on this kind of "urls".

Best regards
Franz