Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Rob McEwen]

On 2/1/2017 12:56 AM, Dave Warren wrote:

They publish SPF records and DKIM sign everything for competent SMTP
receivers to handle in real-time, AND they publish a HTML version for
humans, and yet someone still finds a reason to complain?


Dave,

After the initial question was raised, it took about 11 posts and almost 
24 hours for someone to notice the discussion who happened to know about 
the "HTML version for humans" and mention that. During those 11 posts, a 
well-respected and knowledgeable person was actually defending Yahoo for 
NOT having such a page, which gave the impression that such didn't 
exist. (certainly, that was a head-fake that I fell for, even if such 
was very innocent)


So I think there is a strong argument that the existence of this page 
page isn't exactly common knowledge. Archive.org suggests that this page 
has only existed for a couple of years. I've been looking for it 
(occasionally) for the past 10 years - so I think all my memories of 
past discussions in past years about such a page not existing - were 
probably accurate. By the time this page existed, I had given up on 
finding it. (not that I spend every waking hour looking for it - I think 
I probably looked for it about once every year or two - for some time - 
and the need for this isn't so great with other senders - because few 
senders [even large ones] have such a MASSIVE amount of sending IPs that 
are so particularly hard to find)


Regarding your references about such a page not being needed - all I'm 
going to say is that some systems benefit from having large IP ranges 
preemptively whitelisted for the sake of efficiency. There are scenarios 
in certain very high volume systems where this enables the processing of 
messages at order of magnitudes faster rates than if SPF and DKIM and 
FCrDNS-confirmation had to be checked on every sending IP. MUCH of that 
relies on the response times of 3rd party servers - which (even at 
best!) is order of magnitudes slower than a local rbldnsd query  - or 
than an optimized binary search of an in-memory array - which is even 
faster than rbldnsd or even a high-end in-memory database. Sometimes, 
such 3rd party servers can "freeze up" in their responses, or rate limit 
queries - or firewall such lookups for what is perceived as abuse - 
causing further complications. Caching only does so much to prevent this!


That kind of need for speed is the world in which I live. At 
invaluement, I'm processing dozens of spams per second - and since much 
of these are ones where the "low-hanging fruit" - such as ALREADY 
heavily blacklisted botnet-sent spams are ALREADY filtered out before 
they get to my system - that means that the processing resources per 
spam is already much higher for my system than that of a typical ISP or 
hoster's natural incoming spam. (I process a higher concentration of the 
more sneaky spams and the newer emitters)


With this in mind... if I deleted my IP whitelist, and had to rely on 
SPF and DKIM and FCrDNS-verification for EVERY message, my queues would 
back up considerably - and a lot of worthy blacklistings of IPs and 
domains from new incoming spams would get considerably delayed. (again, 
inevitably - at this volume - issues come up where such 
queries/verification suddenly "freeze up" or get rate limited, 
firewalled, etc)


And I think my need for efficiency is probably not much different than 
some very large hosters and ISPs - who process mail for millions of users?


And I think we've already established that there is no possible way to 
generate "on demand" and remotely efficiently the information on that 
HTML page just via Yahoo's SPF records.


iow - maybe you should have a little more respect and try to be a little 
less snarky in the future - when you don't necessarily know/understand 
others' situation/requirements that may be a little different than your 
particular situation/requirements.


--
Rob McEwen

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Dave Warren]

On 2017-01-30 08:06, Dianne Skoll wrote:

On Mon, 30 Jan 2017 09:06:34 -0500
Rob McEwen  wrote:


On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:

they do and it has been mentioned:
https://help.yahoo.com/kb/SLN23997.html

Cool.  So Yahoo uses an HTML page that's a pain to process by
computer.


They publish SPF records and DKIM sign everything for competent SMTP 
receivers to handle in real-time, AND they publish a HTML version for 
humans, and yet someone still finds a reason to complain?


Maybe it's just me, but hand-maintaining a list of IPs to whitelist is 
so 1997s. The real value of SPF and DKIM is that you don't do any of 
that, you can whitelist by domain and let the sending domain tell you, 
in real time, whether or not the inbound message should be trusted.


Or, if you insist on doing things manually, glance at the HTML source 
and spend a good strong 3 minutes with your favourite regex parser and 
you're good to go.


 
has both the answer and shows my work.


But remember, this list is only valid until it isn't, even big providers 
move things around, sometimes frequently, so expect to update the list 
frequently (or again, don't, just use the tools that exist to do it in 
real time and go watch a movie instead).

Re: Custom rule problem

[Joe Quinn]

On 1/31/2017 3:22 PM, Zinski, Steve wrote:

Sorry for the trouble, everyone… I had been forwarding the spam through my 
personal IMAP account (to test my rule) which was apparently blocking it. I 
forwarded it using my gmail account and my new rule fired. I feel like an idiot.

Steve

I suggest you work on setting things up so you can break down each part 
individually. Mail flow is not always simple thing to keep track of, 
even when you have good tools.

Re: Custom rule problem

[Kevin A. McGrail]

On 1/31/2017 3:22 PM, Zinski, Steve wrote:

Sorry for the trouble, everyone… I had been forwarding the spam through my 
personal IMAP account (to test my rule) which was apparently blocking it. I 
forwarded it using my gmail account and my new rule fired. I feel like an idiot.

No worries.  Rookie mistake.  Just keep fighting the bastard spammers.

Re: Custom rule problem

[Zinski, Steve]

Sorry for the trouble, everyone… I had been forwarding the spam through my 
personal IMAP account (to test my rule) which was apparently blocking it. I 
forwarded it using my gmail account and my new rule fired. I feel like an idiot.

Steve



On 1/31/17, 2:53 PM, "John Hardin"  wrote:

On Tue, 31 Jan 2017, Zinski, Steve wrote:

> Here’s the “view source” of the message in question.
>
> http://pastebin.com/AnwkAf9t
>
> Again, it’s line 88 that I’m trying to match.

...let's try this again...

A uri rule hits that here:

Jan 31 09:21:07.423 [21842] dbg: rules: ran uri rule __ALL_URI ==> got 
hit: 
"http://trc.spam_domain_redacted.com/redirect.php?email=redac...@uronline.net";

It also hits an existing rule:

Jan 31 09:21:07.525 [21842] dbg: rules: ran rawbody rule __BUGGED_IMG 
==> got hit: "http://trc.spam_domain_redacted.com/redirect.php?email=re";


> On 1/31/17, 11:36 AM, "John Hardin"  wrote:
>
>On Tue, 31 Jan 2017, Zinski, Steve wrote:
>
>> I’m trying to write a custom rule to block a certain type of spam. 
When I view the message source, the very last lines of the spam look like this:
>>
>> 
>> http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>
>> 
>> 
>>
>> Every single rule that I’ve written fails to detect that 
redirect.php URI. I’ve even tried a rule that simply reads:
>>
>> Full  my_rule /redirect/is
>> Score  my_rule 10.0
>>
>> No match. I’ve tried full, rawbody, uri, and body, all to no avail. 
I’ve even shortened the search string to “redi” (it’s a unique word) and still 
no match. I’ve been writing rules for many years and this is the first time 
I’ve seen this behavior. Any ideas?
>
>If you have a rule dev environment (vs. testing rules in your live
>install) I've found something like this to be really useful:
>
>   uri __ALL_URI   /.*/
>   tflags  __ALL_URI   multiple
>
>Then all the detected URIs appear in the rule hits debug output.
>
>Post the full email on Pastebin or similar, we can't meaningfully 
comment
>on what you provided beyond "uri *should* work for that".

-- 
  John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
  jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Tomorrow: the 14th anniversary of the loss of STS-107 Columbia

Re: Custom rule problem

[Martin Gregorie]

On Tue, 2017-01-31 at 11:53 -0800, John Hardin wrote:
> On Tue, 31 Jan 2017, Zinski, Steve wrote:
> 
> > Here’s the “view source” of the message in question.
> >
> > http://pastebin.com/AnwkAf9t
> >
> > Again, it’s line 88 that I’m trying to match.
> 
> ...let's try this again...
> 
> A uri rule hits that here:
> 
> Jan 31 09:21:07.423 [21842] dbg: rules: ran uri rule __ALL_URI
> ==> got hit: "http://trc.spam_domain_redacted.com/redirect.php?em
> ail=redac...@uronline.net"
> 
> It also hits an existing rule:
> 
> Jan 31 09:21:07.525 [21842] dbg: rules: ran rawbody rule __BUGGED_IMG
> ==> got hit: "http://trc.spam_domain_redacted.com/redir
> ect.php?email=re"
> 
Like John, the text you posted hits one of my private rules when fed
through my rule testing and development environment. This is a metarule
that fires if a URI subrule finds a PHP script reference OR a BODY
subrule finds a PHP script reference preceded and followed by O-32 non-
whitespace characters.

So, questions:

- how did you capture the text you posted, 
  i.e. is it exactly the same as SA would have seen?

- did you restart SA before running each of the tests you describe?
  Every so often I forget that and then waste time with head scratching
  until I remember to restart SA. 


Martin

Re: Custom rule problem

[John Hardin]

On Tue, 31 Jan 2017, Zinski, Steve wrote:


Here’s the “view source” of the message in question.

http://pastebin.com/AnwkAf9t

Again, it’s line 88 that I’m trying to match.


...let's try this again...

A uri rule hits that here:

Jan 31 09:21:07.423 [21842] dbg: rules: ran uri rule __ALL_URI ==> got hit: 
"http://trc.spam_domain_redacted.com/redirect.php?email=redac...@uronline.net";

It also hits an existing rule:

Jan 31 09:21:07.525 [21842] dbg: rules: ran rawbody rule __BUGGED_IMG ==> got hit: 
"http://trc.spam_domain_redacted.com/redirect.php?email=re";



On 1/31/17, 11:36 AM, "John Hardin"  wrote:

   On Tue, 31 Jan 2017, Zinski, Steve wrote:

   > I’m trying to write a custom rule to block a certain type of spam. When I 
view the message source, the very last lines of the spam look like this:
   >
   > 
   > http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>
   > 
   > 
   >
   > Every single rule that I’ve written fails to detect that redirect.php URI. 
I’ve even tried a rule that simply reads:
   >
   > Full  my_rule /redirect/is
   > Score  my_rule 10.0
   >
   > No match. I’ve tried full, rawbody, uri, and body, all to no avail. I’ve 
even shortened the search string to “redi” (it’s a unique word) and still no 
match. I’ve been writing rules for many years and this is the first time I’ve seen 
this behavior. Any ideas?

   If you have a rule dev environment (vs. testing rules in your live
   install) I've found something like this to be really useful:

uri __ALL_URI   /.*/
tflags  __ALL_URI   multiple

   Then all the detected URIs appear in the rule hits debug output.

   Post the full email on Pastebin or similar, we can't meaningfully comment
   on what you provided beyond "uri *should* work for that".


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
 Tomorrow: the 14th anniversary of the loss of STS-107 Columbia

Re: Custom rule problem

[Zinski, Steve]

Here’s the “view source” of the message in question.

http://pastebin.com/AnwkAf9t

Again, it’s line 88 that I’m trying to match.

Thanks.




On 1/31/17, 11:36 AM, "John Hardin"  wrote:

On Tue, 31 Jan 2017, Zinski, Steve wrote:

> I’m trying to write a custom rule to block a certain type of spam. When I 
view the message source, the very last lines of the spam look like this:
>
> 
> http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>
> 
> 
>
> Every single rule that I’ve written fails to detect that redirect.php 
URI. I’ve even tried a rule that simply reads:
>
> Full  my_rule /redirect/is
> Score  my_rule 10.0
>
> No match. I’ve tried full, rawbody, uri, and body, all to no avail. I’ve 
even shortened the search string to “redi” (it’s a unique word) and still no 
match. I’ve been writing rules for many years and this is the first time I’ve 
seen this behavior. Any ideas?

If you have a rule dev environment (vs. testing rules in your live 
install) I've found something like this to be really useful:

uri __ALL_URI   /.*/
tflags  __ALL_URI   multiple

Then all the detected URIs appear in the rule hits debug output.

Post the full email on Pastebin or similar, we can't meaningfully comment 
on what you provided beyond "uri *should* work for that".


-- 
  John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
  jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
   The promise of nuclear power: electricity too cheap to meter
   The reality of nuclear power: FUD too cheap to meter
---
  Tomorrow: the 14th anniversary of the loss of STS-107 Columbia

Re: Custom rule problem

[John Hardin]

On Tue, 31 Jan 2017, Zinski, Steve wrote:


I’m trying to write a custom rule to block a certain type of spam. When I view 
the message source, the very last lines of the spam look like this:


http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>



Every single rule that I’ve written fails to detect that redirect.php URI. I’ve 
even tried a rule that simply reads:

Full  my_rule /redirect/is
Score  my_rule 10.0

No match. I’ve tried full, rawbody, uri, and body, all to no avail. I’ve even 
shortened the search string to “redi” (it’s a unique word) and still no match. 
I’ve been writing rules for many years and this is the first time I’ve seen 
this behavior. Any ideas?


If you have a rule dev environment (vs. testing rules in your live 
install) I've found something like this to be really useful:


uri __ALL_URI   /.*/
tflags  __ALL_URI   multiple

Then all the detected URIs appear in the rule hits debug output.

Post the full email on Pastebin or similar, we can't meaningfully comment 
on what you provided beyond "uri *should* work for that".



--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The promise of nuclear power: electricity too cheap to meter
  The reality of nuclear power: FUD too cheap to meter
---
 Tomorrow: the 14th anniversary of the loss of STS-107 Columbia

Re: Custom rule problem

[Kevin A. McGrail]

On 1/31/2017 10:45 AM, Zinski, Steve wrote:


Hello, I have a problem that I hope someone can help me with.

I’m trying to write a custom rule to block a certain type of spam. 
When I view the message source, the very last lines of the spam look 
like this:




src="http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>






Every single rule that I’ve written fails to detect that redirect.php 
URI. I’ve even tried a rule that simply reads:


Full my_rule /redirect/is

Score my_rule 10.0

No match. I’ve tried full, rawbody, uri, and body, all to no avail. 
I’ve even shortened the search string to “redi” (it’s a unique word) 
and still no match. I’ve been writing rules for many years and this is 
the first time I’ve seen this behavior. Any ideas?



So I use some old school methods for custom rule development.

I always use my initials and then I like to use mutt as my mail client 
and bind ctrl y (as in why is this spam) with something like this:


macro index \cy "spamassassin -t -D 2>&1 | grep -e KAM -e 
Content\\ analysis\n" "Test Message with Apache SpamAssassin for KAM"


mutt is very old school and let's me see if the message format is 
something odd.  Perhaps the issue you are seeing.  Throw the email up on 
pastebin in mbox format and I'll tell you what I see at least.


Regards,
KAM

Custom rule problem

[Zinski, Steve]

Hello, I have a problem that I hope someone can help me with.

I’m trying to write a custom rule to block a certain type of spam. When I view 
the message source, the very last lines of the spam look like this:


http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.edu";>



Every single rule that I’ve written fails to detect that redirect.php URI. I’ve 
even tried a rule that simply reads:

Full  my_rule /redirect/is
Score  my_rule 10.0

No match. I’ve tried full, rawbody, uri, and body, all to no avail. I’ve even 
shortened the search string to “redi” (it’s a unique word) and still no match. 
I’ve been writing rules for many years and this is the first time I’ve seen 
this behavior. Any ideas?

Re: Custom rule problem

[Antony Stone]

On Tuesday 31 January 2017 at 16:45:34, Zinski, Steve wrote:

> Hello, I have a problem that I hope someone can help me with.
> 
> I’m trying to write a custom rule to block a certain type of spam. When I
> view the message source, the very last lines of the spam look like this:

How are you seeing this?  Asking your mail client to "show source", or looking 
at the email as it appears whilst going through your mail server?

> 
>  src="http://trc.spammersdomain.com/redirect.php?email=redac...@richmond.ed
> u"> 
> 
> 
> Every single rule that I’ve written fails to detect that redirect.php URI.
> I’ve even tried a rule that simply reads:
> 
> Full  my_rule /redirect/is
> Score  my_rule 10.0
> 
> No match. I’ve tried full, rawbody, uri, and body, all to no avail. I’ve
> even shortened the search string to “redi” (it’s a unique word) and still
> no match. I’ve been writing rules for many years and this is the first
> time I’ve seen this behavior. Any ideas?

Is the email as seen by SpamAssassin Base-64 encoded?


Antony.

-- 
APL [is a language], in which you can write a program to simulate shuffling a 
deck of cards and then dealing them out to several players, in four 
characters, none of which appear on a standard keyboard.

 - David Given

   Please reply to the list;
 please *don't* CC me.