Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Rob McEwen]

On 2/1/2017 12:56 AM, Dave Warren wrote:

They publish SPF records and DKIM sign everything for competent SMTP
receivers to handle in real-time, AND they publish a HTML version for
humans, and yet someone still finds a reason to complain?


Dave,

After the initial question was raised, it took about 11 posts and almost 
24 hours for someone to notice the discussion who happened to know about 
the "HTML version for humans" and mention that. During those 11 posts, a 
well-respected and knowledgeable person was actually defending Yahoo for 
NOT having such a page, which gave the impression that such didn't 
exist. (certainly, that was a head-fake that I fell for, even if such 
was very innocent)


So I think there is a strong argument that the existence of this page 
page isn't exactly common knowledge. Archive.org suggests that this page 
has only existed for a couple of years. I've been looking for it 
(occasionally) for the past 10 years - so I think all my memories of 
past discussions in past years about such a page not existing - were 
probably accurate. By the time this page existed, I had given up on 
finding it. (not that I spend every waking hour looking for it - I think 
I probably looked for it about once every year or two - for some time - 
and the need for this isn't so great with other senders - because few 
senders [even large ones] have such a MASSIVE amount of sending IPs that 
are so particularly hard to find)


Regarding your references about such a page not being needed - all I'm 
going to say is that some systems benefit from having large IP ranges 
preemptively whitelisted for the sake of efficiency. There are scenarios 
in certain very high volume systems where this enables the processing of 
messages at order of magnitudes faster rates than if SPF and DKIM and 
FCrDNS-confirmation had to be checked on every sending IP. MUCH of that 
relies on the response times of 3rd party servers - which (even at 
best!) is order of magnitudes slower than a local rbldnsd query  - or 
than an optimized binary search of an in-memory array - which is even 
faster than rbldnsd or even a high-end in-memory database. Sometimes, 
such 3rd party servers can "freeze up" in their responses, or rate limit 
queries - or firewall such lookups for what is perceived as abuse - 
causing further complications. Caching only does so much to prevent this!


That kind of need for speed is the world in which I live. At 
invaluement, I'm processing dozens of spams per second - and since much 
of these are ones where the "low-hanging fruit" - such as ALREADY 
heavily blacklisted botnet-sent spams are ALREADY filtered out before 
they get to my system - that means that the processing resources per 
spam is already much higher for my system than that of a typical ISP or 
hoster's natural incoming spam. (I process a higher concentration of the 
more sneaky spams and the newer emitters)


With this in mind... if I deleted my IP whitelist, and had to rely on 
SPF and DKIM and FCrDNS-verification for EVERY message, my queues would 
back up considerably - and a lot of worthy blacklistings of IPs and 
domains from new incoming spams would get considerably delayed. (again, 
inevitably - at this volume - issues come up where such 
queries/verification suddenly "freeze up" or get rate limited, 
firewalled, etc)


And I think my need for efficiency is probably not much different than 
some very large hosters and ISPs - who process mail for millions of users?


And I think we've already established that there is no possible way to 
generate "on demand" and remotely efficiently the information on that 
HTML page just via Yahoo's SPF records.


iow - maybe you should have a little more respect and try to be a little 
less snarky in the future - when you don't necessarily know/understand 
others' situation/requirements that may be a little different than your 
particular situation/requirements.


--
Rob McEwen

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Dave Warren]

On 2017-01-30 08:06, Dianne Skoll wrote:

On Mon, 30 Jan 2017 09:06:34 -0500
Rob McEwen  wrote:


On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:

they do and it has been mentioned:
https://help.yahoo.com/kb/SLN23997.html

Cool.  So Yahoo uses an HTML page that's a pain to process by
computer.


They publish SPF records and DKIM sign everything for competent SMTP 
receivers to handle in real-time, AND they publish a HTML version for 
humans, and yet someone still finds a reason to complain?


Maybe it's just me, but hand-maintaining a list of IPs to whitelist is 
so 1997s. The real value of SPF and DKIM is that you don't do any of 
that, you can whitelist by domain and let the sending domain tell you, 
in real time, whether or not the inbound message should be trusted.


Or, if you insist on doing things manually, glance at the HTML source 
and spend a good strong 3 minutes with your favourite regex parser and 
you're good to go.


 
has both the answer and shows my work.


But remember, this list is only valid until it isn't, even big providers 
move things around, sometimes frequently, so expect to update the list 
frequently (or again, don't, just use the tools that exist to do it in 
real time and go watch a movie instead).

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[David Jones]

>From: Dianne Skoll 
    
>On Mon, 30 Jan 2017 09:06:34 -0500
>Rob McEwen  wrote:

>> On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:
>> > they do and it has been mentioned:
>> > https://help.yahoo.com/kb/SLN23997.html

>Yahoo Outbound IP addresses | Yahoo Help - SLN23997
>help.yahoo.com
>Yahoo Outbound IP addresses. If you're looking for a list of IP addresses that 
>Yahoo Mail sends emails from, we >have them for you below. Just click a link 
>below to ...

Quick and dirty (I know there are many different ways to do this
so I am not saying this is the only way -- no flaming please.):

elinks -dump https://help.yahoo.com/kb/SLN23997.html | grep -E 
'([0-9]{1,3}\.){3}[0-9]{1,3}(\/([0-9]|[1-2][0-9]|3[0-2]))?' | awk '{print $1}'

>Cool.  So Yahoo uses an HTML page that's a pain to process by
>computer.  Microsoft has  
>https://support.content.office.net/en-us/static/O365IPAddresses.xml,
>which at least is XML.  And Google, so far as I can see, can be mined by
>recursively expanding _spf.google.com.

Everyone else that I have needed to whitelist in postcreen with
postwhite will work fine by recursively expanding out their
TXT SPF record which is exactly what postwhite does.

The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Dianne Skoll]

On Mon, 30 Jan 2017 09:06:34 -0500
Rob McEwen  wrote:

> On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:
> > they do and it has been mentioned:
> > https://help.yahoo.com/kb/SLN23997.html

Cool.  So Yahoo uses an HTML page that's a pain to process by
computer.  Microsoft has 
https://support.content.office.net/en-us/static/O365IPAddresses.xml,
which at least is XML.  And Google, so far as I can see, can be mined by
recursively expanding _spf.google.com.

Yay standards...

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Mon, 30 Jan 2017 13:40:26 +
David Jones  wrote:

> My goal in whitelisting Yahoo servers is to make sure these
> messages get to MailScanner where they are not whitelisted
> and are scores based more on content by Spamassassin rather
> than sender reputation (DNSBLs).

OK, understood now.

I would always err on the side of more flexible filtering rather than
conserving server resources, and I'd use a filter flexible enough to a
avoid an RBL lookup on an SPF "pass" for yahoo.com.  But I understand
that others have different optimization goals.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Rob McEwen]

On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:

they do and it has been mentioned:
https://help.yahoo.com/kb/SLN23997.html


I wasn't aware of this page. If it was mentioned before in this thread, 
I missed it. Thanks!


--
Rob McEwen

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Mon, 30 Jan 2017 04:47:18 +0100
Reindl Harald  wrote:

> on postscreen level there is no SPF

And that's relevant... how?

You use a proper filter to do proper filtering.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

On Sat, 28 Jan 2017 16:33:24 +
David Jones  wrote:

[deleted]

Read back through this thread.  I never said their SPF record is
invalid. All I said is their SPF record is not common and it makes it
very hard for anyone to know what the official Yahoo outbound mail
servers are.


I have read this thread from start. You have said that their list is not
good, called that a lazy approach and called yahoo people incompetent, 
only because others are doing it other way.



On 1/29/2017 7:42 PM, Dianne Skoll wrote:

[deleted]

Can't you just whitelist the domain yahoo.com if
and only if it hits SPF "pass"?


not at postscreen level. postscreren is lightweight smtpd frontend for
postfix, designed to filter out bots/zombies - it can score DNSBL blacklists
and whitelists, temporarily blacklist hosts (similar to greylisting, but
only at source IP level) and the only way to avoid that is having the local
whitelist of cidr ranges.

The OP wants to get CIDR ranges of Yahoo to avoid potscreen checks and
blames Yahoo for not having the IP ranges in SPF records, because he uses SW
named postwhite that extracts such lists from SPF records of given domains,
and it can't be used with yahoo.com because of their SPF.

On 29.01.17 23:40, Rob McEwen wrote:
[deleted]
I know you mentioned that Yahoo may want to have the flexibility to 
change their IPs. But instead of providing a list, they could also 
provide a link to a web page listing the IPs (like what Comcast does) 
- and then just update that web page whenever their IPs change. This 
isn't rocket science.


they do and it has been mentioned:
https://help.yahoo.com/kb/SLN23997.html
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I'm not interested in your website anymore.
If you need cookies, bake them yourself.

Re: Legit Yahoo mail servers list

[David Jones]

>From: Rob McEwen 

>Sent: Sunday, January 29, 2017 10:40 PM

>On 1/29/2017 7:42 PM, Dianne Skoll wrote:
>> On Sat, 28 Jan 2017 16:33:24 +
>> David Jones  wrote:
>>
>>> Read back through this thread.  I never said their SPF record is
>>> invalid. All I said is their SPF record is not common and it makes it
>>> very hard for anyone to know what the official Yahoo outbound mail
>>> servers are.
>>
>> Why is that important?  Can't you just whitelist the domain yahoo.com if
>> and only if it hits SPF "pass"?

See next response below about the 2 different levels of MailScanner
checks.  Postfix postscreen is doing the majority of the DNSBL checks
and is not integrated with SPF checks.  It uses IPs or CIDRs.

>>
>>> We have to work very hard to get our MTAs to whitelist
>>> them.  It's in their own best interest to make this information
>>> easily available to the Internet since so much spam comes out of
>>> their platform.
>>
>> Then why would you whitelist them?
>>

Rob is correct below.  I do not have a complete whitelist of Yahoo
email.  Maybe the confusion is due to how MailScanner works.  As
I also said in this thread previously, MailScanner is not directly
tied to the MTA like amavis-new and others.  I have to whitelist
at the MTA level (Postfix/postscreen) to get past the first level
of checks primarily DNSBL related.  Then the second level is
MailScanner with Spamassassin plus some other unique checks.

My goal in whitelisting Yahoo servers is to make sure these
messages get to MailScanner where they are not whitelisted
and are scores based more on content by Spamassassin rather
than sender reputation (DNSBLs).


>Dianne,

>I can't speak for David, but most or all of your answers don't apply to
>my own anti-spam blacklist's attempt to try to avoid blacklisting Yahoo
>IPs that are both known for sending much spam, but which also would have
>a very high rate of collateral damage if blacklisted. (recognizing that
>some very good DNSBLs, which are more aggressive, are more willing to
>blacklist Yahoo IPs, and that isn't always a bad thing)

Exactly.  I would get too much collateral damage if I didn't whitelist
Yahoo IPs from DNSBL checks.  I have several dozen different DNSBLs
combined to do a very good job of blocking the junk before it has to
get to SA when you exclude Yahoo and other large hosting providers.

The best RBL by far is the Invaluement RBL feed that Rob runs.  Well
worth the low price.  It will save any sysadmin's time easily paying for
itself many times over.

>Also, when David said "whitelist", I can take an educated guess that he
>isn't allowing Yahoo-sent messages free unfiltered access to the inbox -
>he is probably just trying to avoid DNSBL checking of those particular
>IPs - but then he'll probably STILL do other content filtering of those
>messages. That would be my educated guess. And this would be a SMART
>strategy.

Yes.  It does work well.

Dave

Re: Legit Yahoo mail servers list

[Rob McEwen]

On 1/29/2017 7:42 PM, Dianne Skoll wrote:

On Sat, 28 Jan 2017 16:33:24 +
David Jones  wrote:


Read back through this thread.  I never said their SPF record is
invalid. All I said is their SPF record is not common and it makes it
very hard for anyone to know what the official Yahoo outbound mail
servers are.


Why is that important?  Can't you just whitelist the domain yahoo.com if
and only if it hits SPF "pass"?


We have to work very hard to get our MTAs to whitelist
them.  It's in their own best interest to make this information
easily available to the Internet since so much spam comes out of
their platform.


Then why would you whitelist them?


They are too large to not whitelist.


Nobody is too large to not whitelist.  They're obviously too large to
block, but you'd be foolish to accept any and all mail from a Yahoo
server unless you like an awful lot of spam.

Regards,

Dianne.




Dianne,

I can't speak for David, but most or all of your answers don't apply to 
my own anti-spam blacklist's attempt to try to avoid blacklisting Yahoo 
IPs that are both known for sending much spam, but which also would have 
a very high rate of collateral damage if blacklisted. (recognizing that 
some very good DNSBLs, which are more aggressive, are more willing to 
blacklist Yahoo IPs, and that isn't always a bad thing)


...and/or your answer requires more on-going receiver-side resources.

Interestingly, many senders would crawl over broken glass if necessary 
to provide me their IPs, if said I was seeking those for my whitelist.


Also, when David said "whitelist", I can take an educated guess that he 
isn't allowing Yahoo-sent messages free unfiltered access to the inbox - 
he is probably just trying to avoid DNSBL checking of those particular 
IPs - but then he'll probably STILL do other content filtering of those 
messages. That would be my educated guess. And this would be a SMART 
strategy.


Personally, when I get messages from Yahoo into my hosting business - I 
have the IPs generally not checked - since I already have most Yahoo IPs 
whitelisted - then I only content-check the messages - BUT... next I 
AMPLY any content scoring of such messages since these came from Yahoo 
and are more likely to be spam - that is, if the sender isn't already in 
a carefully cultivated exception list of known good Yahoo senders 
(specific to my mail hosting user base) - I do this for all freemail 
senders known to send a high volume of spam.


I know you mentioned that Yahoo may want to have the flexibility to 
change their IPs. But instead of providing a list, they could also 
provide a link to a web page listing the IPs (like what Comcast does) - 
and then just update that web page whenever their IPs change. This isn't 
rocket science.


As it stands, it is mind boggling just how many Yahoo ranges of sending 
IPs there are worldwide. Over the years, I've added 53 yahoo entries to 
my whitelist. Besides the hundreds and hundreds of /24s ranges in there 
(many are multiple consecutive /24s, showing up as just one line of 
those 53 entries), there are also several /16s, too. It would be nice to 
be able to compare that to Yahoo's current list active sending IPs (if 
such were available?), so that I could EFFICIENTLY update/prune that 
part of my whitelist.


And I strongly suspect that iterating though the millions of IPs to 
check FCrDNS would take a very, very long time - and might get such 
probing IPs blacklisted for abuse/intrusion-protection?


--
Rob McEwen

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Sat, 28 Jan 2017 16:33:24 +
David Jones  wrote:

> Read back through this thread.  I never said their SPF record is
> invalid. All I said is their SPF record is not common and it makes it
> very hard for anyone to know what the official Yahoo outbound mail
> servers are.

Why is that important?  Can't you just whitelist the domain yahoo.com if
and only if it hits SPF "pass"?

> We have to work very hard to get our MTAs to whitelist
> them.  It's in their own best interest to make this information
> easily available to the Internet since so much spam comes out of
> their platform.

Then why would you whitelist them?

> They are too large to not whitelist.

Nobody is too large to not whitelist.  They're obviously too large to
block, but you'd be foolish to accept any and all mail from a Yahoo
server unless you like an awful lot of spam.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[David Jones]

>From: Matus UHLAR - fantomas 

>>  Seems to me like
>>Yahoo doesn't have a good list of IPs so they took this shortcut
>>which is technically legitimate but it's making up for their incompetence
>>not having a handle on their mail flow.

>That doesn't mean incompetence. using PTR is official way, and while not so
>often used, it's perfectly valid.

>The whole fact you want their IP ranges does not mean they are not competent
>when they don't publish them in SPF.

>the fact that you can't whitelist their IPs based on their SPF record does
>not mean there's anything wrong with the SPF record itself...

Read back through this thread.  I never said their SPF record is invalid.
All I said is their SPF record is not common and it makes it very hard
for anyone to know what the official Yahoo outbound mail servers are.
We have to work very hard to get our MTAs to whitelist them.  It's in
their own best interest to make this information easily available to
the Internet since so much spam comes out of their platform.  They
are too large to not whitelist.  An SPF record is a useful way to build
such a whitelist when it can be parsed into IPs and CIDRs.  That is all.

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

From: Matus UHLAR - fantomas 
Still no practical difference between using IP ranges or rdns in SPF.


On 28.01.17 14:27, David Jones wrote:

Most SPF records published are not like this.


so... what?


 Seems to me like
Yahoo doesn't have a good list of IPs so they took this shortcut
which is technically legitimate but it's making up for their incompetence
not having a handle on their mail flow.


That doesn't mean incompetence. using PTR is official way, and while not so
often used, it's perfectly valid.

The whole fact you want their IP ranges does not mean they are not competent
when they don't publish them in SPF.

the fact that you can't whitelist their IPs based on their SPF record does
not mean there's anything wrong with the SPF record itself...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Linux - It's now safe to turn on your computer.
Linux - Teraz mozete pocitac bez obav zapnut.

Re: Legit Yahoo mail servers list

[David Jones]

>From: Matus UHLAR - fantomas 
>Still no practical difference between using IP ranges or rdns in SPF.

Most SPF records published are not like this.  Seems to me like
Yahoo doesn't have a good list of IPs so they took this shortcut
which is technically legitimate but it's making up for their incompetence
not having a handle on their mail flow.  They could have specific
mail ranges that all of their outbound mail comes from all over
the world and keep that pretty static so the rest of us could
properly whitelist their email.


>Well - if postscreen was able to use rdns, this discussion would be useless,
>since you'd whitelist .yahoo.com in postscreen, wouldn't you?

Ok.  Postscreen doesn't use RDNS.  I stand corrected.  I will try
to solve this problem in my Postfix settings since it does use
RDNS.  If not, I guess I will give up and continue to have the
occassional issue woth inbound mail from Yahoo.  I doesn't
happen often, I was just trying to handle Yahoo mail in MailScanner/
SpamAssassin compeltely by letting through the MTA.

Re: Legit Yahoo mail servers list

[David Jones]

Am 27.01.2017 um 17:57 schrieb David Jones:

>if you have trouble to get large providers past postscreen your rbl mix
>or scoring is just plain wrong

>configure postscreen proper and adjust RBL scores in spamassassin to get
>the rest killed, we are using the same DNSBL/DNSWL in postscreen and
>spamassassin starting 2014 and until now there hwere very few to zero
>complaints

I have pretty much the exact same postscreen as below.  It all depends on
senders to your recipients.  I may have a rural telecom company in
Wisconsin, USA that needs to send emai; to one of my recipients that is
listed on a couple of RBLs below that your mail server doesn't need to
receive.  I just ran into this yesterday.

>postscreen_dnsbl_threshold = 8
>postscreen_dnsbl_action = enforce
>postscreen_greet_action = enforsce
>postscreen_greet_wait = ${stress?3}${stress:10}s
>postscreen_dnsbl_sites =
>  dnsbl.sorbs.net=127.0.0.10*9
>  dnsbl.sorbs.net=127.0.0.14*9
>  zen.spamhaus.org=127.0.0.[10;11]*8
>  dnsbl.sorbs.net=127.0.0.5*7
>  zen.spamhaus.org=127.0.0.[4..7]*7
>  b.barracudacentral.org=127.0.0.2*7
>  zen.spamhaus.org=127.0.0.3*7
>  dnsbl.inps.de=127.0.0.2*7
>  hostkarma.junkemailfilter.com=127.0.0.2*4
>  dnsbl.sorbs.net=127.0.0.7*4
>  bl.spamcop.net=127.0.0.2*4
>  bl.spameatingmonkey.net=127.0.0.[2;3]*4
>  dnsrbl.swinog.ch=127.0.0.3*4
>  ix.dnsbl.manitu.net=127.0.0.2*4
>  psbl.surriel.com=127.0.0.2*4
>  bl.mailspike.net=127.0.0.[10;11;12]*4
>  bl.mailspike.net=127.0.0.2*4
>  zen.spamhaus.org=127.0.0.2*3
>  score.senderscore.com=127.0.4.[0..20]*3
>  bl.spamcannibal.org=127.0.0.2*3
>  dnsbl.sorbs.net=127.0.0.6*3
>  dnsbl.sorbs.net=127.0.0.8*2
>  hostkarma.junkemailfilter.com=127.0.0.4*2
>  dnsbl.sorbs.net=127.0.0.9*2
>  dnsbl-1.uceprotect.net=127.0.0.2*2
>  all.spamrats.com=127.0.0.38*2
>  bl.nszones.com=127.0.0.[2;3]*1
>  dnsbl-2.uceprotect.net=127.0.0.2*1
>  dnsbl.sorbs.net=127.0.0.2*1
>  dnsbl.sorbs.net=127.0.0.4*1
>  score.senderscore.com=127.0.4.[0..69]*1
>  dnsbl.sorbs.net=127.0.0.3*1
>  hostkarma.junkemailfilter.com=127.0.1.2*1
>  dnsbl.sorbs.net=127.0.0.15*1
>  ips.backscatterer.org=127.0.0.2*1
>  bl.nszones.com=127.0.0.5*-1
>  score.senderscore.com=127.0.4.[90..100]*-1
>  wl.mailspike.net=127.0.0.[18;19;20]*-2
>  hostkarma.junkemailfilter.com=127.0.0.1*-2
>  ips.whitelisted.org=127.0.0.2*-2
>  list.dnswl.org=127.0.[0..255].0*-2
>  dnswl.inps.de=127.0.[0;1].[2..10]*-2
>  list.dnswl.org=127.0.[0..255].1*-3
>  list.dnswl.org=127.0.[0..255].2*-4
>  list.dnswl.org=127.0.[0..255].3*-5

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Fri, 27 Jan 2017 22:23:55 +0100
Benny Pedersen  wrote:

> with use of PTR its always up2date, problem is just that none spf 
> testers are doing FcRDNS checked before saying spf pass

Unlikely.  The SPF spec says that you must do that, and most SPF libraries
probably do the checks.  I use Perl's Net::SPF module and it checks FcRDNS.

> PTR on its own is by defination same as +all

Not true; see above and the spec: http://www.openspf.org/SPF_Record_Syntax#ptr

-- Dianne.

Re: Legit Yahoo mail servers list

[RW]

On Fri, 27 Jan 2017 22:23:55 +0100
Benny Pedersen wrote:

> Dianne Skoll skrev den 2017-01-27 19:02:
> > On Fri, 27 Jan 2017 12:40:16 -0500
> > Rob McEwen  wrote:
> >   
> >> While I have Yahoo sending IPs extensively covered in my whitelist,
> >> I've been trying to get their complete official list of sending IPs
> >> for years.  
> > 
> > Yahoo might want the flexibility to change this list on a regular
> > basis.  
> 
> with use of PTR its always up2date, problem is just that none spf 
> testers are doing FcRDNS checked before saying spf pass

The RFC requires that they do.

Re: Legit Yahoo mail servers list

[Benny Pedersen]

Dianne Skoll skrev den 2017-01-27 19:02:

On Fri, 27 Jan 2017 12:40:16 -0500
Rob McEwen  wrote:


While I have Yahoo sending IPs extensively covered in my whitelist,
I've been trying to get their complete official list of sending IPs
for years.


Yahoo might want the flexibility to change this list on a regular
basis.


with use of PTR its always up2date, problem is just that none spf 
testers are doing FcRDNS checked before saying spf pass


PTR on its own is by defination same as +all

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

the SPF record can change too, so that makes no difference.


On 27.01.17 16:57, David Jones wrote:

We have to assume that a competent mail sysadmin would
make that SPF record change.  It has to be trusted since that's
the whole point of SPF.


The easy workaround is to put ptr: into the SPF record, which is clearly
what yahoo did.  Then it's enough to maintain servers' fcrdns - no
incompetence here.

however, in both cases, some IPs can be added to, as well as removed from
pool. That means, one should do the comparison at time mail is received, not
far later (because the information might be obsolete at later time).

Still no practical difference between using IP ranges or rdns in SPF.


I get it as you need parse mail logs to find out what to put into
postscreen list, since postscreen doesn't use rdns...


Hmm, are you sure about that?


I have checked (just for sure) before sending my email.

what exactly did you mean when talking about log parsing, if not this?

Well - if postscreen was able to use rdns, this discussion would be useless,
since you'd whitelist .yahoo.com in postscreen, wouldn't you?


and postwhite (https://github.com/stevejenkins/postwhite) script can only
parse SPF records, not logs. Luckily ita page shows something that can help
you with yahoo:
https://help.yahoo.com/kb/SLN23997.html


Cool.  Thank you.  This is what I was looking for.

I think I have this solved in Postfix based on FCrDNS but
it good to know that Steve Jenkins is working on the same
thing.


postfix' smtpd can do rdns (and whitelist based on it). postscreen can't.
you mentioned postscren (and postwhite, which is whitelisting for
postscreen), I don't get why you mix postfix here...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson.  -- Daffy Duck & Porky Pig

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Fri, 27 Jan 2017 12:40:16 -0500
Rob McEwen  wrote:

> While I have Yahoo sending IPs extensively covered in my whitelist,
> I've been trying to get their complete official list of sending IPs
> for years.

Yahoo might want the flexibility to change this list on a regular
basis.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Rob McEwen]

While I have Yahoo sending IPs extensively covered in my whitelist, I've 
been trying to get their complete official list of sending IPs for years.


I'm amazed that Yahoo doesn't participate in these conversations - or 
that nobody ever says, "I'll ask my colleague over at Yahoo"


seems very odd...

--
Rob McEwen

Re: Legit Yahoo mail servers list

[David Jones]

>the SPF record can change too, so that makes no difference.

We have to assume that a competent mail sysadmin would 
make that SPF record change.  It has to be trusted since that's
the whole point of SPF.

>MailScanner can still (and its SA plugin will) use the results described
>above.

I know that but I have to get the message past Postfix/postscreen
so it can make it to SA.

>I get it as you need parse mail logs to find out what to put into
>postscreen list, since postscreen doesn't use rdns...

Hmm, are you sure about that?

>and postwhite (https://github.com/stevejenkins/postwhite) script can only
>parse SPF records, not logs. Luckily ita page shows something that can help
>you with yahoo:
>https://help.yahoo.com/kb/SLN23997.html

Cool.  Thank you.  This is what I was looking for.

I think I have this solved in Postfix based on FCrDNS but
it good to know that Steve Jenkins is working on the same
thing.

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

On 26.01.17 19:53, David Jones wrote:

Their SPF record can really only be evaluated by the MTA during
the SMTP conversation.



From: Matus UHLAR - fantomas 
SPF records can be perfectly parser by SA or other software at
different time.


  On 27.01.17 12:43, David Jones wrote:

I think you misunderstood.  PTR records don't change often but
they could.  Their matching A records for FCrDNS could change
too so you can't rely on later processing to know what happened
when that message arrived.


the SPF record can change too, so that makes no difference.

The best we can do here is to put sending host's fcrdns into headers,
probably together with Received-SPF: header, so spam filter will process
there.

Luckily most MTAs do the first, unless you turn off DNS check at SMTP time.


The main problem with parsing mail logs is the chicken-and-the-egg
issue where you may block a Yahoo mail server with an RBL for a
short period until you process the logs.



what informations do you search in logs that are not in mail headers?


I use MailScanner which is not a milter or otherwise directly part of the
MTA (Postfix in my setup).  This basically creates 2 levels of filtering:
the MTA and MailScanner (SpamAssassin plus many other checks).
My RBLs are done by postscreen (really awesome, everyone should
use it) so I have to allow Yahoo mail servers in the first level of filtering
independent of SA.


MailScanner can still (and its SA plugin will) use the results described
above.

I get it as you need parse mail logs to find out what to put into
postscreen list, since postscreen doesn't use rdns...

and postwhite (https://github.com/stevejenkins/postwhite) script can only
parse SPF records, not logs. Luckily ita page shows something that can help
you with yahoo:
https://help.yahoo.com/kb/SLN23997.html


I think I have solved this issue.  Postfix smtpd_client_restrictions
check_client_access does use FCrDNS for domains listed. I will
watch my logs for a few days and make sure this is working properly.


unluckily this is not something for postscreen...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.

Re: Legit Yahoo mail servers list

[David Jones]

>From: Matus UHLAR - fantomas 
>Sent: Thursday, January 26, 2017 2:15 PM
    
>On 26.01.17 19:53, David Jones wrote:
>>I  understand what their SPF record means and how it works
>>but what they are publishing in their SPF record is not common.
>>Normally this would expand out to a list of IPs and CIDRs or DNS
>>records that can be turned into IPs that postwhite can use to build
>>a list for bypassing RBL checks.

>SPF was never designed to create such lists. They can get easily obsolete,
>miss some IPs and/or have some IPS that don't belong there.

I agree. But it turns out it works pretty well since SPF has been taken
more seriously the past couple of years.  When Gmail and others
started putting SPF failed messages into the Junk folder, it's starting
to be worth something.

I am only doing postwhite exclusions for 2 types of senders:
1. Large mail hosting providers that are too large to block and don't
keep their mail server IPs off of RBLs
2. Highly trusted senders that know what they are doing and keep
their SPF record properly maintained that would already score very
low in SA.

>>Their SPF record can really only be evaluated by the MTA during
>>the SMTP conversation. 

>SPF records can be perfectly parser by SA or other software at
>different time.

I think you misunderstood.  PTR records don't change often but
they could.  Their matching A records for FCrDNS could change
too so you can't rely on later processing to know what happened
when that message arrived.

>>The main problem with parsing mail logs is the chicken-and-the-egg
>>issue where you may block a Yahoo mail server with an RBL for a
>>short period until you process the logs.

>what informations do you search in logs that are not in mail headers?

I use MailScanner which is not a milter or otherwise directly part of the
MTA (Postfix in my setup).  This basically creates 2 levels of filtering:
the MTA and MailScanner (SpamAssassin plus many other checks).
My RBLs are done by postscreen (really awesome, everyone should
use it) so I have to allow Yahoo mail servers in the first level of filtering
independent of SA.

>>I think they publish their SPF like this because they have no good
>>list of outbound mail servers themselves so they take the lazy
>>approach.

>I believe that ptr method is one of best methods to implement in spf,
>contrary what the authors say. (I believe) Most of MTAs verify fcrdns of 
>connecting
>server so all required information are available in DNS cache at the time of
>SPF processing.

I think I have solved this issue.  Postfix smtpd_client_restrictions
check_client_access does use FCrDNS for domains listed. I will
watch my logs for a few days and make sure this is working properly.

Re: Legit Yahoo mail servers list

[Michael Orlitzky]

On 01/26/2017 02:53 PM, David Jones wrote:
> 
> I  understand what their SPF record means and how it works
> but what they are publishing in their SPF record is not common.
> Normally this would expand out to a list of IPs and CIDRs or DNS
> records that can be turned into IPs that postwhite can use to build
> a list for bypassing RBL checks.
> 

Are the problematic RBL checks performed by Postfix, or by SpamAssassin?

The possibilities for whitelisting in SpamAssassin are a lot more
flexible, so if I were you, I would tweak postscreen (or my smtpd
restrictions) to the point where it causes no false positives. Then
SpamAssassin can be configured to do the same level of RBL checks that
are occasionally causing false positives now. The double lookups aren't
expensive because they're cached locally. And the false positives are
easy to deal with in SA, where for example you have access to the result
of SPF.

If you can get it to the point where SA is the one blocking Yahoo, then
all you have to do is add a meta rule that subtracts a few points when
the sender's domain belongs to Yahoo and the SPF_PASS rule hits.

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

On 26.01.17 19:53, David Jones wrote:

I  understand what their SPF record means and how it works
but what they are publishing in their SPF record is not common.
Normally this would expand out to a list of IPs and CIDRs or DNS
records that can be turned into IPs that postwhite can use to build
a list for bypassing RBL checks.


SPF was never designed to create such lists. They can get easily obsolete,
miss some IPs and/or have some IPS that don't belong there.


Their SPF record can really only be evaluated by the MTA during
the SMTP conversation. 


SPF records can be perfectly parser by SA or other software at
different time.


The main problem with parsing mail logs is the chicken-and-the-egg
issue where you may block a Yahoo mail server with an RBL for a
short period until you process the logs.


what informations do you search in logs that are not in mail headers?


I think they publish their SPF like this because they have no good
list of outbound mail servers themselves so they take the lazy
approach.


I believe that ptr method is one of best methods to implement in spf,
contrary what the authors say. (I believe) Most of MTAs verify fcrdns of 
connecting
server so all required information are available in DNS cache at the time of
SPF processing.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Fucking windows! Bring Bill Gates! (Southpark the movie)

Re: Legit Yahoo mail servers list

[Dianne Skoll]

Following up on myself:

> IMO, the SPF spec should have specified that a PTR mechanism MUST be
> ignored nuless FCrDNS matches.  (Maybe it does... too lazy to look it
> up. :))

Indeed, the SPF spec does say this.  So a PTR mechanism isn't completely
useless after all.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Thu, 26 Jan 2017 19:53:42 +
David Jones  wrote:

> I think they publish their SPF like this because they have no good
> list of outbound mail servers themselves so they take the lazy
> approach.

Yahoo invented (or was one of the main inventors) of DKIM, so it could
also be a little bit of NIH syndrome.  And maybe they published nonsensical
SPF records to placate badly-designed receiving systems that penalize
domains with no SPF whatsoever.

> Postfix is pretty flexible so maybe there is a way to allow this
> by the PTR when the FCrDNS matches.

IMO, the SPF spec should have specified that a PTR mechanism MUST be
ignored nuless FCrDNS matches.  (Maybe it does... too lazy to look it
up. :))

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[David Jones]

>On 01/26/2017 01:29 PM, Reindl Harald wrote:
>> 
>> SPF_NEUTRAL will NEVER hit SPF_PASS and that's the problem with ?all
>> 
>SPF mechanisms are evaluated in order, and each one has a result type
>associated with it. The default result is "+" for "pass". Another type
>of result is "?" for "neutral."

>The record,

>  v=spf1 ptr:yahoo.com ptr:yahoo.net ?all

>is equivalent to

>  v=spf1 +ptr:yahoo.com +ptr:yahoo.net ?all

>and it means

>  a) PASS if "ptr:yahoo.com" matches
>  b) PASS if "ptr:yahoo.net" matches
>  c) NEUTRAL if "all" matches

I  understand what their SPF record means and how it works
but what they are publishing in their SPF record is not common.
Normally this would expand out to a list of IPs and CIDRs or DNS
records that can be turned into IPs that postwhite can use to build
a list for bypassing RBL checks.

Their SPF record can really only be evaluated by the MTA during
the SMTP conversation.  This would require some mail log parsing
to extract out IPs that have already been seen by your mail server
and not be able to be determined in advance.  This would be better
than nothing but is not ideal.

The main problem with parsing mail logs is the chicken-and-the-egg
issue where you may block a Yahoo mail server with an RBL for a
short period until you process the logs.

I think they publish their SPF like this because they have no good
list of outbound mail servers themselves so they take the lazy
approach.

Postfix is pretty flexible so maybe there is a way to allow this
by the PTR when the FCrDNS matches.  You wouldn't want to
rely on just the PTR record alone since that can be easily spoofed
by a spammer with control of their reverse DNS zone for their IPs.
FCrDNS would make that very difficult to spoof and I am pretty
sure this is the only way Postfix would allow it to pass it's check.

Re: Legit Yahoo mail servers list

[Michael Orlitzky]

On 01/26/2017 01:29 PM, Reindl Harald wrote:
> 
> SPF_NEUTRAL will NEVER hit SPF_PASS and that's the problem with ?all
> 

SPF mechanisms are evaluated in order, and each one has a result type
associated with it. The default result is "+" for "pass". Another type
of result is "?" for "neutral."

The record,

  v=spf1 ptr:yahoo.com ptr:yahoo.net ?all

is equivalent to

  v=spf1 +ptr:yahoo.com +ptr:yahoo.net ?all

and it means

  a) PASS if "ptr:yahoo.com" matches
  b) PASS if "ptr:yahoo.net" matches
  c) NEUTRAL if "all" matches

Re: Legit Yahoo mail servers list

[Benny Pedersen]

Michael Orlitzky skrev den 2017-01-26 19:24:


The OP is looking for a way to whitelist so the "?all" is irrelevant.
Does the sending IP pass the SPF check? If so, whitelist it.


PTR in spf is very hard to forge

treat it as ip4:0.0.0.0/0 -all

yahoo do not want to reject based on spf, but still provide a badly spf 
to follow ignorants

Re: Legit Yahoo mail servers list

[Michael Orlitzky]

On 01/26/2017 12:59 PM, Reindl Harald wrote:
> 
> 
> Am 26.01.2017 um 18:51 schrieb Michael Orlitzky:
>> On 01/26/2017 12:22 PM, David Jones wrote:
>>> ...
>>> They don't publish a good SPF record so I am not able to add
>>> them to my postwhite list.
>>>
>>
>> Isn't that what their SPF record does?
> 
> did you notice the "?all"
> re-read your spf manuals
> 

The OP is looking for a way to whitelist so the "?all" is irrelevant.
Does the sending IP pass the SPF check? If so, whitelist it.

Re: Legit Yahoo mail servers list

[Michael Orlitzky]

On 01/26/2017 12:22 PM, David Jones wrote:
> Anyone know how to get a list of legit mail servers for Yahoo?
> They don't publish a good SPF record so I am not able to add
> them to my postwhite list.
> 
> # dig yahoo.com txt +short
> "v=spf1 redirect=_spf.mail.yahoo.com"
> # dig _spf.mail.yahoo.com txt +short
> "v=spf1 ptr:yahoo.com ptr:yahoo.net ?all"
> 
> The only way I can think of even coming close is to analyse
> my mail logs for clean mail IPs with PTR values ending in
> yahoo.com and yahoo.net. 

Isn't that what their SPF record does?

Legit Yahoo mail servers list

[David Jones]

Anyone know how to get a list of legit mail servers for Yahoo?
They don't publish a good SPF record so I am not able to add
them to my postwhite list.

# dig yahoo.com txt +short
"v=spf1 redirect=_spf.mail.yahoo.com"
# dig _spf.mail.yahoo.com txt +short
"v=spf1 ptr:yahoo.com ptr:yahoo.net ?all"

The only way I can think of even coming close is to analyse
my mail logs for clean mail IPs with PTR values ending in
yahoo.com and yahoo.net.  This of course will not be a
complete list and always be behind as Yahoo adds/changes
their outbound mail servers.

I don't get many complaints about blocking inbound mail,
but most of them are related to yahoo.com and other Yahoo
domains because I don't have a good way to bypass RBLs for
their IPs like I do for Google, Microsoft, Comcast, AOL, and
other free mail hosting providers.

Dave