http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html
SPF is harmful. Adopt it.
You've come to this page because you've said something similar to the
following:
SPF ("sender permitted from" a.k.a. "sender policy framework") is a
scheme designed to prevent forgery of SMTP-based Internet mail and
thus prevent unsolicited bulk mail ("spam"). AOL has already adopted
it.
This is the /Frequently Given Answer/ to such statements.
(You can find different approaches to this answer on John Levine's web
page <http://www.taugh.com./mp/lmap.html>, in an article by Steven M.
Bellovin
<news://news.gmane.org./[EMAIL PROTECTED]>,
and on Brad Knowles' web page
<http://bradknowles.typepad.com./considered_harmful/2004/05/spf.html>.
By the way, whilst he agrees with what is said here about DNS security,
take Brad Knowles' DNS server comparison, that he then refers to, with a
large sackful of salt <http://cr.yp.to/djbdns/knowles.html>.)
SPF is harmful. The architectural ramifications of it are so extensive
and will have such significant changes on the ways that people can
access and can use Internet mail, that it would actually be /less/
costly to switch to an entirely new architecture such as IM2000 Internet
mail <http://homepages.tesco.net/J.deBoynePollard/Proposals/IM2000/>
than it would be to switch to SPF and deal with all of its consequences
properly.
Many of those architectural ramifications have either been incompletely
addressed or not addressed at all as yet. Moreover, SPF usurps the
meaning of an existing and widely used DNS resource record type for its
own purposes, and has not yet been assigned its own actual resource
record type. Anyone adopting SPF right now (which means /actually/
adopting it, rather than merely paying it lip service
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#LipService>)
is adopting a scheme that can at best be described as woefully incomplete.
Most people who have analysed SPF in detail have come to the conclusion
that it is a deeply flawed scheme that should be avoided outright.
On the gripping hand, maybe the fact that SPF is so damaging to the
SMTP-based Internet mail architecture is a good thing. In the battle
against unsolicited bulk mail, we have concentrated upon the wrong
problem time after time
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html>,
with mechanisms that address the wrong thing and that don't address the
actual "unsolicited" and "bulk" qualities of undesirable mail. SMTP has
become less usable, more patchy, and more balkanised with each new
bodge. Perhaps the adoption of SPF will turn out to be the straw that
finally breaks the camel's back, and that thus finally forcibly weans us
off this bad habit of addressing the wrong problem.
So perhaps SPF is a deeply flawed scheme that should be adopted.
Ironically: SPF is also a good counter to one objection to IM2000
Internet mail
<http://homepages.tesco.net/J.deBoynePollard/Proposals/IM2000/>, namely
that it involves changing the structure of the mail system. If people
sending mail and mail hosting companies are clearly willing to accept
the massive structural changes that SPF will entail, they will be
willing to accept the smaller structural changes that IM2000 Internet
mail will entail.
Paying SPF lip service
10,000 domains cannot be wrong! claim the SPF marketing people. (That is
actually quite a small number, of course. To gain perspective, note that
in 2002-12-17 there were 22,009,173 domains /under com. alone/.) But
publishing SPF data (aside from further entrenching the unauthorised
hijacking by SPF of an existing DNS resource record type
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#HijackTXTResourceRecordType>)
is merely the paying of lip service to SPF.
And, ironically, research has shown that most of the domains that /are/
publishing such SPF data are owned by UBM senders, proving, as if any
more proof were needed, that we are still dancing the same old foolish
dance of concentrating upon the wrong problem
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html>,
changing something that is /not/ directly related to "unsolicited" and
"bulk" and seeing the unsolicited bulk mail change to match.
To properly adopt SPF, rather than just to pay it lip service, it is
also necessary to configure one's SMTP Relay server to look up and to
process the SPF data for all mail received.
The adoption rate of those who are actually adopting SPF properly, and
not just paying it lip service, is rather more difficult to measure. The
UBM senders publishing SPF data probably haven't adopted SPF fully, of
course. Interestingly, though, many proponents of SPF have fallen silent
when asked whether, in addition to paying lip service to SPF, they have
also configured their SMTP Relay servers to check SPF data, leading to
the conclusion that it is quite probable that many of the SPF proponents
in those 10,000 domains are paying mere lip service to SPF too.
Some of the flaws in SPF
The flaws in SPF are numerous and severalfold.
* SPF breaks pre-delivery forwarding.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#PreDeliveryForwarding>
* SPF hijacks existing DNS mechanisms.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#HijackTXTResourceRecordType>
* SPF gives ISPs a "lock-in" weapon against their customers.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#ISPLockIn>
* SPF is useless for several entire classes of people.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#Uselessness>
* SPF relies upon DNS for security, but DNS isn't a security
service.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#DatabaseSecurity>
* SPF is vulnerable to race conditions during database changes.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#DatabaseUpdateRace>
* SPF creates new categories of third class citizenship.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#ThirdClassCitizens>
* SPF doesn't actually address unsolicited bulk mail at all.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#WrongProblem>
* SPF hands Verisign its next unwelcome "innovation" on a platter.
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-spf-is-harmful.html#Verisign>
SPF breaks pre-delivery forwarding.
SMTP-based Internet mail is, by design, a "store and forward"
architecture. Mail is transported from machine to machine, stored and
then forwarded on, until it finally reaches its destination.
SPF runs fundamentally counter to this architecture. The primary design
of SPF is that the owner of a domain name publishes a list of SMTP Relay
client machines associated with that domain, and that SMTP Relay servers
reject mail with any such envelope sender mailbox names that does not
come from an SMTP Relay client that is on that list. The architecture
that SPF thus actually mandates is one where mail is delivered directly
from originator to recipient, with no "store and forward" hops in between.
This is directly at loggerheads with the "store and forward" nature of
SMTP-based Internet mail.
SPF is at loggerheads with RFC 1123
RFC 1123 describes simple mail exploders, aliases where the mail system
replaces a particular envelope recipient with zero or more other
envelope recipients and then forwards the message on to those new
recipients. Many MTSes have such features. (For examples: This is the
operation of forwarding instructions in .qmail files and of /etc/aliases
in Sendmail.) Many people employ such features, moreover. (For examples:
Companies will have "system-wide" alias mailboxes such as
all-sales-representatives. Users will choose to forward all mail from
one of their mailboxes to another.)
SPF breaks this. Simple mail exploders don't change the envelope sender.
(Indeed, in the case of system-wide aliases, there's no user account
under whose aegis the forwarding can be said to have occurred, and so no
reasonable value to change the envelope sender to.) By declaring that
mail with a particular envelope sender can /only/ legitimately come from
SMTP Relay clients that the domain name owner designates, SPF prevents
the recipients of mail message from forwarding them on to further
mailboxes, in the very "store and forward" manner that the SMTP-based
Internet mail system is designed to do.
In essence, SPF allows domain name owners to impose an unwaranted degree
of control over what recipients can do /with their own mail/,
eliminating outright a common mechanism that recipients have heretofore
had and have widely employed.
SPF is purported to come with a scheme called "SRS" that supposedly
fixes this problem. But it doesn't. SRS is not even fully thought
through, yet. It has massive problems of its own:
*
SRS is in conflict with existing systems that store information in
the envelope sender mailbox name, such as VERPs.
*
SRS is a system that, after even just two levels of forwarding,
causes envelope sender mailboxes to become so long that they run
the risk of hitting mailbox name length limits in mail softwares.
Example: The SRS1 template for envelope sender mailboxes is
[EMAIL PROTECTED]
Plugging a real-world mailbox name and two SPF supporters into
that template, and using conservative lengths for the hashes
and timestamp (Note that the timestamps already in use
elsewhere in electronic mail, such as those in Message IDs,
are usually longer than this.), yields
[EMAIL PROTECTED]
That's 66 characters, 2 characters more than the 64 character
length limit on mailbox name local parts in SMTP that RFC 2821
§ 4.5.3.1 specifies. (And we didn't even choose
PhilZimmermann.com.)
The section of the SRS specification entitled "A study on the
64 character limit" is empty. This is probably why.
Of course, SMTP already has a mechanism for incorporating
intermediate hop information into mailbox names, the source
route mechanism described in RFC 821 § 3.6. Converting the
above mailbox name to a (hypothetical) SRS1 form that uses RFC
821 source routing yields
@earthlink.net,@brightmail.com:[EMAIL PROTECTED],
which at 41 characters doesn't exceed the RFC 2821 § 4.5.3.1
limit. Ironically, AOL, proponent of SPF, worked hard to
/abolish/ all use of this very mechanism with one of its
previous half-baked attempts to combat UBM
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html#SourceRouting>.
*
SRS creates the possibility of attackers forging "bounce"
messages, reintroducing one of the very things that SPF is touted
to (but, ironically, doesn't actually in any case) prevent.
The only way to "fix" this problem with SPF is, as some of its
proponents have asserted, to simply outlaw pre-delivery forwarding /in
toto/ and to presumptively declare the fact that SPF breaks it therefore
to be a non-problem. According to those proponents, pre-delivery
forwarding is simply wrong in concept and /all/ forwarding must be
post-delivery, i.e. performed by an MUA re-injecting a delivered message
into the system as a new message with a new envelope declaring it to be
from the forwarder.
The outright elimination of pre-delivery forwarding is, of course, a
major change to the way that people use the mail system.
SPF is at loggerheads with RFC 974 and RFC 2821
One of the aspects of the "store and forward" design of SMTP-based
Internet mail is that of fallback mail exchangers. SPF stops fallback
mail exchangers from working.
By declaring that mail with a particular envelope sender can /only/
legitimately come from SMTP Relay clients that the domain name owner
designates, SPF prevents fallback mail exchangers from forwarding
messages on to primary mail exchangers, because of course the domain
name owners of all of the various envelope sender mailboxes haven't
designated the SMTP Relay clients on the fallback mail exchanger as
being legitimate sources of such mail.
Having every domain name owner list the SMTP Relay client sides of every
fallback mail exchanger in the world in xyr SPF data is of course a
completely unfeasible solution to this problem, because of scalability
problems alone.
So therefore every SMTP Relay server that adopts SPF has to disable its
use for (the SMTP Relay client sides of) every fallback mail exchanger
for every domain that it accepts mail for. The consequences of this are
at the very least twofold:
*
There's a greater degree of coupling between fallback mail
exchangers and primary mail exchangers with SPF than there is
without SPF. With SPF, every company that provides fallback mail
exchanger service has to inform its customers, so that they can
reconfigure their exceptions to SPF, every time that the company
moves the SMTP Relay client side of its fallback mail exchanger.
Without SPF, companies that provide fallback mail exchanger
service don't have to tell their customers where their SMTP Relay
clients are.
*
All fallback mail exchangers become holes in the system. Every
fallback mail exchanger is a potential source of mail that has not
passed through SPF's (erroneous and ill-conceived) checks on
envelope sender mailbox names.
For example: If an ISP provides fallback mail exchanger service to
one set of customers and /also/ provides SMTP Submission service
to another set of customers via the same system (a not uncommon
state of affairs), then all mail submitted by any of the second
set of customers via the SMTP Submission service can bypass all
SPF checks when addressed to any of the first set of customers who
have bought fallback mail exchanger service. To avoid this hole,
ISPs have to run /entirely separate/ systems for SMTP Relay and
SMTP Submission. Again, SPF changes the current architecture of
SMTP-based Internet mail.
The same problem applies to organisations where the SMTP Relay server
seen by the outside world is not the final mailhost, but instead mail is
relayed internally across the organisation's own network. Again, the use
of SPF validation on the parts of the internal SMTP Relay servers has to
be disabled, yielding holes in the system, and the same consequences occur.
SPF hijacks existing DNS mechanisms.
Rather than creating a new DNS resource record type for the new data to
be published in the public DNS database, SPF unilaterally redefines the
meaning of an existing DNS resource record type, TXT, for its own
purposes. This creates a conflict between SPF and all existing uses of
TXT resource records.
SPF gives ISPs a "lock-in" weapon against their customers.
In the SMTP-based Internet mail system, the nature of the system allows
people to "roam". People may employ /their own/ mailbox names as the
envelope sender, whatever place they may be submitting the mail into the
system at. For examples:
*
A person who is a guest user on another system may employ their
own mailbox name as the envelope sender on mail that they send
from that system (employing the ${QMAILSHOST} mechanism in qmail,
for example).
*
A person may own a mailbox name, or even an entire domain, and may
submit mail directly, connecting to Internet via various ISPs.
*
A person may have bought the services of multiple ISPs, each
supplying the user with a mailbox name, and may submit mail via
any of those ISPs, or directly, using any of those mailbox names
as the envelope sender.
SPF breaks this, providing ISPs (that provide mail hosting services to
their customers) with a draconian lock-in weapon to wield against their
customers. An ISP can employ SPF to prevent its customers from
submitting mail, using /those customers' own/ mailbox names as the
envelope senders, from elsewhere other than the ISP itself, preventing
the customer from submitting mail directly, via another ISP, or as a
guest of another system.
SPF is useless for several entire classes of people.
Even leaving aside the desirability of doing so, there isn't even a way
with SPF to publish reliable or meaningful SPF data for entire classes
of people.
SPF is useless for SMTP Relay clients with dynamically-assigned
IP addresses.
Whilst it is impractical and a bad idea (because it allows mail to be
intercepted or to be lost) for an SMTP Relay /server/ to listen on a
dynamically assigned IP address, there is no similar prohibition on SMTP
Relay /clients/ connecting from dynamically assigned IP addresses.
However, SPF is useless for people with such SMTP Relay clients.
Supposedly, the SPF "ptr" mechanism is there for dealing with publishing
data about such SMTP Relay clients. However, this is just the flawed
"double-reverse lookup" mechanism
<http://homepages.tesco.net/J.deBoynePollard/FGA/dns-avoid-double-reverse.html>
in a paper-thin disguise. It's a half-baked idea that is conceptually
flawed and that doesn't actually work.
SPF is useless for roaming SMTP Relay clients.
A person may own an entire domain and wish to submit mail, using mailbox
names in xyr own domain as the envelope senders, directly from a system
with a dynamically-assigned IP address (such as, for example, a portable
system that connects via multiple ISPs).
Ironically, the official SPF answer to the question of what SPF data to
publish in these circumstances is, effectively, to /not adopt SPF/:
If you run a personal domain, you can either not publish SPF records
at all, or set up |"v=spf1 +all"| for your domain, and you'll be able
to send mail from your laptop no matter where you are.
SPF relies upon DNS for security, but DNS isn't a security service.
SPF places far too much trust in the DNS. DNS lookups are liable to
attack by spoofed responses. (Many people are surprised when they learn
how easy this is <http://cr.yp.to/djbdns/forgery.html>.)
Moreover, even aside from the potential for spoofing, by relying upon
the DNS for SMTP Relay client validation information SPF relies upon
attacker-supplied data, a foolish design for any sort of validation
system. It's trivial for malicious senders to create throwaway domain
names with published DNS data, that they supply, declaring any SMTP
Relay clients that they like to be (as far as SPF is concerned)
"legitimate". And there's no shortage of such domain names to use once
and then throw away.
SPF is vulnerable to race conditions during database changes.
DNS is a distributed database. Resource record sets are cached in proxy
DNS servers
<http://homepages.tesco.net/J.deBoynePollard/FGA/dns-server-roles.html#Proxy>
around the world. Changing information published in the DNS database
involves careful management, in the shape of employing the "time to die"
features of content DNS server softwares, if one is to avoid the case
where some proxy DNS servers will have old information and others will
have new information. Some content DNS server softwares simply do not
have such features, and the case is thus unavoidable.
This means that there is a race condition whenever the (SPF-defined)
"legitimacy" of an SMTP Relay client is revoked by changing the
published DNS data. During the TTL period of the old TXT resource record
set, some areas of the world will still consider that SMTP Relay client
to be (what SPF defines to be) "legitimate".
SPF creates new categories of third class citizenship.
It's ironic, given its prior history of bodges to SMTP
<http://homepages.tesco.net/J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html#SourceRouting>
that AOL is presented as the champion of SPF.
AOL is already famous for dividing the Internet up into three classes of
citizens, placing the customers of all other ISPs into the third class
<http://homepages.tesco.net/J.deBoynePollard/FGA/maps-dul-is-wrong.html>
and refusing to have any dealings with them for mail service. SPF is
just more of the same sort of discrimination. With SPF, the third class
of citizens with whom AOL will refuse to have dealings is, of course,
those who aren't locked into their ISPs for mail service.
SPF doesn't actually address unsolicited bulk mail at all.
"Sent by an SMTP Relay client that one doesn't expect" is not the same
as "unsolicited bulk". Blocking mail with the former quality won't block
mail with the latter.
Moreover: Consider the case of the unsolicited bulk mail generated by
Microsoft Worms, for example. Microsoft Worms run on infected machines,
and using the /same/ mail submission tools that the machine's owner uses
to submit normal mail, they mail themselves to other people. SPF does
not and cannot distinguish such mail traffic generated by Microsoft
Worms from normal mail traffic sent by the machine's owner. It's
submitted to the same submission service, using the same user
credentials, and travels along the same route, as all other mail.
SPF marketing
But SPF isn't an anti-UBM tool and was never portayed as such! goes the
cry.
Someone appears to have forgotten to tell the SPF webmaster and Meng
Weng Wong himself that, then.
*
SPF reduces inbound spam. -- SPF web site "executive summary"
<http://spf.pobox.com./execsumm.html>
*
If the message fails SPF tests, it's a forgery. That's how you can
tell it's probably a spammer. -- SPF web site FAQ answers
<http://spf.pobox.com./faq.html> (That first sentence is untrue,
of course.)
*
Eventually, as SMTP improves its immunity to spam, we hope
spammers will get discouraged. -- SPF web site FAQ answers
<http://spf.pobox.com./faq.html>
*
'Why should SPF succeed when similar proposals have failed in the
past?' The spam problem was never as bad in the past as it is now.
-- SPF web site FAQ answers <http://spf.pobox.com./faq.html>
*
SPF is ushering in a new set of anti-spam systems, [...] -- SPF
web site <http://spf.pobox.com./>
*
Saving the world from spam is an expensive duty. -- SPF web site
requesting your money <http://spf.pobox.com./donations.html>
*
I really believe if SPF sees fast widespread adoption we can stop
spam sooner than you think. -- Meng Weng Wong <http://mengwong.com./>
SPF hands Verisign its next unwelcome "innovation" on a platter.
We have already witnessed that Verisign is prepared to put into action
schemes to make itself more money at the expense of other Internet
entities. It has already once made a land grab claiming all previously
unowned *.com. and *.net. subdomains as its own
<http://homepages.tesco.net/J.deBoynePollard/FGA/verisign-internet-coup.html>,
in order to sell advertising rights on web pages that it had set up
associated with all of those domains, at the expense of many other
Internet users using many other features of Internet.
SPF hands Verisign another nice little money spinner, in the form of
enabling it to specify what SPF data are published for unowned domains
by simply adding TXT resource record sets to its wildcards, and to sell
the rights to have their SMTP Relay clients properly validated by SPF,
for huge numbers of previously unowned domain names, to the highest
bidding UBM sender.
Some ISPs already have "pink contracts" with UBM senders, essentially
providing them with services in exchange for a cut of the take. And
Verisign is already attempting to bring its *.com. and *.net. wildcards
back into service so that vast numbers of people are directed to its web
servers with the advertising space once again. It seems likely,
therefore, that Verisign would find this additional money spinner very
difficult to resist.