Re: [mailop] NDRs and DSNs was: Re: Delivery to gmail via IPv6
> On 18 Dec 2015, at 01:55, Karen Ballewrote: > > Brandon, > > I really like this idea, even as a mental exercise. There would be a lot of > usefulness in being able to clearly dictate from the enhanced codes when our > MTAs should stop, back off, or go into an extended timeout. Well, that should be pretty clear from the 4xx vs 5xx status. If it’s 4xx, you should queue the message for later delivery, otherwise generate a bounce. I guess if the recipient wants to indicate that you should reduce your sending rate, that’s something that needs a new status, but you could use "4.3.1 Mail system full", or "X.4.5 Mail system congestion" because the sensible response would be to back off. > Perhaps something like having the enhanced code be tied to the primary > reason for an organization not accepting email would be useful? I can't see > where we could (or should) come up with a system where every reason an email > isn't being accepted is communicated in a single NDR. Filters are getting > horribly complicated and there are often multiple reasons why a sender's mail > isn't being accepted. > From an ESP perspective, being able to tell a customer that Gmail clearly > said in a bounce 1) It's you, not them. There are plenty of codes that say this. All of these are errors that could be fixed by the sender (not their MTA) X.1.1 Bad destination mailbox address X.1.2 Bad destination system address X.1.3 Bad destination mailbox address syntax X.1.6 Destination mailbox has moved, No forwarding address X.1.7 Bad sender's mailbox address syntax X.1.8 Bad sender's system address X.2.3 Message length exceeds administrative limit [per mailbox] X.3.4 Message too big for system X.5.3 Too many recipients > 2) Your content/domain rep/IP rep is horrible. X.7.1 Delivery not authorized, message refused The sender is not authorized to send to the destination. But, there’s plenty of scope for extension here: with three digits available for . Perhaps it’s time to look at extending the set of enhanced error codes available. A 20th anniversary celebration! RFC 1893 Mail System Status Codes January 1996 RFC 3463 Enhanced Mail System Status Codes January 2003 > 3) and you should feel bad X.7.10 Go directly to Jail, do not pass GO! ;^) There’s a registry, and a review might be a good idea. https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml#smtp-enhanced-status-codes-2 > would cut through a lot of the pushback we get now. > I could see a lot of use for anyone with a shared IP infrastructure, as well. > If most of the DSNs from shared IP space came back with an enhanced code > pointing to IP reputation but a handful of customers have more specific > content or sender-related codes, it gives us amazing insight into the > troublemakers trying to hide on the shared IPs. How much benefit do you > think most mailbox providers would get if ESPs were able to more reliably > adapt around having that extra certainty in our decision-making process? > > Couple of ideas for it > Mailbox specific - invalid account, recently closed account, mailbox full, > recipient-configured rejection rules > ISP specific - server/network saturation (which we should treat with a full > stop on traffic from all our senders rather than an extended retry for a > single sender with traffic shaping relating to reputation), LDAP failure, > server farm under attack by backhoe or glowing molerats > IP specific - traffic shaping for excessive volume for current reputation > (very handy for the holidays and ramp ups), blocked for complaints, blocked > for excessive invalid users/spamtraps, blocked for phishing/malware, traffic > spread across too many IPs, snowshoe, your provider sucks, get off my lawn > Sender specific - could be broken down for domain, subdomain, or localpart > combinations if we're feeling really ambitious but that could be providing > too much feedback that could be abused. Marketing vs transactional is often > split this way, though. Helpful for customers using their mailbox provider's > domain. Would allow ESPs to set rules to stop mailing for specific senders > immediately and definitely ties in with compliance standards. > Authentication and configuration failures - DMARC, no rDNS, missing A or MX > records (I had a fight with a customer this week about setting those up) > Content specific - Again, could be easily abused but there are some ways it > could be helpful. Redirector who doesn't handle their spam, linking to > commonly phished sites, bad landing page rep (possibly malicious), Normally > good sender who included third party content on a send. > > Karen > > On Wed, Dec 16, 2015 at 11:35 PM, Brandon Long wrote: > We have automated systems, especially for Google groups, which are basically > doing the same thing, and it's a pain. > > I agree that the
Re: [mailop] NDRs and DSNs was: Re: Delivery to gmail via IPv6
Although the reply messages are gamed, as we see too, we do not let the outcome be any different. It is up to the mailserver operator to decide how we are to view bounces. That is their responsibility and not ours. In the end I want to make it possible to explain my actions to everyone involved. Our internal buckets are below. Anything we found could be pressed into these categories, types and subtypes. Yours, David Category of bounce outOfOffice/autoresponder We received an out-of-office email. Default = ignore this email. These mails will be parsed further for definitive indicators of inactive mailboxes. Technical bounce/delayed We received a message that stated that this email was in transit (and not yet delivered). Default = ignore this email. Technical bounce (but no delay indicator, therefore treated as NDR) We received a technical email back from a mailserver. Default = softbounce this email. These mails will be parsed further. See below. Spam, default = softbounce this email. spam blacklisted Some part of the email or the sender was identified as blacklisted. spam content-related The content was identified as spam. spam sender-denied The sender, you or us, was denied for non-specific reasons. spam spf-failure The email was not allowed to be forwarded due to anti-spam configuration issues. spam unknown It was identified as spam for unknown reasons. personalBlacklist personal-blacklist The recipient has a specific blocklist against you. Hardbounce this email. Hardbounce hardBounce domain-unknown The domain is not valid. hardBounce mailbox-disabled The mailbox is there, but no longer in use. hardBounce mailbox-unknown The mailbox is not known to the mailserver. hardBounce unknown The mailbox is not working anymore for non-specific reasons. Softbounce Softbounce access issues softBouncerelay-denied softBounceaccess-denied This often means a configuration error on the domain or the mailserver. It does not accept email for that domain (without authentication). softBouncesender-denied The recipient has not granted you access to email him/her. Softbounce mailbox issues: softBouncemailbox-temp-disabled The mailbox is (temporarily) disabled. softBouncemailbox-temp-error There is an error for this mailbox which could be temporary. softBouncemailbox-full The mailbox is full, which could be temporary. Softbounce message errors: softBouncemessage-error There was something wrong with the message itself softBouncemessage-expired The message could not be delivered within a reasonable time frame. softBouncemessage-size The size of the message was too large. Softbounce networking related softBounceconnection-error There was something wrong while connecting to the mailserver. softBouncerouting-error softBouncerouting-loop The email was sent but returned because it could not be delivered to the inbox (of another domain). Softbounce mailserver related softBounceconfig-error softBouncesystem-error softBounceservice-unavailable Some technical error occurred while sending the mail to that mailserver and that is why it could not accept the message. softBounceunknown Some other error occurred that could not be determined reliably. It could be temporary. Van: mailop [mailto:mailop-boun...@mailop.org] Namens Brandon Long Verzonden: donderdag 17 december 2015 05:35 Aan: Karen Balle CC: mailop; Ian Eiloart Onderwerp: Re: [mailop] NDRs and DSNs was: Re: Delivery to gmail via IPv6 We have automated systems, especially for Google groups, which are basically doing the same thing, and it's a pain. I agree that the SMTP status code is mostly useless from this prospective, and the extended ones aren't much better. It might be nice if there was a concept of whether a failure is message specific, sender specific, etc... But anything like that would be gamed both for antispam and for spammers. Still, may be worth a thought experiment, a replacement for the extended status codes or even a bcp about how they should
Re: [mailop] NDRs and DSNs was: Re: Delivery to gmail via IPv6
Brandon, I really like this idea, even as a mental exercise. There would be a lot of usefulness in being able to clearly dictate from the enhanced codes when our MTAs should stop, back off, or go into an extended timeout. Perhaps something like having the enhanced code be tied to the primary reason for an organization not accepting email would be useful? I can't see where we could (or should) come up with a system where every reason an email isn't being accepted is communicated in a single NDR. Filters are getting horribly complicated and there are often multiple reasons why a sender's mail isn't being accepted. >From an ESP perspective, being able to tell a customer that Gmail clearly said in a bounce 1) It's you, not them. 2) Your content/domain rep/IP rep is horrible. 3) and you should feel bad would cut through a lot of the pushback we get now. I could see a lot of use for anyone with a shared IP infrastructure, as well. If most of the DSNs from shared IP space came back with an enhanced code pointing to IP reputation but a handful of customers have more specific content or sender-related codes, it gives us amazing insight into the troublemakers trying to hide on the shared IPs. How much benefit do you think most mailbox providers would get if ESPs were able to more reliably adapt around having that extra certainty in our decision-making process? Couple of ideas for it Mailbox specific - invalid account, recently closed account, mailbox full, recipient-configured rejection rules ISP specific - server/network saturation (which we should treat with a full stop on traffic from all our senders rather than an extended retry for a single sender with traffic shaping relating to reputation), LDAP failure, server farm under attack by backhoe or glowing molerats IP specific - traffic shaping for excessive volume for current reputation (very handy for the holidays and ramp ups), blocked for complaints, blocked for excessive invalid users/spamtraps, blocked for phishing/malware, traffic spread across too many IPs, snowshoe, your provider sucks, get off my lawn Sender specific - could be broken down for domain, subdomain, or localpart combinations if we're feeling really ambitious but that could be providing too much feedback that could be abused. Marketing vs transactional is often split this way, though. Helpful for customers using their mailbox provider's domain. Would allow ESPs to set rules to stop mailing for specific senders immediately and definitely ties in with compliance standards. Authentication and configuration failures - DMARC, no rDNS, missing A or MX records (I had a fight with a customer this week about setting those up) Content specific - Again, could be easily abused but there are some ways it could be helpful. Redirector who doesn't handle their spam, linking to commonly phished sites, bad landing page rep (possibly malicious), Normally good sender who included third party content on a send. Karen On Wed, Dec 16, 2015 at 11:35 PM, Brandon Longwrote: > We have automated systems, especially for Google groups, which are > basically doing the same thing, and it's a pain. > > I agree that the SMTP status code is mostly useless from this prospective, > and the extended ones aren't much better. > > It might be nice if there was a concept of whether a failure is message > specific, sender specific, etc... But anything like that would be gamed > both for antispam and for spammers. > > Still, may be worth a thought experiment, a replacement for the extended > status codes or even a bcp about how they should be used. > On Dec 16, 2015 6:03 PM, "Karen Balle" wrote: > >> Since there's been a lot of drift on this topic and I'm not talking about >> IPv6, DNSSEC, or delivery to Gmail >> >> For NDRs and DSNs to be truly useful to an ESP, we tend to break them >> down into more buckets than two. >> >> Hard bounce/invalid (do not retry, generally a 5xx) >> Soft bounce (temporary and related to a specific email address, generally >> a 4xx, try again later during the current send, Yahoo uses 4xx where we >> would expect a 5xx such as with some of the traffic shaping deferrals) >> Technical (DNS failures and the like, try again later, generally 5xx but >> depends on the reason) >> Spam or block (stop for this send, try this recipient on future sends, >> mix of 5xx and 4xx) >> Unknown (parser doesn't have an appropriate string, manual review and >> update of systems required) >> >> The primary challenge from a technical perspective is making sure that >> any changes in friendly language in an NDR or DSN is updated on the bounce >> parser. We do tend to MOSTLY ignore 5xx and 4xx, but that is more a >> function of how those are used by mailbox providers. The desired end >> result is that we send all email in a manner that is acceptable to each >> mailbox provider based on their (technical and personal) feedback and >> desired rate limits. Extended codes are