On 08/01/18 19:56, Yuval Levy wrote: > On 2018-01-07 04:32 AM, Peter wrote: >> So to put it simply, they're basically saying that their black box >> thinks that your IP(s) are sending SPAM. > > That's not how I read my conversation with them. My understanding based > on their explanations and their advice on how to fix the problem is that > their black box thinks that the content of the message sent is spam and > has very little to do with the IP(s).
I may have misunderstood, anyways, either way it is based, at least at some point, on the content. > Indeed we can only speculate about what triggers the blackbox. Yep, so it probably doesn't pay to speculate down this line of thinking any further. >>> HTML format? >> >> Highly unlikely. > > Then why Microsoft's advice to "brand" and "properly format" the > message? Coming from the company that has gifted the world the scourge > that is HTML email, and whose view of what represent a properly format > has probably been over-represented in the training sample that was fed > to the neural network (just guessing) powering the filter? Well, this at least is easily tested. Send the same email in both plain text and HTML/Plain multi-part and see if the HTML one goes through and the plain is blocked. >>> FLOSS client software and O/S? >> >> Also highly unlikely, I do the same and don't have issues. Also it's >> all about following the SMTP protocol correctly, which has nothing to do >> with whether the software used is FLOSS or proprietary or anything >> in-between. > > On this one I tend to agree with you, for a different reason. I do not > think it is about following the SMTP protocol correctly because my MTA > received a 250 from their MTA on the messages and because of primary > factors mentioned in Microsoft's vague description of SmartScreen's > outcome being "primarily due to mail content and recipient interaction > with that mail." > > My very wild guess is that their system mixes up correlation with > causality or is skewed against exotic / seldom seen combination of the > User-Agent header just because it does not see it so often, but that > would be stupid given the overwhelmingly large volume of spam vs > ordinary mail. This as well is test-able. Try the same message in different user-agents. Borrow a laptop with windows, configure outlook to send through your server and send a message identical to one that was dropped from the same network. You also mentioned that you're GPG signing the mail and using PDFs. If the PDF is identified as a possible source of a virus (and PDFs can contain viruses) then this could be a legitimate reason for them to drop your email. At any rate the GPG and PDF scenarios are also easy to test by sending a message that is not signed and one with the content that is just plain text (not in a PDF) and seeing if either of these tests go through. One more thing I should note here is that if MS black box is anything like other, open source anti-spam solutions then it will use a scoring system, so an expired cert might give you a half point, no HTML content might add one point, PDF might be another point. Some word or phrase in the email might add a bit more and if your SPAM score gets above a certain threshold it is dropped, so in that respect it is not any one thing that is causing your mail to get dropped but a number of things combined. >>> sending MTA is dynamic/residential> >> This shouldn't be the case, but if your submission server resides on a >> dynamic IP address then this could very well be the case. I say >> "shouldn't", though, because there have been known cases of anti-spam >> appliances in the past that do deep inspection of Received headers and >> compare IP addresses found in them against policy blacklists. These >> types of blacklists are designed only to be used against the IP address >> of the connecting server, not IPs found in headers. That said, I don't >> think that Microsoft is doing that. > > Your argument on this one is convincing. I will probably never know the > answer, at least not in my setup. I realized this was a weak spot and > fixed it with: > > /etc/postfix/main.cf > > mime_header_checks = regexp:/etc/postfix/headers_checks > header_checks = regexp:/etc/postfix/headers_checks > > /etc/postfix/headers_checks > > /^Received:/ IGNORE > /^X-Originating-IP:/ IGNORE > /^X-Mailer:/ IGNORE > /^Mime-Version 1.0.*:/ REPLACE Mime-Version: 1.0 ...and did any of this help? >> I really would not put any malicious intent here. > > Agree. Negligence is easier to prove and in both cases the outcome is > the same and so is the practical fix. Just the punishment is different, > but I am not out to punish Microsoft, I just wish it behaved more > collaboratively. On this we can agree. >>> Bottom line, I think the problem is more ethical than technical. >> >> I certainly think it's technical on their part, but if by ethical you >> mean that Microsoft just doesn't care enough about you to want to solve >> your problem then you're probably right. > > By ethical I mean their way of conducting business, which results in > their technical decisions. I think it is unethical to make the > statement "My MTA received an email from your MTA and it is queued for > delivery" when in fact it should be "My MTA has received an email from > your MTA and has dropped it into a black hole," or even "My MTA has > received an email from your MTA and what will happen next I won't tell > you." Even assuming no malicious intent, the first statement is a > misrepresentation with serious consequences. The last statement makes > email worthless. Well I think at least here I might be able to give some explanation that might help. When an MTA receives a message it has a number of different choices of what it can do with it: * It can accept the message and deliver it (delivering it to SPAM is still considered delivering it here). * It can accept the message and then drop it (which is what they are doing here). This is what they are doing and it's bad. * It can accept the message, send a DSN (bounce email) to the envelope sender address and then drop the message. Look up "backscatter" to see why this is also bad. * It can reject the message with a 5XX code (or defer it with 4XX). What is happening here is that at the time that Microsoft accepts the message they have not processed it through their back box yet, so they can't tell in time to reject it if the message looks like SPAM to them or not. They don't want to send out a DSN because it ends up causing them to become a source of backscatter and so they drop the message, which is also bad for reasons that you have already pointed out. What they should be doing is rejecting it straight away if they possibly can or delivering it into the user's SPAM folder. The only time (as someone else already pointed out) that it is generally considered appropriate to drop mail is if it contains dangerous content such as a virus. In that case delivering it to SPAM is not a good idea because someone may still open the virus to look at the message. > This is why I am still of the opinion that the problem is more ethical > than technical. It is akin to a rental car company renting a car that > will loose a wheel while driving. Sure the immediate solution is > technical, but the overarching solution is ethical: a standard that > distinguish between good behavior, where the accidental loss of a wheel > in circumstances beyond the control of the service provider vs bad > behavior where the accidental loss of a wheel is caused by the > negligence of the service provider and could be avoided with a proper > standard of care (which in this case would be to put the message > identified as spammy in the junk folder; or to let the sending MTA know > that the message was dropped). The analogy of the rental car fails when you realize that neither you, nor the recipient of your messages (who likely do not pay MS for the usage of their hotmail account) are considered by MS to be "customers". They are more accurately described as the "product" and the MS customers are actually advertisers that pay to display ads on the webmail interface. Now you probably know the full legal issues way better than I ever could but it does often times pay to see how a company like MS views it. Anyways, I hope this continues to help you shed light on the situation. Peter
