Re: Bug report: problem with smtp_mx_address_limit = 0
On 3/04/19 17:13, Viktor Dukhovni wrote: > > > It seems you're in a sardonic mood, ... best to not go there. If I gave that impression, I apologize. All I was trying to say is that, because of the bug, you may be more likely to run out of disk space by trying to send to bona fide domains like postfix.org than because of overworked delivery agents, simply because MX's with both A and records are much more common (and growing) than malicious ones. Anyway, the most important thing is that the bug has been noted, and will eventually be fixed, one way or another. Thanks for that. Luc
Re: Bug report: problem with smtp_mx_address_limit = 0
On 3/04/19 01:16, Wietse Venema wrote: > I prefer to remove the ability to disable safety mechanisms. And in your initial response, you also wrote: > Probably better to not allow a limit-less smtp_mx_address_limit, > as it makes Postfix vulnerable to resource exhaustion attack. Both responses seem to translate to "I prefer to remove the ability to set smtp_mx_address_limit to zero". As I said, that's fine with me. My only comment is that this won't bring you much. As you indicate yourself, people would still have the ability to effectively disable the safety mechanism (by setting smtp_mx_address_limit to one bazillion or so). The only difference is that, if they really want to get rid of the limit, they would (have to) disable it in an undocumented way. So, given that you can't prevent people from shooting themselves in the foot anyway, it seems more logical to let them do it in the (currently) documented way. This being said, I personally don't really care whether the ability to set smtp_mx_address_limit to zero is fixed or removed, but something has to be done. As it stands now, people can still set it to zero ... > > ... and watch their mail queues melt down. > ... because of all the mail that will get stuck in the queue with "server unavailable or unable to receive mail" for _any_ MX that has _both_ A and records. One of each is enough to trigger the bug, even if they would all respond if Postfix bothered to try. But it doesn't. It simply defers the mail until it eventually expires and bounces. Anyway, I have reported the issue and am happy to leave it at that. Luc
Re: Bug report: problem with smtp_mx_address_limit = 0
On 2/04/19 13:21, Wietse Venema wrote: > Probably better to not allow a limit-less smtp_mx_address_limit, > as it makes Postfix vulnerable to resource exhaustion attack. > > Wietse > Fair enough, but then the docs for smtp_mx_address_limit ought to be changed to remove the "or zero (no limit)" at the end. I'd also suggest an explicit warning in case it is still set to zero, instead of the rather misleading (and inaccurate) "server unavailable or unable to receive mail" for any MX with both A and records. On the same topic: what if smtp_mx_address_limit was simply made to apply for each family separately? E.g. the default of 5 would mean: keep max 5 IPv6 addresses _and_ max 5 IPv4's ? That would a) eliminate the need for balancing the mix, b) simplify the code, and c) allow the client to honor the smtp_address_preference setting in a safe way. Separate limit parameters for each family would also do the job, but that would probably be overkill. Luc
Bug report: problem with smtp_mx_address_limit = 0
According to the docs, the smtp_mx_address_limit parameter determines
"the maximal number of MX (mail exchanger) IP addresses that can result
from mail exchanger lookups, or zero (no limit)".
However, when setting it to zero, the SMTP client won't even attempt to
deliver to a server that has _both_ IPv4 _and_ IPv6 addresses. Instead,
the mail is deferred forever (and 4 days ;-) with the error message
"4.4.4 server unavailable or unable to receive mail".
Yet a Perl script can connect to all these addresses just fine, and even
deliver mail when talking SMTP to the other side.
The problem seems to be that the function "smtp_balance_inet_proto" (in
smtp_addr.c) does not treat "0" as the special "no limit" case. Instead,
it treats it literally as a limit of zero IP's.
The function contains the line:
if (v4_count > 0 && v6_count > 0 && v4_count + v6_count > addr_limit)
and inside the "if" block, a clever algorithm attempts to create a mix
of IPv4 and IPv6 addresses, by eliminating some of each kind, while
enforcing the limit.
But when addr_limit is set to 0, all IP's are eliminated from the list.
I confirmed that by adding "-v" to the smtp client in master.cf and
sending an e-mail to an address in a domain with one MX that has one A
and one .
The Postfix logs show that both IP's are found in DNS :
"begin address list"
"pref 300 host /
"pref 300 host /
"end address list"
The logs then continue with the tell-tale entry:
"v6_target=0, v4_target=0"
Followed by the two lines:
"begin smtp_balance_inet_proto result address list"
"end smtp_balance_inet_proto result address list"
With nothing in between...
Note that, in the above, I have obfuscated the host name and IP
addresses. I think I could safely post them here, but I suppose that
they won't be needed. Besides, postfix.org has 3 A's and 3 's and
could probably be used as a test case as well ;-)
Adding "addr_limit > 0" as the first condition in the "if" statement
above would likely take care of the problem.
IMHO, a more elegant (because more readable) solution would be to put an
explicit extra test in front of the (already long) "if", like this:
if (addr_limit == 0) {
return (addr_list);
}
A suitable comment in front would make it even more clear.
I have not provided a patch as this seems such a trivial change. In the
Postfix 3.4.5 source code, the offending "if" statement is at line 409
of smtp_addr.c. Alternatively, you can go straight to the
smtp_balance_inet_proto() function in there.
However, this also means that I haven't tested if my proposed solution
actually works. If somebody tells me that it is not (or may not be)
sufficient, I'm willing to dig further myself.
In any case, a fix is not urgent, since there is an easy workaround, and
that is: "don't RTFM", i.e. leave smtp_mx_address_limit at its default of 5.
And if you really want "no limit", you can set it to a ridiculously high
value, such as 100.
>From the changelog, this seems to have been introduced on 20170505, and
in any case, smtp_addr.c has a timestamp of Dec 27, 2017.
If this has been reported already, I apologize. I did a quick search
only and came up empty. If I missed it, this "bug report" becomes a "bug
reminder" :-)
It goes without saying, I hope, that the other client parameters in
main.cf have to be set to the proper values to trigger the bug (e.g.
smtp_balance_inet_proto = yes).
Luc Pardon
Re: Patch: eliminate postfix-script warnings about symlinks
On 05-09-18 18:18, Wietse Venema wrote: > Luc Pardon: >> The first question is obviously: can we disallow symlinks to the outside >> world by definition? I'd say the answer is yes, but $(whoami) ? > > Here is some background on pathname safety. > Thanks for that. Also, the paper - at a first cursory reading - is very helpful, and thought-provoking as well. But we have to be clear about the problem that we are trying to solve here in this particular case. So allow me to re-iterate for a moment. * My problem is that, after pointing "smtpd_tls_CApath" and "smtp_tls_CApath" to a subdir of /etc/postfix, and installing a bunch of root CA certificates in it, my logs are flooded with something like 150 (!) bogus warnings about the symlinks to these certs having the wrong permission. * You correctly observed that my proposed "-L" patch does too much: it suppresses _all_ warnings about _all_ symlinks, safe or not. I do agree that this is an unacceptable loss of functionality. However, you propose to compensate for it by adding - necessarily elaborate - safety checks on symlinks. I don't think we need to go that far. For starters, I'd venture to say that, in the past, symlinks inside the safe_for_X Postfix dirs were never needed nor used in the first place. Neither were they expected, it seems. In any case the postfix-script doesn't "see" that it is dealing with a symlink. It treats it just like a regular file - and trips over its permissions (which are 0777 by definition, hence wrong by definition inside a safe_for_X dir). That also means that _all_ symlinks get warned about - as I found out. The only way for the admin to get rid of such a warning is to remove the symlink altogether, since, unlike a regular file, chmod won't help here. Conclusion: any symlink, that happened to trespass into Postfix territory, must in the past have been systematically removed by those admins that read their logs and heed the warnings (meaning, of course, _all_ admins ). However, although symlinks inside the Postfix dirs were not needed in the past, that has changed by now. They have become necessary because OpenSSL needs them to find its certificates, so we can't just tell the admin to get rid of them. Also, with initiatives like "Let's Encrypt" and "STARTTLS Everywhere" gaining ground, more and more Postfix users will start making real use of certificates, and they too will eventually start complaining about the bogus warnings - or worse: ignore them, together with the important ones. We could of course sidestep the issue by patching the docs, telling people to store their certificates _outside_ /etc/postfix. However, that doesn't feel right. One reason is that, if they are used for handling the mail, they are effectively part of the Postfix config, and /etc/postfix is their proper home. Also, those of us who run SELinux would have to invent and add some custom rules, just to allow Postfix to access those certificates and their dirs. That is cumbersome, tricky to get it right, and - above all - not needed when we store them in /etc/postfix, where they belong. Fortunately, I think that we can remain safe by continuing to warn about symlinks, much as before, except that we would from now on remain silent about symlinks that: a) live inside a "safe_for_X" directory structure, and b) point to a target inside that very same dir structure. And by "safe_for_X dir structure", I do mean of course one of the dirs that are monitored by postfix-script. Such symlinks are inherently safe_for_X, simply because they point to a safe_for_X target. No need to pick them apart to find that out. We can indeed rely on the target being safe, because if it is not, the regular file check of postfix-script would issue a "wrong permission" warning for this target, and the admin would step in to fix that with chmod. And when the target is made safe, the symlink pointing to it automatically becomes safe as well. Case in point: the OpenSSL symlinks happen to live in the same dir as the certificate they point to, and the certs themselves are monitored by postfix-script because they are in $config_directory. It is a different story with symlinks that point to a target _outside_ one of the "safe_for_X" Postfix dirs. Here we know nothing about the target, so we would normally have to go out and investigate closely - as you propose to do. However, such symlinks, pointing all over the place, are not needed today, no more than they were needed yesterday. So we can continue to issue a warning as we did before, and thereby incite the admin to remove them, as before. It doesn't matter whether they are good or bad or ugly. They don't belong in there and they must go, period. Also, I am not convinced that the elaborate checks would have much added value. Quite to the contrary - at least for th
Re: Patch: eliminate postfix-script warnings about symlinks
On 05-09-18 15:04, Luc Pardon wrote:
> The Q shell scriptlet below my sig would probably do the trick
>
> ==
> #!/bin/sh
>
> # This would not be needed if integrated into postfix-script:
> BASE=$(postconf -hx config_directory | sed "s/\n$//")
>
> # Search for dangerous symlinks in $1 and its subdirs
> function inspect
> {
>DIR=$1
>
>for f in $DIR/* ; do
> if [ -L $f ]; then
> # if it points outside $BASE, it starts with "../.."
> DOT=$(realpath --relative-to $BASE $f | cut -d'/' -f1)
> if [ $DOT = ".." ]; then
> echo "ALARM: $f is a symlink to $(realpath $f)"
> fi
> elif [ -d $f ]; then
> inspect $f
> fi
>done
> }
>
> inspect $BASE
> ==
>
Q for sure.
Of course that two lines about "DOT" should better read:
DOT=$(realpath --relative-to $BASE $f | cut -d'/' -f1-2)
if [ $DOT = "../.." ]; then
Sorry about that, and also about breaking the message thread.
Luc
Patch: eliminate postfix-script warnings about symlinks
On 05-09-18 13:26, Wietse Venema wrote:
> Luc Pardon:
>> Hello,
>>
>> Running Postfix 3.3.1 under Linux, postfix-script produces pointless
>> warnings if/when there are symbolic links in or below $config_directory.
>
> The problem is that the symlink may point to any location including
> a file under an unsafe directory such as /var/tmp or /home/user.
>
> Unless you are constraining the target of the symlink (and your
> patch does not), this is a security hole waiting to happen.
>
> Wietse
>
True enough, of course.
But that is a wholly different check than the owner/permission check.
Besides, flooding the logs with unhelpful warnings actually has the
effect of obscuring the security holes.
Finding those would call for a separate check, specifically looking for
symlinks pointing outside the Postfix root-owned directory tree.
And I'd say that this check would belong in the "check-fatal" section of
postfix-script, rather than in "check-warning".
The Q shell scriptlet below my sig would probably do the trick for
$config_directory, but it requires the "realpath" command, part of the
GNU core utilities package. Not sure how portable that would be. Some
obscure incantation of "find" could probably do the job, too.
The first question is obviously: can we disallow symlinks to the outside
world by definition? I'd say the answer is yes, but $(whoami) ?
Luc
==
#!/bin/sh
# This would not be needed if integrated into postfix-script:
BASE=$(postconf -hx config_directory | sed "s/\n$//")
# Search for dangerous symlinks in $1 and its subdirs
function inspect
{
DIR=$1
for f in $DIR/* ; do
if [ -L $f ]; then
# if it points outside $BASE, it starts with "../.."
DOT=$(realpath --relative-to $BASE $f | cut -d'/' -f1)
if [ $DOT = ".." ]; then
echo "ALARM: $f is a symlink to $(realpath $f)"
fi
elif [ -d $f ]; then
inspect $f
fi
done
}
inspect $BASE
==
Patch: eliminate postfix-script warnings about symlinks
Hello,
Running Postfix 3.3.1 under Linux, postfix-script produces pointless
warnings if/when there are symbolic links in or below $config_directory.
1. I installed (CA root) certificates in a subdir of /etc/postfix and
rehash with "openssl rehash . This will of course create a
symlink to each certificate.
But to reproduce, any symlink in (a subdir of) /etc/postfix will do.
Now restart Postfix, since you can't run "postfix-script check-warn"
directly.
For each symlink, a warning is written in the log, like this:
postfix/postfix-script[23592]: warning: group or other writable:
/etc/postfix/./tls/CAcerts/02265526.0
This should not happen, as the permissions of symbolic links are
irrelevant on Linux. They are never used, it is those of the pointed-to
file that count.
See "man 1 chmod" or - if you have it installed - "man 7 symlink".
Of course, these "bogus" warnings do not really hurt, but they do
clutter the logs and thereby obscure the real thing.
The following (very) simple patch to postfix-script 3.3.1 takes care of
them :
@@ -302,7 +302,7 @@
find $todo ! -user root \
-exec $WARN not owned by root: {} \;
- find $todo \( -perm -020 -o -perm -002 \) \
+ find -L $todo \( -perm -020 -o -perm -002 \) \
-exec $WARN group or other writable: {} \;
# Check Postfix mail_owner-owned directory tree owner/permissions.
I suppose the -L parameter could be added to the other occurrences of
"find", but I didn't bother with that.
2. As an aside, it would be cool if those warnings could give the real
name of the offending file. That is, instead of:
/etc/postfix/./tls/CAcerts/
it really should be:
/etc/postfix/tls/CAcerts/
But that is a cosmetic issue only.
NB: my copy of "find" is from an older findutils 4.5.11 package, current
seems to be 4.6.0.
3. What is maybe more important, that is that there were no such
warnings about the symlinks in the chroot jail.
Yet I did copy all the certificates from that CAcerts dir over into its
jail counterpart, and rehashed there as well. So I would have expected
the same bogus warnings about the symlinks in there.
However, the postfix-script doesn't seem to check (all) the subdirs of
the $queue_directory for owner- and permission-related issues. It just
looks at /var/spool/postfix/pip.
Maybe it should check the others as well? Or at least
"/var/spool/postfix/etc/postfix" ?
Luc
[SOLVED] Re: Fall back to relay after [some] 5XX repl[ies] from destination?
(Please note that I also changed the remainder of the subject line to try and avoid further confusion.) On 06-08-18 20:48, Wietse Venema wrote: > Luc Pardon: >> The only "problem" that I am trying to solve is how to detect that the >> other side is telling me to re-send through my ISP, so that I can oblige. >> >> With other types of 5xx, I would of course let the mail bounce as usual. > > Conceivably, you could use the smtp_delivery_status_filter feature > to selectively replace 5xx responses with 4xx if the receiver does > not say things that look like 'user unknown'. > Yes, that was indeed what I ended up using (with the minor difference that I go looking for strings like "blocked using ...", to be more selective). Thanks for confirming that this hook was the proper thing to look at. > Combined with smtp_fallback_relay=[mail.isp.com], this would punt > potentially resendable mail to the ISP. > I decided to work with transport maps instead, one per domain. One of the reasons is that, if I already know that example.com doesn't like my current IP, it doesn't feel right to keep trying again and again for each and every message. With a low traffic site like mine, the extra load on example.com may be negligible, but in any case it serves no useful purpose (unlike greylisting, SAV, etc.). Compare to my restaurant example: if I know that they want me to wear a tie, it is only proper to remember to put one on before I go there next time. Another (very) important reason to _not_ use smtp_fallback_relay is that, unless I am mistaken, *all* the deferred mail would go to the relay, including mails that are being greylisted by domains that would otherwise accept them directly from me after the greylist time. That is Doubleplus Ungood, because - as I repeatedly said - I want to avoid to relay if I can. Sending via an ISP in Belgium is asking for trouble, believe me. Bottom line, when my DSN filter detects one of the "blocked using ..." tell-tale strings, it first creates the appropriate "nexthop=mail.isp.com" transport map for the example.com domain, and then it defers the mail by returning "4xx" instead of the original "5xx". At the next attempt to deliver the mail to john@example.com, the "example.com" transport map will be waiting for it. Problem solved. Of course, the disadvantage (for my purpose) of transport maps is precisely that they are "static". Meaning, once the map for example.com is in place, all the mail for them will go through "mail.isp.com" forever and ever - until the map gets removed again. Now, example.com may refuse my IP today, yet they may start to accept it tomorrow, or next week, or whenever my (then current) IP is not (or no longer) listed by the RBL's they consult. This has happened before, even in the short time (a little over 2 weeks now) that I was forced from static to dynamic, and even though I am forcing a DSL reconnect every night, just to stress-test the whole thing. So yes, it really can happen that example.com "changes their mind" over time. But because of the transport map, created at their first refusal, I would keep sending through the relay for nothing, because they would now accept direct delivery if I tried. So I keep the maps in an SQL table and plugged a tcp_table map script into the "transport_maps" hook. If the table script finds an expired transport map for example.com, that map will be ignored, and the mail will "go direct" again. Time will tell how best to expire the maps, but anyway these details are not relevant to get the overall picture. You may wonder how the DSN filter knows how to insert a map for example.com, since it doesn't know the intended recipient address? Indeed, the filter map gets only queried with this: enhanced-status-code SPACE explanatory-text And of course the explanatory text in this case does not usually mention the recipient... Taking that hurdle was actually the easier part of the job. A few minor changes to the Postfix source code were enough to make the dsn_filter_lookup() function (in global/dsn_filter.c) take an extra "RECIPIENT *rcpt" parameter, so that it now can (and does) query the maps with: recipient-address SPACE enhanced-status-code SPACE explanatory-text Of course, this is a radically incompatible change that brutally breaks all existing DSN filters big time, but I myself had no such filters before, so there were none to break, and, as a private patch (applied here on my own computer when I compile my own Postfix from source for my own use), it serves me just fine. So, as I said, problem solved. It has been working fine for several days now. We'll see what happens in the future. Anyway, all it took was a handful lines of (mostly boilerplate) Perl code. As to Postfix itself, it says much about the
Re: Fall back to relay after a 5XX reply from destination?
On 26-07-18 17:19, Matus UHLAR - fantomas wrote: > Anyway, whole point of this discussion that working around problems by > violating RFC and re-sending hardly (5xx) refused mail through an ISP is > not > way to go. No, the "whole point" of the discussion is *not* that I want to work around just any 5xx problem. The only "problem" that I am trying to solve is how to detect that the other side is telling me to re-send through my ISP, so that I can oblige. With other types of 5xx, I would of course let the mail bounce as usual. In fact, if the other side replies with "5xx No such user", or "5xx mailbox full", there is absolutely no point in re-sending through the relay. That distinction might not be immediately apparent from the subject of my message, but I thought it was clear enough from the body. Let's say I go to a restaurant and get refused entry because I do not wear a tie. So I go put on a tie, come gack to the restaurant and am now allowed in because I'm no longer violating their dress code. That is, I am willing to put on a tie, but only if/when the restaurant requires it. Otherwise, I prefer to go out eating tie-less, and will do so if the restaurant allows it. How is that a Bad Thing? Translated: if/when some server refuses my mail with a "5xx no dynamics", meaning "please relay through your ISP", I am willing to comply with their access policy. OTOH, if the destination has no problems with direct delivery from a dynamic IP, I prefer to use that method. How is that a Bad Thing? Anyhow, by now I have scripted a solution to do just that. Technically, it works. I'll see what happens in practice. Luc
Re: Fall back to relay after a 5XX reply from destination?
First off, thanks for reading and replying to the real question, as always. > And if they were to accept your email, they may very well deliver > your message to the spam folder, because you are sending from an > inappropriate IP address. True, but there are many _other_ reasons why "they" may think that my message belongs in the spam folder. It is also not unheard of that a destination replies "OK" and yet discards or looses the mail. These things happen all the time. If one can't live with that, better not use e-mail at all. Also, IMHO a spam folder is meant for "in doubt" cases, so it is supposed to be inspected regularly. Upon inspection, "they" can see at first sight that my mail is not spam. At the very least, "they" can then release it. They also could (and IMHO should) use it to improve their hit/miss rate. On the other hand, if people choose to use their spam folder as a black hole, they may loose business. My business, or other people's business, I don't care. Their choice, their problem, not mine. That doesn't mean I don't appreciate your heads-up. It only means that the risk is acceptable - to me and my specific situation. But others may have a different situation (volume, mix of mail etc.) and come to a different conclusion. As an aside, I have also read Ralf Hildebrandt's ten-year old analysis of the issues of running a mail server on a dynamic IP, here: https://www.arschkrebs.de/postfix/postfix_why_dyndns_does_not_work.shtml His chapter on "receiving mail" was very helpful for my own thinking through the risks involved, but here too the probability of his "worst case" scenario is sufficiently low - in my case - to consciously accept the risk of loosing some mail in that case. And implementing a test for the presence of a mail server on my old IP (after the change-over) is now on my to-do list . Luc
Fall back to relay after a 5XX reply from destination?
Hello, I've been running a small volume Postfix mail server on a fixed IP for 15+ years or so. Recently, my provider forced me from ADSL (being phased out here) to VDSL, and I now find myself sending mail from a "dynamic" IP address... As expected, some destinations refuse to accept my outgoing mail with a 550 (usually with a "you're blacklisted" message on top of it). So, I am now looking for some magic that would make Postfix: a) first attempt to deliver outgoing mail straight to the destination MX (as it does now), and, b) if that fails with a "550 Bad IP" (or equivalent), fall back to my ISP relay server and send it through them. These 2 steps would have to happen for each message separately, every time. Even if the destination doesn't like my IP right now, it might accept it in a few hours from now (happened this morning). Ideally, that mechanism should be able to "predict" if it's worth a relay attempt. For example, relaying will (hopefully) fix a "550 This IP is blacklisted", but it won't help with a "550 No such user". The "smtp_fallback_relay" param does not seem suitable for this, as the destination host *can* be found and *is* reachable, it's just being rude to me . I am currently "fixing" it with a transport map for the "rude" domains (such as postfix.org ), but that is only a stopgap measure. If my IP changes next week, other domains may dislike the new one although they had no problems with the one before it. Relaying *all* the outgoing mail through my ISP is *not* an option. As long as the message sits on my own box, I can see what's going on (e.g. greylisting). And if it's handed from here to the destination directly, I have all the details on hand to help them debug it at their end. But I loose all that control when I hand the message over to my ISP. So please, no philosophical arguments, I promise I won't be impressed . To paraphrase: relaying through my ISP *must* be kept to an absolute minimum, along the lines described above. The question is: does Postfix have something like this ? Or do I have to write something myself? If yes, any suggestions? I have been RTFM, peeking in the smtpd.c client source, pondered about a special bounce handler (to detect "bad IP" and forward to my ISP), thought of playing with "smtp_dns_reply_filter" (add the relay server at the end of the "real" MX's) etc., but had no "AHA!" experiences yet. I will continue my search, but I thought I'd ask here as well, just in case this wheel has been invented already. If not, building a new one would certainly be cheaper (for me) than obtaining a fixed IP. If available at all, they cost an arm and a leg over here. Better keep my 2 arms and use them to type some code NB: I'm running Postfix 3.2.0 now. Upgrading to 3.3.1 is planned, but (from the readme) currently as low priority. Many thanks in advance, Luc
