per-user usage metering
Hi, Postfix. Long-time fan, first time poster. I need to keep track of per-user use of our SASL-authenticated outbound relay, and to reject mail from users who are exceeding their allowed usage. The records of their usage need to be accessible to me elsewhere over extended durations, although their specific format isn't a huge concern. There is an existing system in place for this, but it's got a serious race condition in it, and I'm not 100% sure that my idea to deal with the problem is a great one. Right now, users authenticate with SASL, and that's fine. The mail then goes through a unix socket policy service via smtpd_sender_restrictions. This looks up the account (based on the sasl_username) and then checks their recent usage in a usage database. If they are over usage, it returns a 450. If they are not over usage, it signals success by prepending a header. Mail with that header is routed to another transport by header_checks. This other transport is responsible for performing a content spam check. If the message is spam, it is sent to an uninteresting destination. If it is not, the message (size, recipients, spam-check score, etc.) is recorded in the usage database and the message is re-injected to its final destination. The race condition is simple: the smtpd can accept a lot of mail before the logging transport can write to the usage database, meaning users can bypass the usage limits. My first moronic attempt to fix this was to move some of the logging to the policy service, and to communicate the record id via the added header to the logging transport, so it could update the record with the spam check score. I had forgotten that the policy service was being queried once *per recipient*, which the obvious problem that each message was logged multiple times. I didn't want to try coordinating based on instance id (incrementing the recipient count each time, etc.) -- and anyway, there is another problem: the mail might pass all the recipient restrictions and then fail during DATA. My current thinking is this: 1. a fast, idempotent policy service will check usage at rcpt time so that we can avoid accepting DATA if the user is over quota; it will signal acceptance with "OK" 2. an end_of_data_restriction will log the recipient count, size, etc; it will signal acceptance by PREPENDing the record identifier 3. the logging transport will still exist, and will do the content checks and update the record with the spam score I'm not sure whether I am worried about the logging done by end_of_data resulting in logging messages that for some reason do not reach the logging transport. In that case, I may mark the records as "pending," with the logging transport marking them "accepted," and another job purging pending records regularly. Does this make sense? Is it a terrible idea? Is this all already covered by some simple interface I have yet to discover? -- rjbs
ignoring bogux MX entries
So, every once in a while, this crops up: $ dig -t mx bollygroup.com ; <<>> DiG 9.4.2-P1 <<>> -t mx bollygroup.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 996 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;bollygroup.com.IN MX ;; ANSWER SECTION: bollygroup.com. 3600IN MX 0 mail. bollygroup.com. 3600IN MX 10 mail.vishalverma.com. These guys have set up "mail." as their lowest distance MX, and Postfix tries to deliver there. It looks up "mail" and finds it under the resolver's search prefix (even though, as far as I understand it, the DNS response is fully qualified to begin with and should not be qualified by the search prefix). This leads to the mail getting stuck in the queue, because the box handling this mail is "mail.prefix.com," and Postfix reports "mail for bollygroup.com loops back to myself" ...but it only does this if you (a) use the bogus "mail." record and (b) qualify "mail." with the search prefix. Is there a way to avoid this problem *without* special casing each domain that has it? That is, to say "don't qualify MX names" or "the MX name 'mail.' is bogus and should be ignored" or anything else? -- rjbs
using/logging client addr as part of SASL auth
Hello! I'm looking for a way to detect and distinguish different kinds of auth failures. Right now, I'm feeling a bit stuck by my inability to get all the data I'd like in one place at the same time. Right now, we're using SASL authentication with pwcheck. pwcheck, of course, only gets two data: username and password. It can't take any action based on the IP address of the remote. Meanwhile, postfix's logs on failure don't appear to show me the username on failed AUTH attempts. I'd like to be able to distinguish the cases resulting from the intersections of (one password over and over / many different passwords), (one username / many usernames), (one IP address, many IP addresses). With these data, I can take better action to detect, classify, and react to bad actors. I'm happy (I guess) to end up having to write code to make this happen, but I'm not sure where I could do it. -- rjbs signature.asc Description: Digital signature
Re: using/logging client addr as part of SASL auth
* Wietse Venema [2014-05-27T17:04:32] > li...@rhsoft.net: > > the problem ist that postfix has no idea of the SASL internals and should > > not need to - in case of dovecot i asked a few days ago to log the username > > because in case of using dovecot as SASL provider that's the only instance > > which decodes the input and verify it against the user-db > > Would not it be sufficient to trigger on repeated authentication > failures, regardless of the login name? It's not ideal. Here are some scenarios: a. one IP, the same username, many different passwords b. one IP, different usernames, many different passwords c. many IPs, the same username, many different passwords d. one IP, the same username, the same (wrong) password repeatedly Scenarios a,b,c are much more likely to be attacks than (d), which looks like somebody just turned on their old laptop without fixing its fetchmail job. Scenario a and b are single-IP bad actors trying to perform a dictionary attack, targeted or not. Scenario c is potentially someone with a botnet trying to get a single target. Taking action based on "repeatedly failed auth from ip X" is a start, but it's not great. I definitely understand the point about not wanting to deal with the SASL internals. Putting aside that question, do you have any suggestions or thoughts about improving the way in which potential attacks could be classified with currently available data? -- rjbs signature.asc Description: Digital signature
Re: using/logging client addr as part of SASL auth
* Wietse Venema [2014-05-27T17:48:03] > Ricardo Signes: > > a. one IP, the same username, many different passwords > > d. one IP, the same username, the same (wrong) password repeatedly > > I suppose that one would log a password hhas, just to be sure. Yes, something like a truncated hash of the password salted with something changed frequently and never stored.It only needs to detect close-in-time "same password" with very low false positives. > It is not practical to implement every SASL protocol in Postfix. > Also, the more secure SASL protocols don't send a fixed password, > instead they use challenge-response. In that case there is no way > to find out whether you are looking at (a.) or (d.). Indeed. Fortunately (?) for me, I don't need to worry about that at this point. > Postfix could log the base64 blobs that the client sends. Yes, I think that, as you suggest, this leads to a scary place. > The general solution requires support in the authentication back-end, > be it Dovecot or the SASL library. I see now that libsasl v2's sasl_server_new takes a string in the form "ipaddr:port" and that postfix's smtpd_sasl_activate seems to pass along the actual remote client IP. (Please tell me if I've grossly misread; I am not familiar with Postfix's source.) As long as we don't put an SMTP proxy in the way, this suggests to me that I can probably do something *fairly* simple with libsasl to get all the relevant data in one place. I see a horrible, useful hack in my future. Thanks for your help. -- rjbs signature.asc Description: Digital signature
more detail on lookup failures?
Lately, I'm seeing a lot more of this from some servers: postfix/cleanup[20330]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[...]" postfix/cleanup[20330]: [ID 947731 mail.warning] warning: 6D0D222182: virtual_alias_maps map lookup problem for [...] -- deferring delivery This then reports "queue file write error", for reasons that seem explained by the source. I'm pretty sure that the problem will lie with "our stuff" and not Postfix, but I'm trying to find out whether I can get more information on what the lookup error was. An incompetent skim over the source makes me think that if the MySQL query was failing, I'd see log lines matching /mysql query failed/, which I am not seeing. (Actually, for the sake of full disclosure, I see it sometimes. When I see it, the failed lookup has been for an email address with a non-7-bit-safe character, and makes sense. It is the *other* failures that are baffling me.) Is there something I can do to get more logging from postfix on this error, before I switch to tracing? -- rjbs signature.asc Description: Digital signature
Re: more detail on lookup failures?
* Wietse Venema [2014-09-04T15:55:07] > Wietse Venema: > > If there is no logfile record with: > > > > warning: mysql query failed: [text from mysql_error() here] > > > > then either your syslog daemon lost that record, > > Yes. Thanks. I'll look harder, but this has happened many, many times. The idea that syslog is losing those records every time seems a bit hard to swallow, but I can probably have dtrace watch the cleanup processes to see what's up for a day or two. -- rjbs signature.asc Description: Digital signature
Re: more detail on lookup failures?
* Wietse Venema [2014-09-04T15:55:07] > No. All match_list errors are logged with the match_error() function. First, let me note that I'm running 2.10.2. Here are all the syslog entries for cleanup for the last half hour on one host that's been having this problem. tl;dr: 32 occurances of mysql map "lookup error"; 32 of "virtual_alias_maps lookup problem"; 7 "query failed", each of which matches up with a non-7-bit-safe recipient. I'm going to go ahead with tracing in not too long, I hope. -- rjbs Sep 4 16:00:07 HOSTNAME postfix/cleanup[58636]: [ID 197553 mail.info] 892EA14BEE: message-id=<> Sep 4 16:00:08 HOSTNAME postfix/cleanup[58588]: [ID 197553 mail.info] 1831814BEF: message-id=<[...]> Sep 4 16:00:16 HOSTNAME postfix/cleanup[58636]: [ID 197553 mail.info] BD48814BF4: message-id=<[...]> Sep 4 16:00:32 HOSTNAME postfix/cleanup[58588]: [ID 197553 mail.info] CC12B14BF5: message-id=<[...]> Sep 4 16:00:35 HOSTNAME postfix/cleanup[58636]: [ID 197553 mail.info] 8C41F14BF6: message-id=<[...]> Sep 4 16:01:08 HOSTNAME postfix/cleanup[58588]: [ID 197553 mail.info] 61DE314BFD: message-id=<> Sep 4 16:02:06 HOSTNAME postfix/cleanup[58636]: [ID 197553 mail.info] AE39414C00: message-id=<> Sep 4 16:02:44 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql query failed: Illegal mix of collations for operation ' IN ' Sep 4 16:02:44 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[8BIT]" Sep 4 16:02:44 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: 1D76114C01: virtual_alias_maps map lookup problem for [8BIT] -- deferring delivery Sep 4 16:02:44 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql query failed: Illegal mix of collations for operation ' IN ' Sep 4 16:02:44 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[8BIT]" Sep 4 16:02:44 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: 258E514C02: virtual_alias_maps map lookup problem for [8BIT] -- deferring delivery Sep 4 16:03:05 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR1]" Sep 4 16:03:05 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: AA12214C05: virtual_alias_maps map lookup problem for [ADDR1] -- deferring delivery Sep 4 16:03:06 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR3]" Sep 4 16:03:06 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: 3C74014C06: virtual_alias_maps map lookup problem for [ADDR3] -- deferring delivery Sep 4 16:03:17 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR6]" Sep 4 16:03:17 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: 8E27214C07: virtual_alias_maps map lookup problem for [ADDR6] -- deferring delivery Sep 4 16:03:19 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR3]" Sep 4 16:03:19 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: 8B34F14C08: virtual_alias_maps map lookup problem for [ADDR3] -- deferring delivery Sep 4 16:03:20 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR6]" Sep 4 16:03:20 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: 9BA2314C09: virtual_alias_maps map lookup problem for [ADDR6] -- deferring delivery Sep 4 16:03:22 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR3]" Sep 4 16:03:22 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: 3F46A14C0A: virtual_alias_maps map lookup problem for [ADDR3] -- deferring delivery Sep 4 16:03:33 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR7]" Sep 4 16:03:33 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: B2CDA14C0B: virtual_alias_maps map lookup problem for [ADDR7] -- deferring delivery Sep 4 16:03:34 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR8]" Sep 4 16:03:34 HOSTNAME postfix/cleanup[58636]: [ID 947731 mail.warning] warning: 1C3A714C12: virtual_alias_maps map lookup problem for [ADDR8] -- deferring delivery Sep 4 16:03:36 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: mysql:/etc/postfix/mysql_list_resend.cf lookup error for "[ADDR3]" Sep 4 16:03:36 HOSTNAME postfix/cleanup[58588]: [ID 947731 mail.warning] warning: 702E914C13: virtual_alias_maps map lookup problem
Yahoo!, DKIM, and Content-Length
I've seen a number of messages about Yahoo! and DMARC failures, but none seem to touch on what I think is the big problem right now. If I missed this happening, I apologize. For some completely inexplicable reason, their DKIM signatures now (often, but not always) look like this: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1412607024; bh=ZJ8Kpz6ZlqWM7sz40HW3fMAm5i4O9s27k2poen3h01U=; h=Received:Received:Received:X-Yahoo-Newman-Property:X-Yahoo-Newman-Id:X-YMail-OSG:Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type:Content-Length:From:Subject; ... Putting everything else aside, THEY SIGNED CONTENT-LENGTH. Unless I am mistaken, cleanup(8) quite reasonably *deletes* Content-Length. This means that any message that Yahoo! has sent with a DKIM signature involving Content-Length is then broken. This means that if Postfix forward mail from Yahoo! to (say) GMail, it will be undeliverable due to DMARC. I assume that I can regenerate Content-Length, if it's correct, but it would be simpler if I could somehow: a) convince Yahoo! to get a grip b) convince cleanup to leave the header be Is one of these possible? Am I mistaken about something significant? Thank you for your time, in advance. -- rjbs signature.asc Description: Digital signature
Re: Yahoo!, DKIM, and Content-Length
* Robert Schetterer [2014-10-06T12:49:09] > > involving Content-Length is then broken. > > hm where in > > http://www.postfix.org/cleanup.8.html > is a delete action documented ? It's documented in the release notes for v1.1. It's mentioned that cleanup(8) does this in the man page for header_checks(5). It appears to be implemented in src/cleanup/cleanup_message.c:cleanup_header_callback but I'm not extremely well-versed in the Postfix source. > gurus may ask this more exactly, but in general it not a good idea to > use "traditional smtp forward" these days by tons of variations of > dkim/spf/dmarc policies. I am well aware of that, but "stop forwarding" is out of the question here. -- rjbs signature.asc Description: Digital signature
Re: PATCH: Yahoo!, DKIM, and Content-Length
* Wietse Venema [2014-10-06T13:22:04] > Good catch. Unfortunately, this behavior is not configurable. > Changing Postfix behavior requires either patching one bit in the > executable file, or rebuilding Postfix from source. Thanks for confirming my suspicion. > By now, maybe is is better to keep this header? I think it probably is, but I find it really frustrating that Yahoo! would not only put it into outbound mail, but also *sign it*. It's utter madness. They already sign a digest of the body! I plan to complain, but I expect nothing to come of it. -- rjbs signature.asc Description: Digital signature
Re: Yahoo!, DKIM, and Content-Length
* Viktor Dukhovni [2014-10-06T13:40:52] > It is also odd that they sign "Received" headers, I thought BCP > was to avoid doing that: Yes, you are correct. The RFC goes on to show examples that use Received, and various strictures about handling trace headers make it plausible that you could use it. I don't even think it's a very bad idea, but it's certainly rare. Note, too, that From and Subject appear in there twice, even though both headers only appear once in the message. Again, this is permissible and can be verified properly, but... what? -- rjbs signature.asc Description: Digital signature
