[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
On Sun, Jan 21, 2024 at 06:53:58PM +0100, Simon Hoffmann via Postfix-users wrote: > > This copies only the message headers and body, but fails to capture the > > message envelope, which contains the true recipient list. With > > per-recipient addressing in "recipient_bcc_maps", and provided the > > archive system captures the message envelope, you don't lose that (IMHO > > essential) information. > > > > MAIL FROM: > > RCPT TO: > > RCPT TO: > > DATA > > From: > > To: > > Subject: Let's commit fraud > > > > ... > > . > > > > With "always_bcc" one might assume that the message went to > > "fake-recipient", where in real-life it was sent to "bcc1" and "bcc2". > > With per-recipient bcc addresses that are captured by the archive, > > this is not the case. > > MailStore only states the "always_bcc" approach in their manual > https://help.mailstore.com/en/server/index.php?title=Archiving_Emails_from_Linux-based_Email_Servers#Archiving_Incoming_and_Outgoing_Emails_Directly That's a nuisance to overcome... :-( > AFAIK, Mailstore only support this "basic mode" where it uses the header > information > when importing from generic servers or the specific Exchange/Office365 > Journaling > format. > > Additionally, I have only a single destination email address available > to import into the archive, as each email address specifies which > (customer) archive this mail should be imported as this is a multi > tenant software. With your approach, I would need a catchall at the > archive software if I understand correctly. You'll need to deliver to a proxy delivery agent that converts the BCC recipient addresses to suitable headers, and then reinjects the modified message for delivery to just the single archive address. > We could however try to implement the Exchange Journal format. That's the right solution. > This creates a new email, adds the envelope from and to address to the > body, and then attaches the actual email as an eml attachment. So I > could either write a script as a milter or use a service in master.cf > that calls a script and passes the envelope and mail data to the > script and the script then builds a new email in Exchange Journal > format and transfers it directly to the archive gateway server without > going through postfix again. Correct. This is what what was implemented at a previous employer when I played a role in making archiving work. > If you're bored feel free to send in suggestions for the script :) ^^ > Otherwise I'll use always_bcc for now and try to figure out the script > in a few weeks when I have more time. The always_bcc approach is flawed, so best to implement the encapsulation delivery agent promptly. The recipient_bcc_maps generated recipients become arguments to the script, which just has to prepend a stock header and MIME boundary, output the envelope encoding, then the message, and append the closing MIME boundary. This is rather simple. You don't need to parse the input message. Various-Headers: ... Content-Type: multipart/...; boundary = $longish-random$ --$longish-random$ Content-Type: --$longish-random$ Content-Type: message/rfc822 --$longish-random$-- -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
Viktor Dukhovni wrote: > > > - You may want to "preserve the envelope", by generating a > > > recipient-specific bcc address. Just "always_bcc" fails > > > to record "bcc" recipients, and may record header recipients > > > who weren't actually (envelope) recipients of the message. > > > > could you please elaborate on that (off-list is okay with me), because I > > don't seem > > to understand your point here. > > This copies only the message headers and body, but fails to capture the > message envelope, which contains the true recipient list. With > per-recipient addressing in "recipient_bcc_maps", and provided the > archive system captures the message envelope, you don't lose that (IMHO > essential) information. > > MAIL FROM: > RCPT TO: > RCPT TO: > DATA > From: > To: > Subject: Let's commit fraud > > ... > . > > With "always_bcc" one might assume that the message went to > "fake-recipient", where in real-life it was sent to "bcc1" and "bcc2". > With per-recipient bcc addresses that are captured by the archive, > this is not the case. MailStore only states the "always_bcc" approach in their manual https://help.mailstore.com/en/server/index.php?title=Archiving_Emails_from_Linux-based_Email_Servers#Archiving_Incoming_and_Outgoing_Emails_Directly AFAIK, Mailstore only support this "basic mode" where it uses the header information when importing from generic servers or the specific Exchange/Office365 Journaling format. Additionally, I have only a single destination email address available to import into the archive, as each email address specifies which (customer) archive this mail should be imported as this is a multi tenant software. With your approach, I would need a catchall at the archive software if I understand correctly. We could however try to implement the Exchange Journal format. This creates a new email, adds the envelope from and to address to the body, and then attaches the actual email as an eml attachment. So I could either write a script as a milter or use a service in master.cf that calls a script and passes the envelope and mail data to the script and the script then builds a new email in Exchange Journal format and transfers it directly to the archive gateway server without going through postfix again. If you're bored feel free to send in suggestions for the script :) ^^ Otherwise I'll use always_bcc for now and try to figure out the script in a few weeks when I have more time. Thanks! Cheers, Simon signature.asc Description: PGP signature ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
> I am currently planning to switch from OpenSMTPd to postfix for two reasons > > - smtpd_sender_login_maps functionality not really implemented in OpenSMTPd > - always_bcc not possible on OpenSMTPd Viktor Dukhovni wrote: FWIW, I'd like to recommend "recipient_bcc_maps" over always_bcc. - You will perhaps before long want to make exceptions. On 20.01.24 17:44, Simon Hoffmann via Postfix-users wrote: This is a Company Mail Server, and in Germany there is a rule to copy all mails to a special archive. This my use of always_bcc to copy all mails going through the server to the archive software import mailbox. Further processing is then done on the archive server with keywords, such as keeping Quotes for x years, invoices for y years and so on. I managed this on one server by storing syslog per-mail along with mail archive. It may be not as elegant though. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 42.7 percent of all statistics are made up on the spot. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
On Sat, Jan 20, 2024 at 05:44:25PM +0100, Simon Hoffmann wrote: > > > I am currently planning to switch from OpenSMTPd to postfix for two > > > reasons > > > > > > - smtpd_sender_login_maps functionality not really implemented in > > > OpenSMTPd > > > - always_bcc not possible on OpenSMTPd > > > > FWIW, I'd like to recommend "recipient_bcc_maps" over always_bcc. > > > > - You will perhaps before long want to make exceptions. > > This is a Company Mail Server, and in Germany there is a rule to copy all > mails to a > special archive. This my use of always_bcc to copy all mails going through > the server > to the archive software import mailbox. > > - You may want to "preserve the envelope", by generating a > > recipient-specific bcc address. Just "always_bcc" fails > > to record "bcc" recipients, and may record header recipients > > who weren't actually (envelope) recipients of the message. > > could you please elaborate on that (off-list is okay with me), because I > don't seem > to understand your point here. This copies only the message headers and body, but fails to capture the message envelope, which contains the true recipient list. With per-recipient addressing in "recipient_bcc_maps", and provided the archive system captures the message envelope, you don't lose that (IMHO essential) information. MAIL FROM: RCPT TO: RCPT TO: DATA From: To: Subject: Let's commit fraud ... . With "always_bcc" one might assume that the message went to "fake-recipient", where in real-life it was sent to "bcc1" and "bcc2". With per-recipient bcc addresses that are captured by the archive, this is not the case. > > Postfix tables present a key/value abstraction. A single key in a > > single result out. For file-based hash tables, there is no natural key > > "order", and any given key can match only once. > > But if I [have] two *sources* specified (either two files, or one sql > and a file, ...) postfix processes those sources in the order > specified in main/master.cf, and never looks at the second source if > the first source results in a match (no matter if that match means the > user is allowed to send with that address not not). As expected. > So in general, if I need postfix to query *both* sources, I need to > use a unionmap, correct? Yes, of course. > yes. each user can also have a few aliases, so I would either provide > correct regexps (and would escape a dot if any actual address would > have it) And use "^" and "$" anchors, which is easy to forget for most (mis)users of RE patterns. > or would provide a hash table. That's generally the mor robust and more scalable approach. > > This is more simply expressed as: > > > > static:internal-software-user > > used in a unionmap, right? Wherever you want an output that does not depend on the lookup key, possibly as part of a unionmap, sure. > Sometimes I send notifications to customers directly without using the ticket > system > because they are just notifications and a ticket is only needed if the > customer has > questions and replies to my notification email. But I could achieve that by > stating > > supp...@example.com ticket-system, myuser > > in the hash map, right? As documented. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
Viktor Dukhovni wrote: > > > I am currently planning to switch from OpenSMTPd to postfix for two reasons > > > > - smtpd_sender_login_maps functionality not really implemented in OpenSMTPd > > - always_bcc not possible on OpenSMTPd > > FWIW, I'd like to recommend "recipient_bcc_maps" over always_bcc. > > - You will perhaps before long want to make exceptions. This is a Company Mail Server, and in Germany there is a rule to copy all mails to a special archive. This my use of always_bcc to copy all mails going through the server to the archive software import mailbox. Further processing is then done on the archive server with keywords, such as keeping Quotes for x years, invoices for y years and so on. > > - You may want to "preserve the envelope", by generating a > recipient-specific bcc address. Just "always_bcc" fails > to record "bcc" recipients, and may record header recipients > who weren't actually (envelope) recipients of the message. could you please elaborate on that (off-list is okay with me), because I don't seem to understand your point here. > > So a more "sophisticated" approach is: Thanks for the code/example! > > While reading up on the postfix manual for smtpd_sender_login_maps I've > > read that > > postfix stops at the first match, so if you specify two files you should > > use a > > unionmap or those files may not have any pattern in common. > > Postfix tables present a key/value abstraction. A single key in a > single result out. For file-based hash tables, there is no natural key > "order", and any given key can match only once. But if I two *sources* specified (either two files, or one sql and a file, ...) postfix processes those sources in the order specified in main/master.cf, and never looks at the second source if the first source results in a match (no matter if that match means the user is allowed to send with that address not not). So in general, if I need postfix to query *both* sources, I need to use a unionmap, correct? > > > I have users on the server that should only be allowed to send with > > their own address, but then there is a ticket system that should > > impersonate a few addresses (like sales@, support@, ...). Furthermore > > I have a user to send email from internal software that should be able > > to impersonate all user accounts. > > > > So if I understand correctly, a simple pcre with the following content > > would not be working for me? > > Correct. Thanks for the confirmation! > > > /user@domain.com/ user@domain.com > > /user@domain.com/ user@domain.com > > /user.th...@domain.com/ user.th...@domain.com > > /sa...@domain.com/ zammad-user > > /supp...@domain.com/ zammad-user > > Yet another misuse of regular expressions, I hope that in a real > deployment you'd either write: > > /^user\.one@domain\.com$/ user@domain.com > /^user\.two@domain\.com$/ user@domain.com > /^user\.three@domain\.com$/ user.th...@domain.com > /^sales@domain\.com$/ zammad-user > /^support@domain\.com$/ zammad-user > > or, better still, for exact matches of literals use a "hash" or "cdb" > table: > > user@domain.com user@domain.com > user@domain.com user@domain.com > user.th...@domain.com user.th...@domain.com > sa...@domain.comzammad-user > supp...@domain.com zammad-user > ... yes. each user can also have a few aliases, so I would either provide correct regexps (and would escape a dot if any actual address would have it) or would provide a hash table. The above was just meant as a simplified example to have content in the file to talk about, but to not use actual production data. I should have stated that. > > > /.*/ internal-software-user > > This is more simply expressed as: > > static:internal-software-user used in a unionmap, right? I think when reading the docs the "A table that always returns its name as the lookup result" threw me of because I was still thinking in a "search pattern in first column, result in second column" way and dismissed this then because it would only return one value. But since the whole pcre always only returns one value it's the same thing. Thank you for catching that! :) > > I should also be mention that naturally sender_login_mismatch > presumes a login, so only works at the first hop MSA to which the user's > MUA authenticates. It won't work at a smarthost relay. I am aware of that, thanks, but this server will only be directly connected to by either actual humans and their MUA of choice or by the ticket system and additional internal software. There will never be another server connecting to this server on ports 465/587 and the logins maps will only be applied to these ports. > > > Another alternative I thought of would be to create three separate > > pcre file and use a unionmap. > > Replacing "pcre" with better/safer equivalents as much as
[pfx] Re: Preparation of switch from OpenSMTPd to Postfix -> behaviour of smtpd_sender_login_maps pattern matching
On Sat, Jan 20, 2024 at 03:42:52PM +0100, Simon Hoffmann via Postfix-users
wrote:
> I am currently planning to switch from OpenSMTPd to postfix for two reasons
>
> - smtpd_sender_login_maps functionality not really implemented in OpenSMTPd
> - always_bcc not possible on OpenSMTPd
FWIW, I'd like to recommend "recipient_bcc_maps" over always_bcc.
- You will perhaps before long want to make exceptions.
- You may want to "preserve the envelope", by generating a
recipient-specific bcc address. Just "always_bcc" fails
to record "bcc" recipients, and may record header recipients
who weren't actually (envelope) recipients of the message.
So a more "sophisticated" approach is:
main.cf:
pcre = pcre:${config_directory}/
recipient_bcc_maps = ${pcre}bcc.pcre
bcc.pcre:
if !/\.bcc\.invalid$/
# More specific patterns/exceptions as needed
/(.*)/ ${1}.bcc.invalid
endif
and then deliver all addresses ending in ".bcc.invalid" to the
archive
main.cf:
indexed = ${default_database_type}:${config_directory}/
transport_maps = ${indexed}transport
# No need to "split the envelope" when archiving
archive_recipient_limit = 1000
# Every message goes to the archive, give it more
# concurrency. Ideally, low latency should help avoid
# congestion: throughput = concurrency/latency.
archive_concurrency_limit = 50
transport:
.bcc.invalidarchive:[archive.example]
master.cf:
archive unix ... smtp
# Safety net, nothing should be rejected here...
-o soft_bounce=yes
> While reading up on the postfix manual for smtpd_sender_login_maps I've read
> that
> postfix stops at the first match, so if you specify two files you should use a
> unionmap or those files may not have any pattern in common.
Postfix tables present a key/value abstraction. A single key in a
single result out. For file-based hash tables, there is no natural key
"order", and any given key can match only once.
First match logic is however appropriate for PCRE and CIDR tables, used
e.g. with access(5) where a "DUNNO" result allows for exceptions to a
subsequent general rule.
Bottom line, yes, for pattern-based tables, where multiple keys can
match a result, the first match is natural and least surprising.
> I have users on the server that should only be allowed to send with
> their own address, but then there is a ticket system that should
> impersonate a few addresses (like sales@, support@, ...). Furthermore
> I have a user to send email from internal software that should be able
> to impersonate all user accounts.
>
> So if I understand correctly, a simple pcre with the following content
> would not be working for me?
Correct.
> /user@domain.com/ user@domain.com
> /user@domain.com/ user@domain.com
> /user.th...@domain.com/ user.th...@domain.com
> /sa...@domain.com/ zammad-user
> /supp...@domain.com/ zammad-user
Yet another misuse of regular expressions, I hope that in a real
deployment you'd either write:
/^user\.one@domain\.com$/ user@domain.com
/^user\.two@domain\.com$/ user@domain.com
/^user\.three@domain\.com$/ user.th...@domain.com
/^sales@domain\.com$/ zammad-user
/^support@domain\.com$/ zammad-user
or, better still, for exact matches of literals use a "hash" or "cdb"
table:
user@domain.com user@domain.com
user@domain.com user@domain.com
user.th...@domain.com user.th...@domain.com
sa...@domain.comzammad-user
supp...@domain.com zammad-user
...
> /.*/ internal-software-user
This is more simply expressed as:
static:internal-software-user
I should also be mention that naturally sender_login_mismatch
presumes a login, so only works at the first hop MSA to which the user's
MUA authenticates. It won't work at a smarthost relay.
> Another alternative I thought of would be to create three separate
> pcre file and use a unionmap.
Replacing "pcre" with better/safer equivalents as much as possible, yes.
$ postmap -q foo 'unionmap:{inline:{ {foo = bar} }, inline:{ {foo = baz} },
static:that}'
bar,baz,that
> In the first file I would list all named/personal users with their email
> addresses and their login names,
Using instead simple "hash" or "cdb" tables and "static" as appropriate:
main.cf:
sender_login_maps =
unionmap: {
${indexed}sender-login,
${indexed}ticket-sender,
static:internal-software-user
}
sender-login:
us...@example.com user1
us...@example.com user2
...
ticket-sender:
tick...@example.com ticket-system
tick...@example.com ticket-system
...
without any PCRE tables at all
