Re: [nmh-workers] logging outgoing messages
Hi Ken, > My main issue is that this is the mailing list for nmh, not for > Postfix. When people email here with problems that _ARE NOT NMH > PROBLEMS_, but problems with postfix/sendmail/exim/god knows what, > clearly this is the wrong forum. I think it's the right forum for establishing where along the nmh/MTA boundary the problem lies. We're more likely to understand than an MUA/Postfix or MUA/Exim list that doesn't know nmh. And if it's an MTA problem, but with a simple configuration change by the questioner, or referring them to a particular bit of MTA documentation, then I don't think it adds much noise to the list. > I will gladly help people with nmh problems, but not their MTA > problems. Which is perfectly fine; just don't reply. :-) Perhaps others will, particularly those with a similar set up. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>I could not disagree more, and if its for political reasons; >i think that today with TLS plain passwords are all you need, >other cruft should leave codebases as soon as possible. Well, obviously I disagree with that but I will point out that without the right architecture in place if all you support is PLAIN over TLS then doing anything ELSE is hard because you don't have the architecture in place to expand beyond that. Whatever else you say about nmh, at least we have what I feel is a reasonable network security architecture now, with the concept of handling different SASL mechanisms and enough abstraction that if a new SASL mechanism is developed it's easy to add it to all network protocols at once. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Ken Hornstein wrote in <20190710152824.2d9b961...@pb-smtp21.pobox.com>: ... |all of the time? Secondly ... I am seeing more and more authentication |methods that require keeping some kind of state and possibly user |interaction in the MUA (GSSAPI and XOAUTH2 are two examples that I have |personally encountered), and that makes doing authentication in the MTA I could not disagree more, and if its for political reasons; i think that today with TLS plain passwords are all you need, other cruft should leave codebases as soon as possible. The only exception comes with the availability of a system like Kerberos, which can provide you local tickets, with timeouts etc. as requested, shared in between multiple applications in a secure way. I once had "kdestroy -A" in my shell logout file, today i would hook that into my on-lid-close script. Unfortunately i am too stupid to do the real thing and use GELI on FreeBSD aka dm-crypt/LUKS on Linux, ie block level encryption, but even i have an encfs directory which serves my config files, and one encfs loaded once a week which stores the keys. The former includes a PGP encrypted .netrc-style file, which holds all the credentials for Google and my S/MIME keys (my MUA supports "pseudo-hosts" like USER@HOST.smime-cert-key, .smime-cert-cert and .smime-include-certs), and becomes decrypted on the fly. Of course my MUA is still primitive and kees that decrypted stuff in clear, neither does it mprotect() the region nor zeroes that after use. I do not use suspend-to-disk, but still. And it would be better with encfs2, but that will not happen i guess. |very problematic. I think the days of embedded plaintext passwords in |your MTA configuration file are slowly coming to an end. Some kind of shared TLS private key and password service that daemons can use to load such, before they start their privilege- separated childs which only have the readily prepared sessions. And to be unlocked with a Yubikey first. (And with an option to implant that under the skin of an administrators living flesh.) Brave new world. |Like I said in my previous email ... we'll continue to support that. |But I can't recommend it to the average nmh user. | |--Ken | |-- |nmh-workers |https://lists.nongnu.org/mailman/listinfo/nmh-workers --End of <20190710152824.2d9b961...@pb-smtp21.pobox.com> --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>But for the larger issue of whether or not you should submit email to >your own SMTP server or your email provider's ... well, obviously my >OPINION is that you should submit it to your email provider's server >directly from nmh (see previous emails on why I think this). But plenty >of people disagree with me on this, and that's fine. If you're the sort >of person who doesn't have a problem configuring your own SMTP server, >then fine, you should do that! Thank you. Between this and other comments, I've decided to revert to having post communicate directly with my local SMTP server. >But I think recommending that to people is a mistake; it creates the >impression that you need to run your own SMTP server to use nmh, and that >is absolutely not true. Understood. I'm comfortable enough with sendmail that running it on my home system isn't a problem, but I'm well aware that many people would prefer not to have to do that. - Steven -- ___ Steven Winikoff| Montreal, QC, Canada | "Stars are facts; constellations are s...@smwonline.ca | theories." http://smwonline.ca| - Michael F. Flynn -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>>So, yes, lots of people DO rely on giant email providers. Why would >>they not? > >In answer to your (retorical) question... privacy. I mean, yeah ... but I will note Gmail is NOT the only option when it comes to email providers (you will note I am not using gmail). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
In message <20190710150334.894c178...@pb-smtp20.pobox.com>, Ken Hornstein wrote: >So, yes, lots of people DO rely on giant email providers. Why would they >not? In answer to your (retorical) question... privacy. https://www.cnbc.com/2019/07/05/google-gmail-purchase-history-cant-be-deleted.html -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Back to logging outgoing messages. A reason that people want to do this is so that they can track whether emails get delivered or not. Given spam, nobody reads postmaster bounces anymore. (I remember when I got CC's of all bounces...) So three things that I've wanted to implement for awhile (20 years..) are: 1) processing of postmaster bounces, with a way to link back to anno so that the outgoing message can be marked. Yes, bounces come in all sorts of forms, but they always reference the message-id. At one point, I wanted to put something cryptographic in it, but essentially DKIM has done this for us. 2) processing of Return-Receipt-To: messages to anno that the email got delivered. Not a guarantee that they were read, but it does prove they got off your laptop. (whether you queue locally, because... airplanes, or deliver directly) 3) processing of Disposition-Notification-To: which does not work in a lot of MUAs, but where it works, it works hilariously well. Basically, I'm asking for TCP ACK at application layer. I regularly re-edit emails in my outbox with mh-e, although there are issues relating to GPG signatures and some other headers that mean I have to clean stuff up before sending. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Date:Wed, 10 Jul 2019 11:28:20 -0400 From:Ken Hornstein Message-ID: <20190710152824.2d9b961...@pb-smtp21.pobox.com> | But do the users provide you the queue-id? They will (sometimes) provide whatever info has been returned to them, if the queue-id has been included with that, then it will often be included (the user might have no idea what it is, or why it is there, but if the mailer told them that - logged it somewhere that they can find it - then, at least for some users, that's what will be supplied. More commonly though, a single lost message is simply written off. When messages are being regularly "misplaced" though, usually the user will consult the local (or ISP) expert for assistance - that kind of person should know how to monitor the SMTP transaction, and extract the queue-id to send to the managers of the MTA that accepted the mail, to see what happened to it next.If there's a queue-id available for a past lost message, that can save time (and perhaps avoid heisenberg effects). | First ... what unreliable email providers do you guys use that go down | all of the time? It isn't the email servers that go down, but the network link that gets to them - and not necessarily in the "something is broken" sense (though that happens too) but in the "I am not currently connected to he net anywhere, but I want to send e-mail anyway - it can get delivered later, but if I defer sending it I will either forget what I wanted to say, or that I was supposed to reply at all..." [Aside: I run my own MTA, always have, I don't see that ever changing.] kre -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>> But I can't recommend it to the average nmh user. > >Thereby nobbling support for non-nmh email on their Unix system, and >that's a shame if they don't consider that as part of the decision. My main issue is that this is the mailing list for nmh, not for Postfix. When people email here with problems that _ARE NOT NMH PROBLEMS_, but problems with postfix/sendmail/exim/god knows what, clearly this is the wrong forum. I will gladly help people with nmh problems, but not their MTA problems. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Hi Ken, > But do the users provide you the queue-id? Very occasionally on first contact. Those that do clearly know their onions. When roles are reversed, I do. :-) Those that don't get asked to try and supply it if they're likely to return soon. > > I'd keep the 250 as I expect we'd be doing this for 25x and its > > value is significant. Also, there might be multiple lines of 25x. > > Sigh. It's these sorts of things that make stuff spin off the rails. > I am aware that in theory any 2xx code can be returned by the DATA > command, but I think 99.9% of the time it will just be a 250. Assuming you're right, that 0.1% is probably important. > And there's no way we should return a multiline response; Where is the being returned to? What stops it being multi-line? > I think just the last output of the positive repsonse code (after the > 2xx) is fine. In the multi-line responses I've seen, it's the start that's specific followed by a bit of an explanation. > > I agree. Something specialising in queueing locally and then > > forwarding on will do a better job and offer more options than > > extending nmh to include those options over time. I see that nmh > > has to talk TLS SMTP to submit an email locally, e.g. same host or > > company, and that can be used to also submit across the Internet, > > but that doesn't mean it's a good idea... > > First ... what unreliable email providers do you guys use that go down > all of the time? I never said the provider goes down all of the time. And kre suggested when he was in an aeroplane. I mainly pointed out the advantages of centralising MTA configuration in one place, the MTA, given many programs like to send email on Unix. > But I can't recommend it to the average nmh user. Thereby nobbling support for non-nmh email on their Unix system, and that's a shame if they don't consider that as part of the decision. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>>Personally, I'd just suggest keeping the local MTA, having post deliver >>to that, and let it do the logging > >That's exactly what I've always done, from time immemorial until just >about two weeks ago. > >Ironically enough I actually prefer to do it this way, but I was under the >impression that this is deprecated in modern configurations. I'd be happy >to be wrong about that. Let me expand on that a bit. I have zero plans on deprecating support for the sendmail/smtp and sendmail/pipe MTS transports in nmh. I did think about removing spost support (that's how sendmail/pipe USED to work, you used a different post program called 'spost') but a bunch of people spoke up and I figured since there was a need, we should keep it. We did merge the spost into post so support was easier and cleaned up the documentation a fair amount. It's here to stay. Now yes, I still think it's a terrible IDEA to use it; that's why I make jokes about it. However, that's different than the sendmail/smtp and 'pure' smtp interfaces, as they are still speaking SMTP and you can do things there you can't do with sendmail/pipe (in case you are not familiar, sendmail/smtp involves running sendmail with some special flags that let you speak SMTP to it on standard input; many MUAs do not support this). But for the larger issue of whether or not you should submit email to your own SMTP server or your email provider's ... well, obviously my OPINION is that you should submit it to your email provider's server directly from nmh (see previous emails on why I think this). But plenty of people disagree with me on this, and that's fine. If you're the sort of person who doesn't have a problem configuring your own SMTP server, then fine, you should do that! But I think recommending that to people is a mistake; it creates the impression that you need to run your own SMTP server to use nmh, and that is absolutely not true. This is the classic argument when it comes to nmh; where do we fall on the toolbox approach? Is an MUA really a MUA if it cannot send and read email in modern configurations without the help of other tools? I think the answer is 'no', but others have their own opinions. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>That's unfortunate. I've mostly worked with sendmail, and I've never >seen a case where the QID wasn't sent back to the originating MTA, so >I wasn't aware that the RFCs don't require that behaviour. Officially, everything after the "250 " is just ASCII text without any specific format (that's not true for ALL SMTP messages, but it is true for this one). And yes, sending the queue identifier is extremely common, but like I said it's not a requirement (also, we use sendmail and we don't have that specific format of the response code you have). >> As a practical matter I've never had to give anyone the queue >> identifier of a message (because it's not normally logged on the >> client; really, most people are happy with recipients and a time, and >> they are really happy if you have a message-id). > >That doesn't match my experience. I can say when I've done the searching, yes, I figure OUT the queue identifier and use that for searching. But no one I've ever asked for information for ever had any idea what the queue identifier was. And when I've been asked for tracking information for email, no one has ever asked me for the queue identifier. >>I think this should be a lot more generic. So ... an alternate proposal. >> >> [ details snipped for brevity, but the summary is be to create a >> "post hook" and use that instead ] > >I'd have no problem with that as long as the post hook provides the same >information gathered in my patch (i.e., sender and recipient addresses, >message ID, relay server and port, and resulting status and QID). Just so I make it clear ... I'm not proposing writing this myself. I think it would be worthwhile code to add, but if you want this I would recommend you writing it yourself; my to-do list is kind of long. Now if you want some help with that, I'd be glad to provide it (and I'd be willing to write a test for it). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>I grovel around MTA logs on a machine hosting Mailman for all the UK LUG >lists and the queue ID is the key thing. It's the first thing I have to >work out if it's not provided. But do the users provide you the queue-id? That's what I'm really curious about. >I'd keep the 250 as I expect we'd be doing this for 25x and its value is >significant. Also, there might be multiple lines of 25x. Sigh. It's these sorts of things that make stuff spin off the rails. I am aware that in theory any 2xx code can be returned by the DATA command, but I think 99.9% of the time it will just be a 250. And there's no way we should return a multiline response; I think just the last output of the positive repsonse code (after the 2xx) is fine. >Is it just 25x that should trigger this hook, or 4xx, etc., too? To me this hook should only be triggered if the post is successful. None of the other hooks are triggered on failure. >I agree. Something specialising in queueing locally and then forwarding >on will do a better job and offer more options than extending nmh to >include those options over time. I see that nmh has to talk TLS SMTP to >submit an email locally, e.g. same host or company, and that can be used >to also submit across the Internet, but that doesn't mean it's a good >idea... First ... what unreliable email providers do you guys use that go down all of the time? Secondly ... I am seeing more and more authentication methods that require keeping some kind of state and possibly user interaction in the MUA (GSSAPI and XOAUTH2 are two examples that I have personally encountered), and that makes doing authentication in the MTA very problematic. I think the days of embedded plaintext passwords in your MTA configuration file are slowly coming to an end. Like I said in my previous email ... we'll continue to support that. But I can't recommend it to the average nmh user. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>Any rational (MTA) client does. No argument there. >MUA's typically don't, but those should >not really ever be talking to anything but their local MTA. Right, and since nmh is a MUA ... >What is >different now than what used to be true, is what people regard as their >local MTA, which in the past would normally have been either on the same >host, or at absolute worst, in the same organisation as the sender (and >usually same department of the organisation when it is big enough to have >such things). (Organisation can mean household for personal e-mail). >Way too many people today are relying upon "free" giant e-mail providers >to do everything for them. Well, out of the three email providers I deal with on a semi-regular basis, all of which I have a financial relationship with, exactly zero would let me perform final delivery from a MTA that I ran myself (this is enforced via firewall rules and other things like SPF and DKIM records; I suppose in the case of SPF and DKIM it would more mean that the odds are high that if I tried to perform final delivery from my own machines the email would be rejected as spam). So I think it is very very common in the modern Internet that you cannot perform final delivery yourself but need to submit it to a MTA you don't control and have it do it for you. Now, exactly how you DO this is, of course, up to you. All of the email providers I deal with require authentication if you're submitting messages across the global Internet. Having a client MTA perform the necessary authentication requiries extra configuration. And ... well ... configuring an MTA is hard. The nmh mailing list is FULL of people who have problems configuring their MTA. Two people ON THIS THREAD have long delays when they submit email to their local MTA. I don't notice these things because I let someone else deal with it, and it turns out they deal with it just fine (because they get tons of complaints if it doesn't work). So, yes, lots of people DO rely on giant email providers. Why would they not? For most purposes they work just fine, and you don't have to spend your time figuring out why submitting email takes so long. I can type "send" at the What Now? prompt, email gets submitted in less than a second, I get email showing up in my mailbox, and I don't have to think about my postfix configuration at all. This seems like a perfectly reasonable set of affairs. So that's why when people are having problems with the MTA THAT THEY CONTROL, I always advise that they configure nmh to submit directly to their email provider's MTA. Number one, that's something we can provide support for if they run into problems. Number two, it's the way the vast majority of email users are configured so you'll be working with a setup that is the common case and will likely receiver much better support from your email provider. Now, I must be honest and say one downside to this configuration is that you don't get automatic queuing if your email provider is down. But when I think about it, I realize I cannot remember the last time this has happened to me. It turns out email providers have spent a lot of time and money creating redundancy, to the point where if GMail is down it makes the news. So if being able to automatically queue email is a hard requirement for you, then yes, you SHOULD configure a local MTA. But I would humbly suggest that for most people that this is optimizing for the extremely uncommon case. Now in nmh we're going to continue support submitting to your own MTA, be it via SMTP, sendmail/SMTP, or the hated sendmail/pipe interface. So configure your own MTA if you want; we will continue to make that work. But it's clear to me that MOST PEOPLE should not be doing that. A good test would be: if you have to send email to nmh-workers about problems with your MTA configuration, you shouldn't be running one. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>I've just added > >send: -push > >to ~/.mh_profile. I would caution people that we had a user who discovered some of his emails were going nowhere and didn't realize it because he was using -push and mhbuild was erroring out, but he didn't know that because of the use of -push. Specific details here: https://lists.gnu.org/archive/html/nmh-workers/2016-10/msg00233.html Now, THIS user, for reasons I cannot understand, wanted to send 8-bit characters in their drafts but didn't want to specify a non-ASCII character set. But my larger point is that if you're using -push if stuff doesn't work you might not know about it. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>It is MUCH faster than trying to feed the message to Postfix >(aka Sendmail) via SMTP/587 because in the case of just piping the >message, Postfix doesn't make me wait until it has done the >DNS lookups it thinks it needs to do in order to process >the message. I think something is wrong with your setup. I submit over the global Internet, with extra round trips for authentication and encryption, and I see submission times in less than a second (you can see more details here): https://lists.gnu.org/archive/html/nmh-workers/2015-07/msg00019.html --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Hi kre, > The need is less common today, than it once was, since more and more > e-mail is direct from sender's MTA to recipient's - but back when more > mail relaying was done (when there was more than just "the internet") > the queue-id along with the transfer timestamp I grovel around MTA logs on a machine hosting Mailman for all the UK LUG lists and the queue ID is the key thing. It's the first thing I have to work out if it's not provided. The Message-ID field isn't necessarily a one-to-one mapping to QID, as the same email with the same Message-ID can arrive and be queued more than once. > Any rational (MTA) client does. MUA's typically don't, but those > should not really ever be talking to anything but their local MTA. Here, here. > the only rational solution is to log all of the 250 response (after > the 250 itself). I'd keep the 250 as I expect we'd be doing this for 25x and its value is significant. Also, there might be multiple lines of 25x. Is it just 25x that should trigger this hook, or 4xx, etc., too? > Personally, I'd just suggest keeping the local MTA, having post > deliver to that, and let it do the logging - it will also do queueing > in case your local ISP link is down (like you want to send e-mail > while in an airplane...). That is, I wouldn't bother with this at all > - there is an alternative (and better) solution already available. I agree. Something specialising in queueing locally and then forwarding on will do a better job and offer more options than extending nmh to include those options over time. I see that nmh has to talk TLS SMTP to submit an email locally, e.g. same host or company, and that can be used to also submit across the Internet, but that doesn't mean it's a good idea... I run a local Postfix so it has all the configuration about the next smarthost based on the hat I'm wearing, not nmh which, IMO, is the wrong place. Thus mail(1) benefits too. And I'm aware of mhmail(1), but it's not the same. crontab(5)'s MAILTO, at(1)'s LOGNAME, ~/.forward files, ... all benefit from central configuration rather than inside nmh. Ditto fetching email. I don't fetch large emails by default because they're typically spam; the features of the specialist program for fetching emails already provide for that, and lots more; no patching of inc(1) required. :-) -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
I had greylisting turned on and my postfix wasn’t setup quite right so when there was a long cc list, there was a long wait. -push was the easiest way to fix the delay problem! > On Jul 10, 2019, at 1:42 AM, Ralph Corderoy wrote: > > Hi Ronald, > >> It is MUCH faster than trying to feed the message to Postfix (aka >> Sendmail) via SMTP/587 because in the case of just piping the message, >> Postfix doesn't make me wait until it has done the DNS lookups it >> thinks it needs to do in order to process the message. > > I have send(1) submit to the local Postfix and it appears instant. > Bakul pointed out send's -push, but I don't use that and it shouldn't be > necessary. Michael's probably on the right track; check your local > Postfix log for the length of the delay and what occurred around it. > > -- > Cheers, Ralph. > > -- > nmh-workers > https://lists.nongnu.org/mailman/listinfo/nmh-workers -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Hi Ronald, > It is MUCH faster than trying to feed the message to Postfix (aka > Sendmail) via SMTP/587 because in the case of just piping the message, > Postfix doesn't make me wait until it has done the DNS lookups it > thinks it needs to do in order to process the message. I have send(1) submit to the local Postfix and it appears instant. Bakul pointed out send's -push, but I don't use that and it shouldn't be necessary. Michael's probably on the right track; check your local Postfix log for the length of the delay and what occurred around it. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>I agree with that, and even when ifdef's are added, they should be >positive, not double negative, so > #ifndef NOSYSLOG >is just perferse, Of course it is. As I mentioned in my previous message... > #ifdef USE_SYSLOG >would work just as well (it does mean the name needs to be explicitly >defined to get the new code, ...I was just too lazy to do that for a proof of concept. There's no question that you're right if such a patch were to be added in production while using #ifdef > | - It is not clear to me that you can state with certainly that the > | 250 response code will contain the queue identifier > >No, you can't, but these days it almost always does. That matches my experience. >Personally, I'd just suggest keeping the local MTA, having post deliver >to that, and let it do the logging That's exactly what I've always done, from time immemorial until just about two weeks ago. Ironically enough I actually prefer to do it this way, but I was under the impression that this is deprecated in modern configurations. I'd be happy to be wrong about that. - Steven -- ___ Steven Winikoff| Montreal, QC, Canada | Don't use no double negatives. s...@smwonline.ca | http://smwonline.ca| -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>>Is there any interest in adding an improved version of this to the code >>base? > >So ... maybe? But, some thoughts. Thank you (and everyone else!) for taking the time to reply to this. Before I say anything else, I never meant to ask for my patch to be incorporated as-is -- I know there are many ways in which it would need to be improved for production use. I sent it mostly as a proof of concept (it's currently just barely good enough to do what I personally need :-/), and partly in hopes it might help anyone else if something like it isn't added to nmh itself. >- We don't, in general, want to have any more #ifdefs in the code unless > they are completely unavoidable (e.g., operating system differences or > optional third-party libraries like OpenSSL). So this would require > some run-time configuration. Understood, of course. I used those mostly as an easy way to mark the code I added -- and for those wondering why I chose to write them in the negative, that was purely out of laziness (so that I didn't have to add -DSYSLOG to the configure process). Again, this was never intended for production use, and I apologize if I didn't make that clear originally. >- It is not clear to me that you can state with certainly that the > 250 response code will contain the queue identifier (that is, in > fact, not a concept that appears anywhere that I can find in the SMTP > RFCs). That's unfortunate. I've mostly worked with sendmail, and I've never seen a case where the QID wasn't sent back to the originating MTA, so I wasn't aware that the RFCs don't require that behaviour. > As a practical matter I've never had to give anyone the queue > identifier of a message (because it's not normally logged on the > client; really, most people are happy with recipients and a time, and > they are really happy if you have a message-id). That doesn't match my experience. >I think this should be a lot more generic. So ... an alternate proposal. > > [ details snipped for brevity, but the summary is be to create a > "post hook" and use that instead ] I'd have no problem with that as long as the post hook provides the same information gathered in my patch (i.e., sender and recipient addresses, message ID, relay server and port, and resulting status and QID). - Steven -- ___ Steven Winikoff| "...and every single one of them wanted Montreal, QC, Canada | to be involved in the decision-making s...@smwonline.ca | process without necessarily going http://smwonline.ca| through the intelligence-using process | first." - Terry Pratchett -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
On Jul 9, 2019, at 5:56 PM, Ronald F. Guilmette wrote: > > In message <20190710004749.89c1b163...@pb-smtp1.pobox.com>, > Ken Hornstein wrote: > >> If I could make sendmail/pipe punch the user in the face every time a >> message was sent using it... > > Please don't. I'm using it. > > It is MUCH faster than trying to feed the message to Postfix > (aka Sendmail) via SMTP/587 because in the case of just piping the > message, Postfix doesn't make me wait until it has done the > DNS lookups it thinks it needs to do in order to process > the message. I've just added send: -push to ~/.mh_profile. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Ronald F. Guilmette wrote: >> If I could make sendmail/pipe punch the user in the face every time a >> message was sent using it... > Please don't. I'm using it. > It is MUCH faster than trying to feed the message to Postfix > (aka Sendmail) via SMTP/587 because in the case of just piping the > message, Postfix doesn't make me wait until it has done the > DNS lookups it thinks it needs to do in order to process > the message. Then you are missing an entry in /etc/hosts for 127.0.0.1 and ::1, or you are missing your hostname in that file. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
In message <20190710004749.89c1b163...@pb-smtp1.pobox.com>, Ken Hornstein wrote: >If I could make sendmail/pipe punch the user in the face every time a >message was sent using it... Please don't. I'm using it. It is MUCH faster than trying to feed the message to Postfix (aka Sendmail) via SMTP/587 because in the case of just piping the message, Postfix doesn't make me wait until it has done the DNS lookups it thinks it needs to do in order to process the message. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Date:Tue, 09 Jul 2019 19:39:08 -0400 From:Ken Hornstein Message-ID: <20190709233912.db2aa73...@pb-smtp20.pobox.com> | - We don't, in general, want to have any more #ifdefs in the code unless | they are completely unavoidable I agree with that, and even when ifdef's are added, they should be positive, not double negative, so #ifndef NOSYSLOG is just perferse, where #ifdef USE_SYSLOG would work just as well (it does mean the name needs to be explicitly defined to get the new code, but that's the way it should be, rather than requiring a new name to be defined to retain the previous behaviour). | - It is not clear to me that you can state with certainly that the | 250 response code will contain the queue identifier No, you can't, but these days it almost always does. | As a practical matter I've never had to give anyone the queue | identifier of a message Then you have been lucky, are are relatively new to e-mail tracking. The need is less common today, than it once was, since more and more e-mail is direct from sender's MTA to recipient's - but back when more mail relaying was done (when there was more than just "the internet") the queue-id along with the transfer timestamp was the info that you could send to a relay postmaster and actually expect them to look and see what happened to the message (because that generally makes it trivial for them to do ... send the message-id instead, or worse, just addresses, and it was typically a lot more work, which resulted in requests for tracking sometimes simply being ignored). | (because it's not normally logged on the client; Any rational (MTA) client does. MUA's typically don't, but those should not really ever be talking to anything but their local MTA. What is different now than what used to be true, is what people regard as their local MTA, which in the past would normally have been either on the same host, or at absolute worst, in the same organisation as the sender (and usually same department of the organisation when it is big enough to have such things). (Organisation can mean household for personal e-mail). Way too many people today are relying upon "free" giant e-mail providers to do everything for them. | really, most people are happy with recipients and a time, and | they are really happy if you have a message-id). time yes, recipients, no, that's close to useless (though it can work if the recipient receives very little e-mail and the site postmaster is in a helpful mood at the time a request is made). The message-id is better, but the queue-id (along with time) is perfect. (With a message-id (with or without the time) the first thing that normally needs doing to track a message is to convert that into the relevant local queue-id - avoiding the need for that step makes it just that much more likely that a message can, and will, be traced.) | But there's no way we could | accept this patch as-is, because it depends on the specific format of | your ISP's response message. Yes, that's no good - the only rational solution is to log all of the 250 response (after the 250 itself). It is not only the queue-id in it which can be helpful, sometimes the wording of the response can give more info to the site which issued it (when you ask them to see what happened) - it can differ for mail which was queued there, immediately forwarded before being ack'd, or which was delivered locally. | It looks like your ISP's format is "250 id=QUEUE-ID". So that's 3 | servers and 3 different formats. I used to have my mailer say something like 250 OK Receipt: queue-id to make it clear that that was the info that I wanted sent to me if you wanted me to track your e-mail. Not sure if I still do that or not, it has been a while since munnari did any significant e-mail relaying (it used to connect 5 or 6 different e-mail environments). | I think this should be a lot more generic. So ... an alternate proposal. Personally, I'd just suggest keeping the local MTA, having post deliver to that, and let it do the logging - it will also do queueing in case your local ISP link is down (like you want to send e-mail while in an airplane...). That is, I wouldn't bother with this at all - there is an alternative (and better) solution already available. kre -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>Could we log the entire result, and let the post hook take care of the >various queue formats? That was what I suggested. Clearly nmh shouldn't be in the business of figuring out what (if any) the queue identifier is based on the SMTP DATA response message. >> I am neutral about this being made to work with sendmail/pipe; it would >> actually be a lot of work to do that. We could just accept that it is >> one more thing that doesn't work with sendmail/pipe. > >slowly, that interface is dying. I have mixed feelings about that. If I could make sendmail/pipe punch the user in the face every time a message was sent using it, I would. Sadly, that does not seem possible given the current constraints of POSIX. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Ken Hornstein wrote: > - It is not clear to me that you can state with certainly that the > 250 response code will contain the queue identifier (that is, in > fact, not a concept that appears anywhere that I can find in the SMTP > RFCs). As a practical matter I've never had to give anyone the queue > identifier of a message (because it's not normally logged on the I have both asked for it and provided it when debugging bizarre situations. Could we log the entire result, and let the post hook take care of the various queue formats? > I am neutral about this being made to work with sendmail/pipe; it would > actually be a lot of work to do that. We could just accept that it is > one more thing that doesn't work with sendmail/pipe. slowly, that interface is dying. I have mixed feelings about that. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
On Tue, 09 Jul 2019 17:43:06 -0400, Steven Winikoff said:
> sm_reply.length = rp - sm_reply.text;
> sm_reply.text[sm_reply.length] = 0;
> +#ifndef NOSYSLOG
> +if (strncmp(sm_reply.text, "OK id=", 6) == 0)
> +{
This is highly dependent on the remote MTA.
Google, for instance, returns:
250 2.0.0 OK 1562718444 r17sm180161qtf.26 - gsmtp
The 250 is required. 2.0.0 (or similar) if the remote server does extended
return codes. The rest is up for grabs.
pgp9KL4F2M5Mn.pgp
Description: PGP signature
--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>Is there any interest in adding an improved version of this to the code >base? So ... maybe? But, some thoughts. - We don't, in general, want to have any more #ifdefs in the code unless they are completely unavoidable (e.g., operating system differences or optional third-party libraries like OpenSSL). So this would require some run-time configuration. - It is not clear to me that you can state with certainly that the 250 response code will contain the queue identifier (that is, in fact, not a concept that appears anywhere that I can find in the SMTP RFCs). As a practical matter I've never had to give anyone the queue identifier of a message (because it's not normally logged on the client; really, most people are happy with recipients and a time, and they are really happy if you have a message-id). But ... fine. I can imagine cases where it would be helpful. But there's no way we could accept this patch as-is, because it depends on the specific format of your ISP's response message. I tested out two email providers that I use and their formats differ, specifically: 250 2.0.0 Ok: queued as QUEUE-ID 250 2.0.0 QUEUE-ID Message accepted for delivery It looks like your ISP's format is "250 id=QUEUE-ID". So that's 3 servers and 3 different formats. I think this should be a lot more generic. So ... an alternate proposal. Right now we have the hooks interface (see doc/nmh/README-HOOKS). It should be relatively straightforward to create a "post hook" that could be invoked when email is submitted to a mail server. One of the arguments to the post hook could be the response message from the SMTP server. Another one of the arguments to the post hook could be the message draft so you could interrogate it with scan(1) to extract out anything useful you might want like the message-id. You could then use your favorite shell tools to process the SMTP server response and then send it to syslog with logger(1). I realize that would be a lot more work than the code you submitted, and we'd have to think about the arguments to the post hook, but I don't see any huge blockers other than WRITING the code. I am neutral about this being made to work with sendmail/pipe; it would actually be a lot of work to do that. We could just accept that it is one more thing that doesn't work with sendmail/pipe. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] logging outgoing messages
I recently modified my configuration for nmh-1.7.1 to connect directly to
my ISP's sendmail, rather than going through sendmail on my desktop Linux
system.
This works perfectly, but as a side effect I lost all logging of outgoing
messages. This isn't the end of the world, but it's a pain because there
are times when outgoing messages may need to be traced, and in cases like
that it's important to be able to quote the ISP's assigned QID.
...so I hacked the appended patch together, and it seems to work, but I'm
sure there must be a better way to do this. Problems with my code include
(but are almost certainly not limited to) the following:
- It doesn't check to see whether it's connecting to a local sendmail
(if so, each message will be logged twice).
- It logs successful delivery, but not delivery failures (no matter
the reason).
- It seems to break two of the tests in make check:
# make check
[...]
PASS: test/post/test-messageid
send: message not delivered to anyone
FAIL: test/post/test-mts
send: message not delivered to anyone
FAIL: test/post/test-post-aliases
PASS: test/post/test-post-basic
[...]
===
2 of 112 tests failed
Please report to nmh-workers@nongnu.org
===
Makefile:4763: recipe for target 'check-TESTS' failed
make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory '/big/local/pkg/nmh/nmh-1.7.1'
Makefile:5019: recipe for target 'check-am' failed
Is there any interest in adding an improved version of this to the code
base?
- Steven
8<- cut here >8
--- h/mts.h.original2017-11-22 09:37:56.0 -0500
+++ h/mts.h 2019-07-09 16:51:34.260314328 -0400
@@ -57,3 +57,15 @@
* Global MailDelivery File
*/
extern char *maildelivery;
+
+#ifndef NOSYSLOG
+# define SYSLOG_FIELD_SIZE 1024
+# define SYSLOG_FIELD_LAST 1023
+
+ char syslog_from [SYSLOG_FIELD_SIZE];
+ char syslog_to[SYSLOG_FIELD_SIZE];
+ char syslog_msgid [SYSLOG_FIELD_SIZE];
+ char syslog_server[SYSLOG_FIELD_SIZE];
+ char syslog_port [SYSLOG_FIELD_SIZE];
+ char syslog_qid [SYSLOG_FIELD_SIZE];
+#endif
--- mts/smtp/smtp.c.original2018-03-06 14:05:55.0 -0500
+++ mts/smtp/smtp.c 2019-07-09 17:17:34.382593987 -0400
@@ -13,6 +13,9 @@
#include
#include
+#ifndef NOSYSLOG
+ #include
+#endif
/*
* This module implements an interface to SendMail very similar
@@ -74,6 +77,24 @@
int debug, int sasl, const char *saslmech, const char *user,
const char *oauth_svc, int tls)
{
+#ifndef NOSYSLOG
+/** ensure fields are properly initialized to empty strings: **/
+
+syslog_from [0] = '\0';
+syslog_to [0] = '\0';
+syslog_msgid[0] = '\0';
+syslog_qid [0] = '\0';
+
+
+/** ...except server and port, which can take their real values: **/
+
+(void)strncpy(syslog_server, server, SYSLOG_FIELD_SIZE);
+syslog_server[SYSLOG_FIELD_LAST] = '\0';
+
+(void)strncpy(syslog_port, port, SYSLOG_FIELD_SIZE);
+syslog_port[SYSLOG_FIELD_LAST] = '\0';
+#endif
+
if (sm_mts == MTS_SMTP)
return smtp_init (client, server, port, watch, verbose,
debug, sasl, saslmech, user, oauth_svc, tls);
@@ -454,6 +475,13 @@
}
}
+#ifndef NOSYSLOG
+/** record from field for syslog: **/
+
+(void)strncpy(syslog_from, from, SYSLOG_FIELD_SIZE);
+syslog_from[SYSLOG_FIELD_LAST] = '\0';
+#endif
+
switch (smtalk (SM_MAIL, "MAIL FROM:<%s>%s", from, mail_parameters)) {
case 250:
sm_addrs = 0;
@@ -473,6 +501,37 @@
int
sm_wadr (char *mbox, char *host, char *path)
{
+#ifndef NOSYSLOG
+{
+ /** record recipient address for syslog: **/
+
+ char address[SYSLOG_FIELD_SIZE];
+ int length, used, remaining;
+
+ if (host && *host)
+ (void)snprintf(address, SYSLOG_FIELD_SIZE, "<%s%s@%s>",
+ FENDNULL(path), mbox, host);
+ else
+ (void)snprintf(address, SYSLOG_FIELD_SIZE, "<%s%s>",
+ FENDNULL(path), mbox);
+
+ length= strlen(address);
+ used = strlen(syslog_to);
+ remaining = (SYSLOG_FIELD_SIZE - used) - 1;
+
+ if ((used == 0) && (length < remaining))
+ {
+ (void)strcat(syslog_to, address);
+ }
+ else if ((length + 1) < remaining)
+ {
+ (void)strcat(syslog_to, " ");
+ (void)strcat(syslog_to, address);
+ }
+
+ syslog_to[SYSLOG_FIELD_LAST] = '\0';
+}
+#endif
switch (smtalk (SM_RCPT, host && *host ? "RCPT TO:<%s%s@%s>"
: "RCPT TO:<%s%s>",
FENDNULL(path), mbox, host)) {
@@ -717,6 +776,19 @@
