Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 09/22/2017 12:47 PM, Michael Fox wrote: > Option 3: Use unix socket in Postfix chroot jail This looks to be what I do. I'm running Postfix and MIMEDefang on Ubuntu, both from packages. Postfix runs as the postfix user, and there's a defang group. I run Postfix in a chroot. These appear to be the relevant parts of my install script: adduser --quiet postfix defang install -d -o defang -g defang -m 750 \ /var/spool/postfix/var/spool/MIMEDefang chown -R defang:defang \ /var/lib/MIMEDefang \ /var/spool/MIMEDefang \ /var/spool/postfix/var/spool/MIMEDefang sed -i 's|^\(# \)\?\(SOCKET\)=.*|\2=/var/spool/postfix/var/spool/MIMEDefang/mimedefang.sock|' \ /etc/default/mimedefang I believe we have some sort of trouble if one of the daemons is restarted, but not the other, or if it's done in the wrong order or something. I don't have a lot of specifics off the top of my head. In practice, we hardly ever restart one or the other. It's usually either stopping both (and starting MIMEDefang first, to give slaves a chance to spin up), or rebooting the server. I hope this helps. If you have specific questions, I'll try to dig into my config if I can. I'm currently out of the office, though. -- Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
Thanks Richard, BTW, I also installed from packages. So, with a unix socket you still have the same problem I do with the inet socket, which is: the order of start/restart matters and can break things. That's not good. This seems like a clear bug in MIMEDefang. Evidence: 1) Other milter(s) share a socket with Postfix and don't care which order they are started/restarted (example: OpenDKIM) 2) Dovecot shares a socket with Postfix and doesn't care which order they are started/restarted 3) Amavisd-new is a different mechanism, but it also doesn't care which order it is started/restarted 4) And even MIMEDefang didn't care about the order in v2.73 I'm new to the list. What's the process for reporting/resolving issues? Thanks, Michael > -Original Message- > From: Richard Laager [mailto:rlaa...@wiktel.com] > Sent: Friday, September 22, 2017 9:51 PM > To: Michael Fox > Cc: mimedefang@lists.roaringpenguin.com > Subject: Re: [Mimedefang] REVISED: postfix/mimedefang socket > > On 09/22/2017 12:47 PM, Michael Fox wrote: > > Option 3: Use unix socket in Postfix chroot jail > > This looks to be what I do. I'm running Postfix and MIMEDefang on > Ubuntu, both from packages. Postfix runs as the postfix user, and > there's a defang group. I run Postfix in a chroot. > > These appear to be the relevant parts of my install script: > > adduser --quiet postfix defang > > install -d -o defang -g defang -m 750 \ > /var/spool/postfix/var/spool/MIMEDefang > > chown -R defang:defang \ > /var/lib/MIMEDefang \ > /var/spool/MIMEDefang \ > /var/spool/postfix/var/spool/MIMEDefang > > sed -i 's|^\(# > \)\?\(SOCKET\)=.*|\2=/var/spool/postfix/var/spool/MIMEDefang/mimedefang.so > ck|' \ > /etc/default/mimedefang > > I believe we have some sort of trouble if one of the daemons is > restarted, but not the other, or if it's done in the wrong order or > something. I don't have a lot of specifics off the top of my head. In > practice, we hardly ever restart one or the other. It's usually either > stopping both (and starting MIMEDefang first, to give slaves a chance to > spin up), or rebooting the server. > > I hope this helps. If you have specific questions, I'll try to dig into > my config if I can. I'm currently out of the office, though. > > -- > Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Fri, 22 Sep 2017 23:59:38 -0700 "Michael Fox" wrote: > This seems like a clear bug in MIMEDefang. Not at all. This problem is not present with Sendmail, so I have no idea what would cause it with Postfix. Removing the MIMEDefang socket prior to starting MIMEDefang should fix this. The sample init script given with MIMEDefang does exactly this. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 22 Sep 2017, at 13:47, Michael Fox wrote: Sorry if this has been asked/answered before. I've searched and searched and found no consistent, complete answers. What are the steps to configure MIMEDefang and Postix to share a socket? The workable solutions depend on your OS (and with Linux, your distribution,) your filesystem, and whether you chroot the Postfix smtpd program. Here's what I tried: Option 1: Use inet socket /etc/default/mimedefang: SOCKET=inet:8899@localhost /etc/postfix/main.cf smtpd_milters = inet:localhost:8899 This works for Ubuntu 14.04/MIMEDefang 2.73. But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already running, and MIMEDefang is restarted, I get the following in /var/log/mail.err: mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port inet:8899@localhost: Address already in use mimedefang[27605]: MIMEDefang-2.78: Unable to create listening socket on conn inet:8899@localhost That's odd. Given your other options, I'd try to figure out what exactly is going on here and fix it. I.e. what process actually has taken that port. As for using a unix socket, you don't need to create the socket inside the chroot if it's on the same filesystem as /var/spool/MIMEDefang/ , you can just hardlink the existing socket. Also, you may not need to add the postfix user to the defang group, if you have a filesystem that supports BSD default group semantics for setgid directories, e.g.: chgrp postfix /var/spool/MIMEDefang/ chmod g+ws /var/spool/MIMEDefang/ With the setgid bit set on a directory, newly created files (or sockets) take their group from the parent directory rather than the creating process. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
Thanks Bill, After some more research I found the bug and I think I've "fixed" it. See below. > > Option 1: Use inet socket > > > > /etc/default/mimedefang: > > SOCKET=inet:8899@localhost > > > > /etc/postfix/main.cf > > smtpd_milters = inet:localhost:8899 > > > > This works for Ubuntu 14.04/MIMEDefang 2.73. > > But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already > > running, and MIMEDefang is restarted, I get the following in > > /var/log/mail.err: > > > > mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port > > inet:8899@localhost: Address already in use > > mimedefang[27605]: MIMEDefang-2.78: Unable to create listening > > socket on conn inet:8899@localhost > > That's odd. Given your other options, I'd try to figure out what exactly > is going on here and fix it. I.e. what process actually has taken that > port. Yes, odd. I've continued to research and just found this, which is my problem. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807078 The problem occurs for me in MIMEDefang 2.78 (package for Ubuntu 16.04), which is the version reported in the above bug. Apparently, the problem is that when using "systemctl restart mimedefang" it doesn't wait for itself to completely stop before it tries to start again. And I can confirm that. If I use "systemctl stop ...", then wait for it to stop, then "systemctl start ...", then it works. The bug report says it's fixed in v2.79 and v2.79 is in the Ubuntu 17.04 package. According to https://ubuntu.pkgs.org/17.04/ubuntu-universe-i386/mimedefang_2.79-2_i386.deb.html , on 2016-12-23 the init stop action was changed to always wait. So here's what I did to "fix" it in Ubuntu 16.04/MIMEDefang 2.78: In /etc/init.d/mimedefang, in the stop_it() function: #if [ "$1" = "wait" ] ; then #fi In other words, just comment out the conditional wait so that the "waiting" part inside the IF statement always runs. When I tried to run systemctl restart mimedefang, I got: Warning: mimedefang.service changed on disk. Run 'systemctl daemon-reload' to reload units. So, I did that. And then: systemctl restart mimedefang seems to work just fine. I'd appreciate it if any of the developers can verify that what I did shouldn't cause some other unintended consequence. Thanks much, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 23 Sep 2017, at 20:37, Michael Fox wrote: Thanks Bill, After some more research I found the bug and I think I've "fixed" it. See below. Option 1: Use inet socket /etc/default/mimedefang: SOCKET=inet:8899@localhost /etc/postfix/main.cf smtpd_milters = inet:localhost:8899 This works for Ubuntu 14.04/MIMEDefang 2.73. But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already running, and MIMEDefang is restarted, I get the following in /var/log/mail.err: mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port inet:8899@localhost: Address already in use mimedefang[27605]: MIMEDefang-2.78: Unable to create listening socket on conn inet:8899@localhost That's odd. Given your other options, I'd try to figure out what exactly is going on here and fix it. I.e. what process actually has taken that port. Yes, odd. I've continued to research and just found this, which is my problem. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807078 The problem occurs for me in MIMEDefang 2.78 (package for Ubuntu 16.04), which is the version reported in the above bug. Apparently, the problem is that when using "systemctl restart mimedefang" it doesn't wait for itself to completely stop before it tries to start again. And I can confirm that. If I use "systemctl stop ...", then wait for it to stop, then "systemctl start ...", then it works. I kinda wanted to blame the abomination of systemd and/or the Debian/Ubuntu packagers who regularly 'fix' things that are not broken but since I had nothing more solid than my own bile, I refrained... The bug report says it's fixed in v2.79 and v2.79 is in the Ubuntu 17.04 package. According to https://ubuntu.pkgs.org/17.04/ubuntu-universe-i386/mimedefang_2.79-2_i386.deb.html , on 2016-12-23 the init stop action was changed to always wait. As the "restart" action has been in the distribution's included init scripts for as long as I've used MD (over a decade) and I suspect forever. There's no rationale I can think of to always wait on a 'stop' action but it isn't really harmful, beyond delaying shutdown for (normally) a few seconds. So here's what I did to "fix" it in Ubuntu 16.04/MIMEDefang 2.78: In /etc/init.d/mimedefang, in the stop_it() function: #if [ "$1" = "wait" ] ; then #fi In other words, just comment out the conditional wait so that the "waiting" part inside the IF statement always runs. When I tried to run systemctl restart mimedefang, I got: Warning: mimedefang.service changed on disk. Run 'systemctl daemon-reload' to reload units. So, I did that. And then: systemctl restart mimedefang seems to work just fine. I'd appreciate it if any of the developers can verify that what I did shouldn't cause some other unintended consequence. Dianne will likely give a definitive authoritative answer but having adapted the MD startup script to multiple system control environments, I don't see anything especially risky. The only potential issue I can think of is if you're trying to shut down or reboot in a pathological circumstance where the mimedefang.pl worker perl sub-processes refuse to exit and the supervisory process (e.g. init) has no last resort mechanism to arbitrarily slaughter hung children. There's nothing quite so annoying as a "hung going down" system. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> Dianne will likely give a definitive authoritative answer but having > adapted the MD startup script to multiple system control environments, I > don't see anything especially risky. The only potential issue I can > think of is if you're trying to shut down or reboot in a pathological > circumstance where the mimedefang.pl worker perl sub-processes refuse to > exit and the supervisory process (e.g. init) has no last resort > mechanism to arbitrarily slaughter hung children. There's nothing quite > so annoying as a "hung going down" system. Thanks Bill, That makes sense. One option would be for the MIMEDefang team to provide the necessary systemd control files (whatever they're called). That way, they could define restart any way they like, such as "stop wait" followed by "start". I know systemd is hated by many. But it is what Ubuntu 16.04 LTS uses. Without the proper files, apparently what happens is systemd auto-generates the file(s) it needs from the init script. And in the autogenerated file, a 'restart' is simply a 'stop' followed by a 'start'. I guess they can depend on at least those two basic commands being in every init script. And because 'stop' doesn't wait unless the added "wait" option is given, we have the problem. A friend of mine found the actual "fix" that was submitted. It's a bit different than what I did. Instead of commenting out the if statement in stop_it() like I did, they changed the line that calls stop_it from "stop_ip $2" to "stop_it wait", so stop_it is always called with the "wait" option. See below. Same result - the (formerly conditional) wait part always runs. 1 line changed instead of 2. Michael *** mimedefang-16.042017-09-23 17:53:51.046863700 -0700 --- mimedefang-17.042017-09-23 17:50:18.456864400 -0700 *** *** 412,418 ;; stop) ! stop_it $2 ;; restart|force-reload) --- 412,418 ;; stop) ! stop_it wait ;; restart|force-reload) ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 24 Sep 2017, at 17:13, Michael Fox wrote: Dianne will likely give a definitive authoritative answer but having adapted the MD startup script to multiple system control environments, I don't see anything especially risky. The only potential issue I can think of is if you're trying to shut down or reboot in a pathological circumstance where the mimedefang.pl worker perl sub-processes refuse to exit and the supervisory process (e.g. init) has no last resort mechanism to arbitrarily slaughter hung children. There's nothing quite so annoying as a "hung going down" system. Thanks Bill, That makes sense. One option would be for the MIMEDefang team to provide the necessary systemd control files (whatever they're called). They are called "unit files" but including one in the base code would not help. That way, they could define restart any way they like, such as "stop wait" followed by "start". Nope. A service definition in a unit file has an ExecStart definition and maybe an ExecStop definition, but there is no support in systemd for an ExecRestart attribute. It's a design flaw in systemd, which ALWAYS implements "systemctl restart" as a stop then a start. I know systemd is hated by many. Don't get me started... But it is what Ubuntu 16.04 LTS uses. So it's the Ubuntu/Debian package maintainer's job to create a suitable unit file just as they've created their own init script. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> > That way, they could define restart any way they like, such as "stop > > wait" followed by "start". > > Nope. A service definition in a unit file has an ExecStart definition > and maybe an ExecStop definition, but there is no support in systemd for > an ExecRestart attribute. It's a design flaw in systemd, which ALWAYS > implements "systemctl restart" as a stop then a start. Ah. I guess I was thinking of reload. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 09/25/2017 12:14 AM, Bill Cole wrote: > A service definition in a unit file has an ExecStart definition > and maybe an ExecStop definition ExecStop must not return until the service has actually stopped: https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStop= There are a couple ways to address this. As discussed, the init script can be changed to always wait on stop. Then the automatic mapping in systemd should Just Work. Otherwise, as discussed, we need to create a unit file. For a temporary measure, if one is willing to depend on the SysV init script, it should be possible to do something simple like: ExecStop=/etc/init.d/mimedefang stop ; /etc/init.d/mimedefang wait The fully native approach would be to include (either directly, or in a small helper script) the wait loop (but not the status nor the killing bits, as systemd handles those things). Unless someone beats me to it, I'll whip up a systemd unit file in the next few days, test it, and submit a patch to Debian. -- Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Mon, 25 Sep 2017 16:08:46 -0500 Richard Laager wrote: > There are a couple ways to address this. As discussed, the init script > can be changed to always wait on stop. Then the automatic mapping in > systemd should Just Work. The sample init script that ships with MIMEDefang does wait on stop. If you have a script that doesn't, that's a distribution bug and not a MIMEDefang bug and it's definitely more productive to take this up with the distribution maintainer. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 26 Sep 2017, at 9:24, Dianne Skoll wrote: On Mon, 25 Sep 2017 16:08:46 -0500 Richard Laager wrote: There are a couple ways to address this. As discussed, the init script can be changed to always wait on stop. Then the automatic mapping in systemd should Just Work. The sample init script that ships with MIMEDefang does wait on stop. If you have a script that doesn't, that's a distribution bug and not a MIMEDefang bug and it's definitely more productive to take this up with the distribution maintainer. What am I missing? This is in a pristine MD distribution tree: bigsky:mimedefang-2.82 bill$ grep -A3 'stop)' examples/init-script.in redhat/mimedefang-init.in examples/init-script.in: stop) examples/init-script.in- stop_it $2 examples/init-script.in-;; examples/init-script.in- -- redhat/mimedefang-init.in:stop) redhat/mimedefang-init.in- stop $2 redhat/mimedefang-init.in- RETVAL=$? redhat/mimedefang-init.in- ;; ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Tue, 26 Sep 2017 09:38:18 -0400 "Bill Cole" wrote: > bigsky:mimedefang-2.82 bill$ grep -A3 'stop)' The stop_it function in init-script.in and the stop function in redhat/mimedefang-init.in both wait for the daemons to exit. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 26 Sep 2017, at 11:53, Dianne Skoll wrote: On Tue, 26 Sep 2017 09:38:18 -0400 "Bill Cole" wrote: bigsky:mimedefang-2.82 bill$ grep -A3 'stop)' The stop_it function in init-script.in and the stop function in redhat/mimedefang-init.in both wait for the daemons to exit. I'm uneasy telling any author that they're wrong about their own code but as far as I can tell, that is only conditionally true: $ grep -C2 -n Waiting examples/init-script.in redhat/mimedefang-init.in examples/init-script.in-331- examples/init-script.in-332-if [ "$1" = "wait" ] ; then examples/init-script.in:333: printf "Waiting for daemons to exit." examples/init-script.in-334- WAITPID="" examples/init-script.in-335- test -f $PID && WAITPID=`cat $PID` -- redhat/mimedefang-init.in-227- redhat/mimedefang-init.in-228-if [ "$1" = "wait" ] ; then redhat/mimedefang-init.in:229: printf "Waiting for daemons to exit" redhat/mimedefang-init.in-230- WAITPID="" redhat/mimedefang-init.in-231- test -f /var/run/$prog.pid && WAITPID=`cat /var/run/$prog.pid` ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Tue, 26 Sep 2017 15:48:42 -0400 "Bill Cole" wrote: > I'm uneasy telling any author that they're wrong about their own code > but as far as I can tell, that is only conditionally true: Oh dear... you are correct. :) Seeing as I don't use those scripts myself, I've never encountered the problem. :( Mea culpa. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> I'm uneasy telling any author that they're wrong about their own code > but as far as I can tell, that is only conditionally true: Bill, you and Dianne are both right, depending on the version. See my previous message. I provided a diff between v2.78 and v2.79. In v2.79, 'wait' is always passed to stop_it, so the if statement you showed will always evaluate to true But in v2.78, which is where I had my problem, the main part of the script passes $2 to stop_it. So stop_it may or may not see "wait". Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 26 Sep 2017, at 23:47, Michael Fox wrote: Bill, you and Dianne are both right, depending on the version. See my previous message. I provided a diff between v2.78 and v2.79. That's only in the Debian variant, not Dianne's original code. Everything in the top-level 'debian' directory of the source package for Debian (and Debian derivatives like Ubuntu) is created and maintained by the Debian packager. I tried to figure out where they inserted the bug, but it seems to have been in a version control gap between 2.75 and 2.78 (or maybe between Debian and Ubuntu???) that I don't understand. It may simply be that in earlier versions the distribution init scripts' "restart" command was used directly rather than generating a generic systemd unit file that synthesizes restart with plain "start" and "stop" commands. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> It may > simply be that in earlier versions the distribution init scripts' > "restart" command was used directly rather than generating a generic > systemd unit file that synthesizes restart with plain "start" and "stop" > commands. That makes sense. In my own case, I started with MD on Ubuntu 14.04 (MD v2.73). The documentation I found said to use "/etc/init.d/mimedefang reread" after changes to the filter file. But that had no effect (that I could see). So I started using "/etc/init.d/mimedefang restart". Once I got things working, I installed on Ubuntu 16.04 (v2.78). Ubuntu 16.04 uses systemd. So I simply converted to "systemctl restart mimedefang". And that's when I saw the problem caused by no "wait"ing. BTW, in Ubuntu 16.04: -- systemctl reread mimedefang results in "Unknown operation reread" -- systemctl reload mimedefang does run. But I haven't tested to see if it actually picks up the changes to the filter file. I wonder: Is the Debian package maintainer on this list? We seem to be speculating on questions that the packager could answer definitively. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
For those of you that don't use systemd, you can just ignore this. None of this affects the SysV init scripts in any way. Here's a first run at the diff from the generated unit to a manual unit to fix the issue. The relevant change is making ExecStop use "stop wait" instead of just "stop". The init script (at least on my system, which is Ubuntu 16.04) does not do a "wait" on stop, but only on restart. That arguably seems fine in the SysV world. The other changes I made in the [Service] section weren't required for this particular problem, but seem like things we'd want. This fixes the problem for me. I think a more correct solution for systemd is to make this two units, one for mimedefang and one for mimedefang-multiplexor. I'm going to work on that next, before submitting anything to Debian. --- /run/systemd/generator.late/mimedefang.service 2017-09-16 01:25:06.77200 -0500 +++ /lib/systemd/system/mimedefang.service 2017-09-28 18:41:38.651481894 -0500 @@ -1,10 +1,5 @@ -# Automatically generated by systemd-sysv-generator - [Unit] -Documentation=man:systemd-sysv-generator(8) -SourcePath=/etc/init.d/mimedefang -Before=multi-user.target -Before=multi-user.target +Documentation=man:mimedefang(8) Before=multi-user.target Before=graphical.target Before=shutdown.target @@ -14,12 +9,11 @@ [Service] Type=forking -Restart=no +Restart=on-failure TimeoutSec=5min -IgnoreSIGPIPE=no -KillMode=process +KillMode=control-group GuessMainPID=no RemainAfterExit=yes ExecStart=/etc/init.d/mimedefang start -ExecStop=/etc/init.d/mimedefang stop +ExecStop=/etc/init.d/mimedefang stop wait ExecReload=/etc/init.d/mimedefang reload -- Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
I have attached fully split, native, Type=simple (not forking) unit
files, which I have tested on Ubuntu 16.04.
They honor the settings in /etc/default/mimedefang. This includes the
MX_USER setting (i.e. systemd does not directly control the service user).
They are configured such that if you do start/stop/reload/restart the
mimedefang service (as before), the actions propagate to
mimedefang-multiplexor appropriately.
Some questions (for Dianne?):
1) Is it necessary to rm $SOCKET and $MX_SOCKET *before starting*?
2) Is it necessary to rm $SOCKET and $MX_SOCKET *after stopping*?
3) The init script (at least on Ubuntu) says, '# Locale should be set to
"C" for generating valid date headers'. I see a strftime() in
mimedefang-multiplexor.c. Plus, the Perl code inherits from the
multiplexor, so I can understand this. Is setting LC_ALL=C necessary for
the mimedefang process too, or just mimedefang-multiplexor?
4) The mimedefang-multiplexor process exits with status 1 on SIGTERM. I
am absolutely sure (because I'm doing it by hand for testing) that I am
only sending a SIGTERM, and only to the main mimedefang-multiplexor
process (not the children). It logs 'Received SIGTERM: Stopping slaves
and terminating' and then reaps the slaves as expected; there are no
errors in the log. Is that exit status expected?
I have built the unit files assuming the following answers:
1) Yes.
2) Yes.
3) Not necessary for mimedefang, only mimedefang-multiplexor.
4) "Yes", as this happens (whether expected or not).
--
Richard
[Unit]
Description=MIMEDefang E-mail Filter
Documentation=man:mimedefang(8)
Before=multi-user.target
Before=postfix.service
Before=sendmail.service
After=remote-fs.target
After=systemd-journald-dev-log.socket
BindsTo=mimedefang-multiplexor.service
After=mimedefang-multiplexor.service
PropagatesReloadTo=mimedefang-multiplexor.service
[Service]
Type=simple
Restart=on-failure
TimeoutStopSec=30s
Environment=MX_SOCKET=/var/spool/MIMEDefang/mimedefang-multiplexor.sock
MX_USER=defang SOCKET=/var/spool/MIMEDefang/mimedefang.sock
EnvironmentFile=-/etc/default/mimedefang
ExecStartPre=/bin/rm -f $SOCKET
ExecStart=/bin/sh -c 'exec /usr/bin/mimedefang -D \
`[ -n "$LOOPBACK_RESERVED_CONNECTIONS" ] && echo "-R
$LOOPBACK_RESERVED_CONNECTIONS"` \
-m $MX_SOCKET \
`[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"` \
`[ -n "$MX_USER" ] && echo "-U $MX_USER"` \
`[ -n "$SYSLOG_FACILITY" ] && echo "-S $SYSLOG_FACILITY"` \
`[ "$MX_RELAY_CHECK" = "yes" ] && echo "-r"` \
`[ "$MX_HELO_CHECK" = "yes" ] && echo "-H"` \
`[ "$MX_SENDER_CHECK" = "yes" ] && echo "-s"` \
`[ "$MX_RECIPIENT_CHECK" = "yes" ] && echo "-t"` \
`[ "$KEEP_FAILED_DIRECTORIES" = "yes" ] && echo "-k"` \
`[ "$MD_EXTRA" != "" ] && echo $MD_EXTRA` \
`[ "$MD_SKIP_BAD_RCPTS" = "yes" ] && echo "-N"` \
"`[ -n "$X_SCANNED_BY" ] && \
( [ "$X_SCANNED_BY" = "-" ] && \
echo "-X" || echo "-x$X_SCANNED_BY" )`" \
`[ "$MD_ALLOW_GROUP_ACCESS" = "yes" ] && echo "-G"` \
`[ "$ALLOW_NEW_CONNECTIONS_TO_QUEUE" = "yes" ] && echo "-q"` \
-p $SOCKET'
ExecStopPost=/bin/rm -f $SOCKET
# Make this service eligible for a reload, so we can propagate it to
# mimedefang-multiplexor.service.
ExecReload=/bin/true
[Install]
WantedBy=multi-user.target
[Unit]
Description=MIMEDefang E-mail Filter (Multiplexor)
Documentation=man:mimedefang-multiplexor(8)
After=remote-fs.target
After=systemd-journald-dev-log.socket
PartOf=mimedefang.service
[Service]
Type=simple
Restart=on-failure
TimeoutStopSec=30s
KillMode=mixed
# Locale should be set to "C" for generating valid date headers
Environment=LC_ALL=C MX_BUSY=600 MX_LOG=yes MX_MAXIMUM=10 MX_MINIMUM=2
MX_SOCKET=/var/spool/MIMEDefang/mimedefang-multiplexor.sock MX_USER=defang
EnvironmentFile=-/etc/default/mimedefang
SuccessExitStatus=1
ExecStart=/bin/sh -c 'HOME=${SPOOLDIR:=/var/spool/MIMEDefang} \
exec /usr/bin/mimedefang-multiplexor -D \
`[ "$MX_EMBED_PERL" = "yes" ] && echo "-E"` \
`[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"` \
`[ -n "$FILTER" ] && echo "-f $FILTER"` \
`[ -n "$SYSLOG_FACILITY" ] && echo "-S $SYSLOG_FACILITY"` \
`[ -n "$SUBFILTER" ] && echo "-F $SUBFILTER"` \
`[ -n "$MX_MINIMUM" ] && echo "-m $MX_MINIMUM"` \
`[ -n "$MX_MAXIMUM" ] && echo "-x $MX_MAXIMUM"` \
`[ -n "$MX_MAP_SOCKET" ] && echo "-N $MX_MAP_SOCKET"` \
`[ -n "$MX_LOG_SLAVE_STATUS_INTERVAL" ] && echo "-L
$MX_LOG_SLAVE_STATUS_INTERVAL"` \
`[ -n "$MX_USER" ] && echo "-U $MX_USER"` \
`[ -n "$MX_IDLE" ] && echo "-i $MX_IDLE"` \
`[ -n "$MX_BUSY" ] && echo "-b $MX_BUSY"` \
`[ -n "$MX_REQUESTS" ] && echo "-r $MX_REQUESTS"` \
`[ -n "$MX_SLAVE_DELAY" ] && echo "-w $MX_SLAVE_DELAY"` \
`[ -n "$MX_MIN_SLAVE_DELAY" ] && echo "-W $MX_MIN_SLAVE_DELAY"` \
`[ -n "$MX_MAX_RSS" ] && echo "-R $MX_MAX_RSS"` \
`[ -n "$MX_MAX_AS" ] && echo "-M $MX_MAX_AS"` \
`[ "$MX_LOG" = "yes" ] && echo "-l"` \
`[ "$MX_STATS" = "yes" ] && echo "-t /var/log/mimedefang/st
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Thu, 28 Sep 2017 23:11:18 -0500 Richard Laager wrote: > 1) Is it necessary to rm $SOCKET and $MX_SOCKET *before starting*? It's best to do so because if mimedefang or mimedefang-multiplexor crashes, the socket files will be left lying around and startup will likely fail. > 2) Is it necessary to rm $SOCKET and $MX_SOCKET *after stopping*? It's best to do so in order that stale socket files are not left lying around. > 3) The init script (at least on Ubuntu) says, '# Locale should be set > to "C" for generating valid date headers'. I see a strftime() in > mimedefang-multiplexor.c. Plus, the Perl code inherits from the > multiplexor, so I can understand this. Is setting LC_ALL=C necessary > for the mimedefang process too, or just mimedefang-multiplexor? I don't know; it's probably safer for both. > 4) The mimedefang-multiplexor process exits with status 1 on SIGTERM. This is a bug that was corrected in MIMEDefang 2.82. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On 09/28/2017 11:11 PM, Richard Laager wrote: > I have attached fully split, native, Type=simple (not forking) unit > files, which I have tested on Ubuntu 16.04. I have submitted the slightly updated versions to Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877663 Dianne, you may want to ship these (or similar) upstream, but that's obviously up to you. It doesn't make any difference to me personally, since I use the downstream packaging. -- Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Wed, 4 Oct 2017 18:58:39 -0500 Richard Laager wrote: > I have submitted the slightly updated versions to Debian: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877663 > Dianne, you may want to ship these (or similar) upstream, but that's > obviously up to you. It doesn't make any difference to me personally, > since I use the downstream packaging. I might as well put it in upstream. Thanks for the work! Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
I ran into a problem with my systemd units for MIMEDefang. There is a
race condition between mimedefang-multiplexor creating the socket and
mimedefang trying to access it. If the multiplexor doesn't create the
socket in time, mimedefang fails on startup.
This is happening because Type=simple services are assumed to be running
immediately. This is documented in systemd.service(5), which says, "In
this mode, if the process offers functionality to other processes on the
system, its communication channels should be installed before the daemon
is started up (e.g. sockets set up by systemd, via socket activation),
as systemd will immediately proceed starting follow-up units."
I have attached updated versions to correct this. These change to
Type=forking and drop the -D flag. With the change to Type=forking, I
have also brought back pid files. Since systemd requires the pid file
path to be hardcoded (as the argument to PIDFile), I put it in /run
rather than the $SPOOLDIR (which the administrator can change). It can't
be directly in /run, as the daemon writes the pidfile after dropping
privileges, so we have to create a /run/mimedefang subdirectory owned by
$MX_USER. This requires some indirection, as $MX_USER can also be
customized by the administrator. I only create this directory for
mimedefang-multiplexor.service (and not mimedefang.service), as
mimedefang.service is already ordered after mimedefang-multiplexor.service.
Long term, the ideal answer is to use systemd's socket activation. This
would allow the services to startup in parallel. Another alternative
would be to use Type=notify. Either requires changes to the mimedefang C
code.
--
Richard
[Unit]
Description=MIMEDefang E-mail Filter
Documentation=man:mimedefang(8)
Before=multi-user.target
Before=postfix.service
Before=sendmail.service
After=remote-fs.target
After=systemd-journald-dev-log.socket
BindsTo=mimedefang-multiplexor.service
After=mimedefang-multiplexor.service
PropagatesReloadTo=mimedefang-multiplexor.service
[Service]
Type=forking
PIDFile=/run/mimedefang/mimedefang.pid
Restart=on-failure
TimeoutStopSec=30s
# LC_ALL=C may not be necessary for mimedefang, but it is for
# mimedefang-multiplexor, so upstream prefers it here also to be consistent.
Environment=LC_ALL=C
MX_SOCKET=/var/spool/MIMEDefang/mimedefang-multiplexor.sock MX_USER=defang
SOCKET=/var/spool/MIMEDefang/mimedefang.sock
EnvironmentFile=-/etc/default/mimedefang
ExecStartPre=/bin/rm -f $SOCKET
ExecStart=/bin/sh -c 'exec /usr/bin/mimedefang \
`[ -n "$LOOPBACK_RESERVED_CONNECTIONS" ] && echo "-R
$LOOPBACK_RESERVED_CONNECTIONS"` \
-m $MX_SOCKET \
`[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"` \
`[ -n "$MX_USER" ] && echo "-U $MX_USER"` \
`[ -n "$SYSLOG_FACILITY" ] && echo "-S $SYSLOG_FACILITY"` \
`[ "$MX_RELAY_CHECK" = "yes" ] && echo "-r"` \
`[ "$MX_HELO_CHECK" = "yes" ] && echo "-H"` \
`[ "$MX_SENDER_CHECK" = "yes" ] && echo "-s"` \
`[ "$MX_RECIPIENT_CHECK" = "yes" ] && echo "-t"` \
`[ "$KEEP_FAILED_DIRECTORIES" = "yes" ] && echo "-k"` \
`[ "$MD_EXTRA" != "" ] && echo $MD_EXTRA` \
`[ "$MD_SKIP_BAD_RCPTS" = "yes" ] && echo "-N"` \
"`[ -n "$X_SCANNED_BY" ] && \
( [ "$X_SCANNED_BY" = "-" ] && \
echo "-X" || echo "-x$X_SCANNED_BY" )`" \
`[ "$MD_ALLOW_GROUP_ACCESS" = "yes" ] && echo "-G"` \
`[ "$ALLOW_NEW_CONNECTIONS_TO_QUEUE" = "yes" ] && echo "-q"` \
-p $SOCKET -P /run/mimedefang/mimedefang.pid'
ExecStopPost=/bin/rm -f $SOCKET
# Make this service eligible for a reload, so we can propagate it to
# mimedefang-multiplexor.service.
ExecReload=/bin/true
[Install]
WantedBy=multi-user.target
[Unit]
Description=MIMEDefang E-mail Filter (Multiplexor)
Documentation=man:mimedefang-multiplexor(8)
After=remote-fs.target
After=systemd-journald-dev-log.socket
PartOf=mimedefang.service
[Service]
Type=forking
PIDFile=/run/mimedefang/mimedefang-multiplexor.pid
Restart=on-failure
TimeoutStopSec=30s
KillMode=mixed
# Locale should be set to "C" for generating valid date headers
Environment=LC_ALL=C MX_BUSY=600 MX_LOG=yes MX_MAXIMUM=10 MX_MINIMUM=2
MX_SOCKET=/var/spool/MIMEDefang/mimedefang-multiplexor.sock MX_USER=defang
EnvironmentFile=-/etc/default/mimedefang
ExecStartPre=/bin/rm -f $MX_SOCKET
ExecStartPre=/bin/sh -c '/usr/bin/install -d \
`[ -n "$MX_USER" ] && echo "-o $MX_USER"` \
/run/mimedefang'
ExecStart=/bin/sh -c 'HOME=${SPOOLDIR:=/var/spool/MIMEDefang} \
exec /usr/bin/mimedefang-multiplexor \
`[ "$MX_EMBED_PERL" = "yes" ] && echo "-E"` \
`[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"` \
`[ -n "$FILTER" ] && echo "-f $FILTER"` \
`[ -n "$SYSLOG_FACILITY" ] && echo "-S $SYSLOG_FACILITY"` \
`[ -n "$SUBFILTER" ] && echo "-F $SUBFILTER"` \
`[ -n "$MX_MINIMUM" ] && echo "-m $MX_MINIMUM"` \
`[ -n "$MX_MAXIMUM" ] && echo "-x $MX_MAXIMUM"` \
`[ -n "$MX_MAP_SOCKET" ] && echo "-N $MX_MAP_SOCKET"` \
`[ -n "$MX_LOG_SLAVE_STATUS_INTERVAL" ] && echo "-L
$MX_LOG_SLAVE_STATUS_INTERVAL"`
Re: [Mimedefang] REVISED: postfix/mimedefang socket
On Mon, 23 Jul 2018 03:31:06 -0500 Richard Laager wrote: > I have attached updated systemd service files, with the PID file > directly in /run, which I have tested with 2.84. This change also > requires specifying a lock file in $SPOOLDIR. Thanks! I'll put these in the next release. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
