Re: Postscreen Logfile Analyser

[Tamás Gérczei]

I use pflogsumm.

On 9/11/20 9:08 PM, Michael Orlitzky wrote:
> On 2020-09-11 14:24, Jos Chrispijn wrote:
>> Can someone recommend a reliable Postscreen logfile analyser (FreeBSD 12)?
>> Thanks in advance!
> I still use postfix-logwatch (http://logreporters.sourceforge.net/), but
> there are a few patches to apply since the maintainer went AWOL:
>
> https://gitweb.gentoo.org/repo/gentoo.git/tree/net-mail/postfix-logwatch/files
>
> Postfix is stable enough that I can afford to learn perl whenever the
> log format changes.

Re: lightweight/milter Spamassassin-integtration options for Postfix -- current experience / faves?

[Tamás Gérczei]

I run the dusty one: 'spamass-milt' 0.4, it appears to be working well here.

On 6/9/20 1:18 AM, PGNet Dev wrote:
> i run postfix 3.5.2
>
> i'm 
>
> -- revisiting spamassassin integration with postfix
>
> -- not interested in amavisd integration or rspamd alternative
>
> -- looking for lightweight & known to be (still/currently) reliable & active
>
> -- ideally, tho not absolutely req'd, milter-protocol
>
> -- aware of 3 options, in order of decreasing 'dustiness',
>
>https://savannah.nongnu.org/projects/spamass-milt/
>https://github.com/mpaperno/spampd
>https://gitlab.com/glts/spamassassin-milter
>
> anyone have any current experience with any of these?
>
> recommendations, thoughts, etc?
>
> or other SA-based options?
>

Re: Ready: PROXY protocol v2 support

[Tamás Gérczei]

I've just tested it by spinning up an instance of this version behind an
AWS NLB and connecting to the load balancer from the outside - it worked
well, nevertheless I'd encourage others to test as well. Log snippets
follow:

# with smtpd_upstream_proxy_protocol defaulted to empty
postfix-test-7cbd54cdfc-twv79 postfix Jan 07 10:50:55 mail
postfix/master[1]: daemon started -- version 3.5-20200105-nonprod,
configuration /postfix/config-live
postfix-test-7cbd54cdfc-twv79 postfix Jan 07 10:51:19 mail
postfix/smtpd[76]: connect from
ip-10-36-0-0.eu-central-1.compute.internal[10.36.0.0]
postfix-test-7cbd54cdfc-twv79 postfix Jan 07 10:51:19 mail
postfix/smtpd[76]: improper command pipelining after QUIT from
ip-10-36-0-0.eu-central-1.compute.internal[10.36.0.0]:
!\021\000T\2620\030\265\254\024\000\272\260\273\000\031\003\000\004\206n\304Q\004\000>\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000

# with smtpd_upstream_proxy_protocol = haproxy
postfix-test-7cbd54cdfc-65x22 postfix Jan 07 10:52:24 mail
postfix/master[1]: daemon started -- version 3.5-20200105-nonprod,
configuration /postfix/config-live
postfix-test-7cbd54cdfc-65x22 postfix Jan 07 10:52:28 mail
postfix/smtpd[76]: connect from fejezd.be[178.48.24.181]

Again: thanks Wietse!

T.

On 1/6/20 12:42 AM, Wietse Venema wrote:
> You can test haproxy v2 protocol support in postfix-3.5-20200105-nonprod
> (http://ftp.porcupine.org/mirrors/postfix-release/index.html). I
> have done all the testing that I can do. It would be great is someone
> can test it against some real haproxy client.
>
> Haproxy v2 protocol support is limited to TCP over IPv4 and TCP
> over IPv6. It supports non-proxied connections (typically used for
> heartbeats).
>
> This will be part of the Postfix 3.5 stable release early this year.
>
>   Wietse

Re: Ready: PROXY protocol v2 support

[Tamás Gérczei]

Thank you Wietse, I will test this week and let you know.

On 1/6/20 12:42 AM, Wietse Venema wrote:
> You can test haproxy v2 protocol support in postfix-3.5-20200105-nonprod
> (http://ftp.porcupine.org/mirrors/postfix-release/index.html). I
> have done all the testing that I can do. It would be great is someone
> can test it against some real haproxy client.
>
> Haproxy v2 protocol support is limited to TCP over IPv4 and TCP
> over IPv6. It supports non-proxied connections (typically used for
> heartbeats).
>
> This will be part of the Postfix 3.5 stable release early this year.
>
>   Wietse

Re: PROXY protocol v2 support

[Tamás Gérczei]

Amazing. Thank you!

On 1/2/20 1:41 AM, Wietse Venema wrote:
 Maybe you can try to implement v2 support ? Parsing v2 when v1 is already
 supported is quite easy, especially at the same level of support (i.e. no
 TLV field support for TLS or whatever). You can have a look at
 conn_recv_proxy() in haproxy:src/connection.c which supports the two
 versions. Feel free to steal any code I wrote there, if that helps :-)
>>> I'm looking into this, and a Postfix-quality implementation including
>>> tests(!) is going to take more than a few hours. Primarily, because
>>> v2 uses a binary protocol, which complicates everything. I'll put a
>>> few hours in today.
>> I found that I first have to refactor the existing Postfix
>> implementation, because of a change in the haproxy message format.
>> The "diff -u" output is about 750 lines -- and that is refactoring
>> before adding v2 support.
>>
>> The existing Postfix code completely separates the haproxy message
>> reader from the haproxy message parser. The v1 haproxy message has
>> a unique message terminator, and the haproxy message reader can
>> pull the entire message from the input queue before handing it off
>> to the haproxy message parser.
> The refactor is done (postfix-3.5-20200101-nonprod), and I verified
> that version 1 protocol still support works as before. 
>
> It took a littie over 200 lines to add version 2 protocol support.
> That code will need some testing before I can share it.
>
>   Wietse

Re: PROXY protocol v2 support

[Tamás Gérczei]

Many thanks in advance for all your efforts in this regard, Wietse!

On 12/31/19 5:08 PM, Wietse Venema wrote:
> Willy Tarreau:
>> On Tue, Dec 31, 2019 at 08:21:05AM +0100, Tam?s G?rczei wrote:
>>> Thanks Wietse, this is what I thought and found out during my
>>> experiments,That said, now knowing that only v1 is supported, may I ask
>>> whether you have considered implementing v2 support? I'm about to
>>> migrate to a setup where I'm behind a load balancer that only speaks v2.
>> Maybe you can try to implement v2 support ? Parsing v2 when v1 is already
>> supported is quite easy, especially at the same level of support (i.e. no
>> TLV field support for TLS or whatever). You can have a look at
>> conn_recv_proxy() in haproxy:src/connection.c which supports the two
>> versions. Feel free to steal any code I wrote there, if that helps :-)
> I'm looking into this, and a Postfix-quality implementation including
> tests(!) is going to take more than a few hours. Primarily, because
> v2 uses a binary protocol, which complicates everything. I'll put a
> few hours in today.
>
>   Wietse

Re: PROXY protocol v2 support

[Tamás Gérczei]

Thanks Willy, I appreciate the clue and your helpful intention -
unfortunately this isn't something I can personally do owing to lack of
knowledge. I don't know whether the v1 implementation had been a
community patch or something Wietse or Viktor have done.

On 12/31/19 8:35 AM, Willy Tarreau wrote:
> On Tue, Dec 31, 2019 at 08:21:05AM +0100, Tamás Gérczei wrote:
>> Thanks Wietse, this is what I thought and found out during my
>> experiments,That said, now knowing that only v1 is supported, may I ask
>> whether you have considered implementing v2 support? I'm about to
>> migrate to a setup where I'm behind a load balancer that only speaks v2.
> Maybe you can try to implement v2 support ? Parsing v2 when v1 is already
> supported is quite easy, especially at the same level of support (i.e. no
> TLV field support for TLS or whatever). You can have a look at
> conn_recv_proxy() in haproxy:src/connection.c which supports the two
> versions. Feel free to steal any code I wrote there, if that helps :-)
>
> Regards,
> Willy

Re: PROXY protocol v2 support

[Tamás Gérczei]

Thanks Wietse, this is what I thought and found out during my
experiments,That said, now knowing that only v1 is supported, may I ask
whether you have considered implementing v2 support? I'm about to
migrate to a setup where I'm behind a load balancer that only speaks v2.

Yours,
Tamás

On 12/30/19 9:38 PM, Wietse Venema wrote:
> Tam?s G?rczei:
>> Hello List,
>>
>> I'd like to ask if PROXY protocol v2 is supported by Postfix?
> It's not mentioned in documentation, therefore it is not supported.
> Ditto for memcached v2 protocol.
>
>   Wietse

PROXY protocol v2 support

[Tamás Gérczei]

Hello List,

I'd like to ask if PROXY protocol v2 is supported by Postfix?

Thanks,
Tamás

Re: Docker Postfix logging

[Tamás Gérczei]

Hi,

Nice one, albeit you can use this
 since 3.4 - I have
it set to the value '/dev/stdout'.

Yours,
Tamás

On 3/20/19 11:10 AM, Matthias Schneider wrote:
> Hi,
>
> since i don't want to mount /dev/log into a postfix container, i
> created a small tool in golang to create a syslog unix socket and
> print all logs to stdout,
> the command `postfix start-fg` is wrapped.
>
> Just want to share this with the community:
>
> https://github.com/mschneider82/postfixstdoutlogger/
>
>
> Best Regards
>
> Matthias Schneider
>

Re: PATCH: non-Postfix processes in init mode

[Tamás Gérczei]

Sure, I must suck at expressing myself as well. Not only had it been
left intact and moved into the conditional construct, it had still been
there without that in addition, thereby defeating the very purpose of
patching. Like I said, contrary to what I might seem, I'm no moron and
I've realized the mistake when You told me the behaviour didn't seem
like being patched in the first place. I've since then corrected my
patch, applied, compiled and packaged it, imaged it and am running it -
and it works. I'm grateful!

On 2019. 02. 18. 20:41, Wietse Venema wrote:
> Tam?s G?rczei:
>> Indeed, I should have had some rest before applying your patch... I did
>> create the condition, branching on init mode towards continue as
>> intended, yet also left the original call there right thereafter :D It
> You MUST KEEP the original panic call, like this:
>
> if ((proc = (MASTER_PROC *) binhash_find(master_child_table,
> (void *) , sizeof(pid))) == 0) {
> if (init_mode)
> continue;   /* non-Postfix process */
> msg_panic("master_reap: unknown pid: %d", pid);
> }

Re: PATCH: non-Postfix processes in init mode

[Tamás Gérczei]

Indeed, I should have had some rest before applying your patch... I did
create the condition, branching on init mode towards continue as
intended, yet also left the original call there right thereafter :D It
applied and compiled well and I was sure I used the new packages and
image so I was reluctant to follow along. Now I have retraced my steps
and looked at the code after applying my mangled patch and everything
became very obvious. Thank You for pointing it out!

On 2019. 02. 18. 17:36, Wietse Venema wrote:
> Tam?s G?rczei:
>>   1 root  0:00 /usr/libexec/postfix/master -i
>> ...
>>  78 postfix   0:00 smtpd -n smtp -t inet -u -o stress=
>>
>> Feb 18 09:36:57 mail-postfix-0 postfix/master[1]: panic: master_reap:
>> unknown pid: 78*
>>> +   if (init_mode)
>>> +   continue;   /* non-Postfix process */
>>> msg_panic("master_reap: unknown pid: %d", pid);
> You are not running a correctly patched master daemon. Your code
> probably looks like this:
>
> if (init_mode)
> msg_panic("master_reap: unknown pid: %d", pid);
>
> That is consistent with your logging and 'ps' output.
>
>   Wietse

Re: PATCH: non-Postfix processes in init mode

[Tamás Gérczei]

So I've now configured init mode in foreground with logging properly on
stdout and your patch applied to the last 3.5 snapshot and I got the
same panic, although this time not against my piped spamd process,
rather smtpd, it would appear:

/ # ps
PID   USER TIME  COMMAND
    1 root  0:00 /usr/libexec/postfix/master -i
   75 postfix   0:00 pickup -l -t unix -u
   76 postfix   0:00 qmgr -l -t unix -u
   77 postfix   0:00 postlogd -l -n postlog -t unix-dgram -u
 *  78 postfix   0:00 smtpd -n smtp -t inet -u -o stress=*
   79 postfix   0:00 tlsmgr -l -t unix -u
   80 postfix   0:00 anvil -l -t unix -u
   81 postfix   0:00 trivial-rewrite -n rewrite -t unix -u
   82 postfix   0:00 spawn -z -n policyd-spf -t unix user=policyd-spf
argv=/usr/bin/policyd-spf /policyd-spf/config/policyd-spf.conf
   83 policyd-  0:01 {policyd-spf} /usr/bin/python3.6
/usr/bin/policyd-spf /policyd-spf/config/policyd-spf.conf
   84 postfix   0:00 cleanup -z -t unix -u
   85 postfix   0:00 pipe -n spamassassin -t unix user=spamc
argv=/usr/bin/spamc -d mail-spamassassin-0.mail-spamassassin-service -e
/usr/sbin/sendmail -oi -f ${sender} ${recipient}
   89 postfix   0:00 lmtp -t unix -u
   90 root  0:00 /bin/sh
   97 root  0:00 ps
/ # command terminated with exit code 137

tgerczei@altar ~ $ kc logs po/mail-postfix-0 -f
Feb 18 09:33:39 mail-postfix-0 postfix/postfix-script[63]: warning:
group or other writable: /postfix/config-live/.
Feb 18 09:33:39 mail-postfix-0 postfix/postfix-script[74]: starting the
Postfix mail system
Feb 18 09:33:39 mail-postfix-0 postfix/master[1]: daemon started --
version 3.5-20190212, configuration /postfix/config-live
Feb 18 09:35:14 mail-postfix-0 postfix/smtpd[78]: connect from
mail-qt1-f182.google.com[209.85.160.182]
Feb 18 09:35:15 mail-postfix-0 postfix/smtpd[78]: Anonymous TLS
connection established from mail-qt1-f182.google.com[209.85.160.182]:
TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Feb 18 09:35:15 mail-postfix-0 postfix/smtpd[78]: NOQUEUE: filter: RCPT
from mail-qt1-f182.google.com[209.85.160.182]:
: Sender address triggers FILTER
spamassassin:dummy; from= to=
proto=ESMTP helo=
Feb 18 09:35:17 mail-postfix-0 postfix/smtpd[78]: 63C912D0E6C:
client=mail-qt1-f182.google.com[209.85.160.182]
Feb 18 09:35:17 mail-postfix-0 postfix/cleanup[84]: 63C912D0E6C:
message-id=
Feb 18 09:35:17 mail-postfix-0 postfix/qmgr[76]: 63C912D0E6C:
from=, size=2872, nrcpt=1 (queue active)
Feb 18 09:35:17 mail-postfix-0 postfix/smtpd[78]: disconnect from
mail-qt1-f182.google.com[209.85.160.182] ehlo=2 starttls=1 mail=1 rcpt=1
bdat=1 quit=1 commands=7
Feb 18 09:35:27 mail-postfix-0 postfix/pickup[75]: DBB5F2D0E71: uid=103
from=
Feb 18 09:35:27 mail-postfix-0 postfix/pipe[85]: 63C912D0E6C:
to=, relay=spamassassin, delay=13,
delays=2.2/0.02/0/10, dsn=2.0.0, status=sent (delivered via spamassassin
service)
Feb 18 09:35:27 mail-postfix-0 postfix/qmgr[76]: 63C912D0E6C: removed
Feb 18 09:35:27 mail-postfix-0 postfix/cleanup[84]: DBB5F2D0E71:
message-id=
Feb 18 09:35:29 mail-postfix-0 postfix/qmgr[76]: DBB5F2D0E71:
from=, size=3497, nrcpt=1 (queue active)
Feb 18 09:35:29 mail-postfix-0 postfix/lmtp[89]: DBB5F2D0E71:
to=,
relay=mail-dovecot-0.mail-dovecot-service.default.svc.cluster.local[10.244.1.53]:54321,
delay=2.9, delays=2.7/0.03/0.05/0.12, dsn=2.0.0, status=sent (250 2.0.0
 y4PZF2F8alx7GwAAHo2YKg Saved)
Feb 18 09:35:29 mail-postfix-0 postfix/qmgr[76]: DBB5F2D0E71: removed
*Feb 18 09:36:57 mail-postfix-0 postfix/master[1]: panic: master_reap:
unknown pid: 78*

I don't get this since smtpd had surely been spawned by master. Should I
have used 3.4? Does 3.5 have anything new in there in this regard?
(Anything below 3.4 wouldn't support logging to stdout, hence my choice.)

On 2019. 02. 17. 15:42, Wietse Venema wrote:
> Wietse Venema:
>> Tam?s G?rczei:
>>> Hello List,
>>>
>>> I'd like to ask whether You're aware of any change which might cause
>>> breakage in my setup involving spamc with a completely unchanged
>>> configuration in between - I'm getting the following error:
>>>
>>> *"panic: master_reap: unknown pid"*
>> Is the Postfix master daemon running as PID 1? Historically this
>> PID was reserved for the init daemon which becomes the parent of
>> orphaned processes, i.e. processes that terminate without a parent
>> waiting for them. Postfix does not create such processes, but if
>> you run other programs inside the Postfix container then they
>> might do that.
>>
>> Options:
>>
>> - Run Postfix as PID != 1.
>>
>> - Don't co-locate Postfix with other software. That means pipe into
>> socket instead of into a program in the same container.
>>
>> - Make the master sloppier, and accept events from processes that
>> it did not create.
> As per the patch below.
>
> --- src/master/master_spawn.c-2014-12-06 20:35:33.0 -0500
> +++ src/master/master_spawn.c 2019-02-17 09:38:52.0 -0500
> @@ -301,8 +301,11 @@
>   if (msg_verbose)
>   msg_info("master_reap_child: pid 

Re: PATCH: non-Postfix processes in init mode

[Tamás Gérczei]

Hm.. Thanks to You both, I'll go take a look at this.

On 2019. 02. 17. 16:14, Wietse Venema wrote:
> Tam?s G?rczei:
>> Thanks Wietse, I'll definitely try this patch -? but this code didn't
>> change in quite a bit of time.
> If in doubt, look RTFM the Postfix 3.3.1 announcement.
>
>  *  Postfix did not support running as a PID=1 process, which
> complicated Postfix management in containers. The "postfix
> start-fg" command will now run the Postfix master daemon as a
> PID=1 process if possible. Thanks to inputs from Andreas Schulze,
> Eray Aslan, and Viktor Dukhovni.
>
>   Wietse

Re: PATCH: non-Postfix processes in init mode

[Tamás Gérczei]

Thanks Wietse, I'll definitely try this patch -  but this code didn't
change in quite a bit of time. Can this behaviour I'm seeing somehow
relate to a change introduced between 3.3.0 and 3.3.2 ? I have zero
problems with the exact same setup and configuration on 3.3.0. Anyway
I'm invoking postfix-script in order to start master:

/ # ps auxwww -o pid,ppid,user,time,comm
PID   PPID  USER TIME  COMMAND
    1 0 root  0:00 postfix-script
    9 1 root  0:01 rsyslogd
   92 1 root  0:00 master
   94    92 postfix   0:00 qmgr
   96    92 postfix   0:00 tlsmgr
  235    92 postfix   0:00 pickup
  275 0 root  0:00 sh
  283    92 postfix   0:00 smtpd
  284    92 postfix   0:00 anvil
  286   275 root  0:00 ps

T.

On 2019. 02. 17. 15:42, Wietse Venema wrote:
> Wietse Venema:
>> Tam?s G?rczei:
>>> Hello List,
>>>
>>> I'd like to ask whether You're aware of any change which might cause
>>> breakage in my setup involving spamc with a completely unchanged
>>> configuration in between - I'm getting the following error:
>>>
>>> *"panic: master_reap: unknown pid"*
>> Is the Postfix master daemon running as PID 1? Historically this
>> PID was reserved for the init daemon which becomes the parent of
>> orphaned processes, i.e. processes that terminate without a parent
>> waiting for them. Postfix does not create such processes, but if
>> you run other programs inside the Postfix container then they
>> might do that.
>>
>> Options:
>>
>> - Run Postfix as PID != 1.
>>
>> - Don't co-locate Postfix with other software. That means pipe into
>> socket instead of into a program in the same container.
>>
>> - Make the master sloppier, and accept events from processes that
>> it did not create.
> As per the patch below.
>
> --- src/master/master_spawn.c-2014-12-06 20:35:33.0 -0500
> +++ src/master/master_spawn.c 2019-02-17 09:38:52.0 -0500
> @@ -301,8 +301,11 @@
>   if (msg_verbose)
>   msg_info("master_reap_child: pid %d", pid);
>   if ((proc = (MASTER_PROC *) binhash_find(master_child_table,
> -   (void *) , sizeof(pid))) == 0)
> + (void *) , sizeof(pid))) == 0) {
> + if (init_mode)
> + continue;   /* non-Postfix process */
>   msg_panic("master_reap: unknown pid: %d", pid);
> + }
>   serv = proc->serv;
>  
>  #define MASTER_KILL_SIGNAL   SIGTERM

panic after upgrading to 3.3.2 from 3.3.0

[Tamás Gérczei]

Hello List,

I'd like to ask whether You're aware of any change which might cause
breakage in my setup involving spamc with a completely unchanged
configuration in between - I'm getting the following error:

*"panic: master_reap: unknown pid"*

I'm running Docker containers from Alpine images orchestrated by
Kubernetes on ARMv6 with all services running in separate pods, but I
have also tested on Debian to figure out whether it's an Alpine-specific
problem - alas, it also occurs on Debian. The fatal step is invoking
spamc via pipe, although that succeeds since spamd does receive the
call, that is proven. (Everything is alright until that call is made,
and then it reproducibly perishes every single time.) Spamd is 3.4.2
even in working cases, but failures also occur with both 3.4.1 and 3.4.2
client versions. I've spent 2 days trying to debug this on my own, but
fail to comprehend the root cause of the problem, not even when diff'ing
'postconf -x' outputs. I thought it's either a different spamc version
exiting in an unexpected manner or the pipe service handling exits from
external processes differently, but changing spamc versions doesn't
change the symptom, plus the code for pipe in Postfix doesn't seem to
have changed in several years now. As hinted, reverting to a container
with 3.3.0 immediately fixes the problem, but I'd like to be able to
upgrade.

Could You please assist me figure out what's going wrong here?
Documentation says panics are meant to be resolved by the developers, so
this is a justified call for help - I'd be happy to supply any
information necessary.

Thanks,
Tamás