[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Gerben Wierda via Postfix-users 
> On 31 May 2024, at 16:13, Wietse Venema via Postfix-users 
>  wrote:
> 
> Gerben Wierda via Postfix-users:
>>> On 31 May 2024, at 14:53, Wietse Venema  wrote:
>>> 
>>> Gerben Wierda via Postfix-users:
 
> On 31 May 2024, at 13:20, pat...@patpro.net wrote:
> 
> Hello,
> 
> Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
> errors/abuse/half-baked connections?
 
 Not blacklisting as I understand it, but as HAproxy makes a connection to 
 test if the service is up and then breaks the connection I always see this 
 on both systems:
 
 On the postfix 3.9 instance
 May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: connect from 
 router.rna.nl[192.168.2.2]
 May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: improper command 
 pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
 May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: disconnect from 
 router.rna.nl[192.168.2.2] commands=0/0
>>> 
>>> Yep, turn off smtpd_forbid_unauth_pipelining and try again..
>>> 
>>> Wietse
>> 
>> Actually, changing the health check on submission to 
>> 
>> "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\n"
>> 
>> (without the added "QUIT\r\n") did the trick as well. It might
>> have been that in a previous situation HAproxy would 'never' finish
>> the health check, I don't recall why I added "QUIT\r\n". Maybe it
>> is needed for postscreen or dovecot and I just copied it to all
>> and now it stopped working for submission.
> 
> Does not work?
> 
> - Logging would be extremely helpful.
> 
> - A machine-readable before-after configration diff would also be extremely 
> helpful. 

Maybe I was unclear.

My problem was solved by removing the extra "QUIT" line from the data HAproxy 
sends to submission as a health check. Simply sending that single "PROXY TCP4 
192.168.2.2 192.168.2.2 65535 587\r\n" line returns a "220" result, which 
enable HAproxy to detect that the service is available.

G___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Wietse Venema via Postfix-users 
Gerben Wierda via Postfix-users:
> > On 31 May 2024, at 14:53, Wietse Venema  wrote:
> > 
> > Gerben Wierda via Postfix-users:
> >> 
> >>> On 31 May 2024, at 13:20, pat...@patpro.net wrote:
> >>> 
> >>> Hello,
> >>> 
> >>> Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
> >>> errors/abuse/half-baked connections?
> >> 
> >> Not blacklisting as I understand it, but as HAproxy makes a connection to 
> >> test if the service is up and then breaks the connection I always see this 
> >> on both systems:
> >> 
> >> On the postfix 3.9 instance
> >> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: connect from 
> >> router.rna.nl[192.168.2.2]
> >> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: improper command 
> >> pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
> >> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: disconnect from 
> >> router.rna.nl[192.168.2.2] commands=0/0
> > 
> > Yep, turn off smtpd_forbid_unauth_pipelining and try again..
> > 
> > Wietse
> 
> Actually, changing the health check on submission to 
> 
> "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\n"
> 
> (without the added "QUIT\r\n") did the trick as well. It might
> have been that in a previous situation HAproxy would 'never' finish
> the health check, I don't recall why I added "QUIT\r\n". Maybe it
> is needed for postscreen or dovecot and I just copied it to all
> and now it stopped working for submission.

Does not work?

- Logging would be extremely helpful.

- A machine-readable before-after configration diff would also be extremely 
helpful. 

Wietse


Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Gerben Wierda via Postfix-users 
> On 31 May 2024, at 14:53, Wietse Venema  wrote:
> 
> Gerben Wierda via Postfix-users:
>> 
>>> On 31 May 2024, at 13:20, pat...@patpro.net wrote:
>>> 
>>> Hello,
>>> 
>>> Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
>>> errors/abuse/half-baked connections?
>> 
>> Not blacklisting as I understand it, but as HAproxy makes a connection to 
>> test if the service is up and then breaks the connection I always see this 
>> on both systems:
>> 
>> On the postfix 3.9 instance
>> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: connect from 
>> router.rna.nl[192.168.2.2]
>> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: improper command 
>> pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
>> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: disconnect from 
>> router.rna.nl[192.168.2.2] commands=0/0
> 
> Yep, turn off smtpd_forbid_unauth_pipelining and try again..
> 
>   Wietse

Actually, changing the health check on submission to 

"PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\n"

(without the added "QUIT\r\n") did the trick as well. It might have been that 
in a previous situation HAproxy would 'never' finish the health check, I don't 
recall why I added "QUIT\r\n". Maybe it is needed for postscreen or dovecot and 
I just copied it to all and now it stopped working for submission.

G___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Wietse Venema via Postfix-users 
Viktor Dukhovni via Postfix-users:
> On Fri, May 31, 2024 at 02:01:50PM +0200, Gerben Wierda via Postfix-users 
> wrote:
> 
> > It sends: "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\nQUIT\r\n"
> > It expects a response that matches regex ^220
> 
> Don't send "QUIT\r\n", just send the PROXY handshake and wait for 220,
> and then drop the connection, or if not difficult to specify, send QUIT
> *after* the 220.

Viktor is correct. Your probe talks SMTP too soon, and the real
fix is to not send QUIT before Postfix responds.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Wietse Venema via Postfix-users 
Gerben Wierda via Postfix-users:
> 
> > On 31 May 2024, at 13:20, pat...@patpro.net wrote:
> > 
> > Hello,
> > 
> > Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
> > errors/abuse/half-baked connections?
> 
> Not blacklisting as I understand it, but as HAproxy makes a connection to 
> test if the service is up and then breaks the connection I always see this on 
> both systems:
> 
> On the postfix 3.9 instance
> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: connect from 
> router.rna.nl[192.168.2.2]
> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: improper command 
> pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
> May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: disconnect from 
> router.rna.nl[192.168.2.2] commands=0/0

Yep, turn off smtpd_forbid_unauth_pipelining and try again..

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Viktor Dukhovni via Postfix-users 
On Fri, May 31, 2024 at 02:01:50PM +0200, Gerben Wierda via Postfix-users wrote:

> It sends: "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\nQUIT\r\n"
> It expects a response that matches regex ^220

Don't send "QUIT\r\n", just send the PROXY handshake and wait for 220,
and then drop the connection, or if not difficult to specify, send QUIT
*after* the 220.

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Viktor Dukhovni via Postfix-users 
On Fri, May 31, 2024 at 01:06:20PM +0200, Gerben Wierda via Postfix-users wrote:

> Hmm, I just noticed (all outgoing smtp was going to a backup server
> that works) that one of my postfix instances cannot send mail (smtp
> doesn't work, postscreen and smtpd work fine).

What *exactly* do you mean by "smtp" doesn't work?  What concrete
evidence can you post to substantiate and detail this?

> # submission (587)
> submission inet n   -   n   -   -   smtpd
>   -o smtpd_tls_security_level=encrypt
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_tls_auth_only=yes
>   -o syslog_name=submission
> 990 inet n   -   n   -   -   smtpd
>   -o smtpd_tls_security_level=encrypt
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_tls_auth_only=yes
>   -o syslog_name=submission_haproxy
>   -o smtpd_upstream_proxy_protocol=haproxy
> 
> The one that haproxy sees as down has been recently updated to postfix 3.9

Often, Postfix updates are part of a broader update of other system
packages, perhaps the issue is with one of those.

> So, HAproxy sees smtpd as up on postfix 3.9 but it sees smtp as down.
> In reality, both are up.

How could "haproxy" "see" "smtp" as down, the smtp(8) delivery agent
is not a network listener and haproxy does not connect to it.  If you
mean incoming SMTP on port 25 (the "smtp/inet" service in master.cf),
that's still "smtpd", so best to not call it "smtp".

Also why not post that master.cf entry?  And some logging for
"postfix/smtpd" (assuming default syslog_name).

> What should I do? Revert to postfix 3.8? I rather not, I rather would
> upgrade the other to 3.9 (but if I do that, I probably lose all smtp
> behind HAproxy for now)

Reverting Postfix is unlikely to help, Postfix is very stable software,
and a configuration that isn't working with 3.9 likely won't work also
with 3.8.

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Gerben Wierda via Postfix-users 

> On 31 May 2024, at 13:20, pat...@patpro.net wrote:
> 
> Hello,
> 
> Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
> errors/abuse/half-baked connections?

Not blacklisting as I understand it, but as HAproxy makes a connection to test 
if the service is up and then breaks the connection I always see this on both 
systems:

On the postfix 3.9 instance
May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: connect from 
router.rna.nl[192.168.2.2]
May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: improper command 
pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
May 26 05:31:33 hermione submission_haproxy/smtpd[21485]: disconnect from 
router.rna.nl[192.168.2.2] commands=0/0

On the postfix 3.8.6 instance:
May 25 22:02:16 snape submission_haproxy/smtpd[28756]: connect from 
router.rna.nl[192.168.2.2]
May 25 22:02:16 snape submission_haproxy/smtpd[28756]: improper command 
pipelining after CONNECT from router.rna.nl[192.168.2.2]: QUIT\r\n
May 25 22:02:16 snape submission_haproxy/smtpd[28756]: disconnect from 
router.rna.nl[192.168.2.2] quit=1 commands=1

And the test that HAproxy does if port 25 is up are identical too:

On the postfix 3.9 instance
May 26 05:39:29 hermione smtp_haproxy/postscreen[21786]: CONNECT from 
[192.168.2.2]:65535 to [192.168.2.2]:25
May 26 05:39:29 hermione smtp_haproxy/postscreen[21786]: ALLOWLISTED 
[192.168.2.2]:65535
May 26 05:39:29 hermione smtp/smtpd[21788]: connect from 
router.rna.nl[192.168.2.2]
May 26 05:39:29 hermione smtp/smtpd[21788]: disconnect from 
router.rna.nl[192.168.2.2] quit=1 commands=1

On the postfix 3.8.6 instance:
May 25 22:10:57 snape smtp_haproxy/postscreen[28766]: CONNECT from 
[192.168.2.2]:65535 to [192.168.2.2]:25
May 25 22:10:57 snape smtp_haproxy/postscreen[28766]: ALLOWLISTED 
[192.168.2.2]:65535
May 25 22:10:57 snape smtp/smtpd[28768]: connect from router.rna.nl[192.168.2.2]
May 25 22:10:57 snape smtp/smtpd[28768]: disconnect from 
router.rna.nl[192.168.2.2] quit=1 commands=1

Actually, it looks like the response from postfix 3.9 has changed with respect 
to postfix 3.8.6 so in the HAproxy log I see

2024-05-23T01:28:29 Alert   haproxy Server mail.rna.nl.990/hermione-990 is 
DOWN. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 
0 requeued, 0 remaining in queue.
2024-05-23T01:28:29 Notice  haproxy Health check for server 
mail.rna.nl.990/hermione-990 failed, reason: Layer7 invalid response, info: 
"TCPCHK did not match content (regex) at step 2", check duration: 45ms, status: 
0/2 DOWN.   
2024-05-23T01:27:23 Notice  haproxy Health check for backup server 
mail.rna.nl.991/snape-991 succeeded, reason: Layer7 check passed, code: 0, 
info: "(tcp-check)", check duration: 14ms, status: 3/3 UP.

HAproxy is configured:
It sends: "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587\r\nQUIT\r\n"
It expects a response that matches regex ^220

Now, weirdly enough, when I send "PROXY TCP4 192.168.2.2 192.168.2.2 65535 587" 
via nc both react the same:

On the postfix 3.8.6 instance:

root@hermione ~ # nc -v 192.168.2.125 990
Connection to 192.168.2.125 port 990 [tcp/ftps] succeeded!
PROXY TCP4 192.168.2.2 192.168.2.2 65535 587
220 mail.rna.nl
^C

On the postfix 3.9 instance
root@hermione ~ # nc -v 192.168.2.86 990 
Connection to 192.168.2.86 port 990 [tcp/ftps] succeeded!
PROXY TCP4 192.168.2.2 192.168.2.2 65535 587
220 mail.rna.nl
^C

Could it be that the immediate QUIT command in that health check is creating 
this problem on 3.9 because it is sent before 220 is received?

G

> 
> May 31, 2024 1:06 PM, "Gerben Wierda via Postfix-users" 
>  >
>  wrote:
> Hmm, I just noticed (all outgoing smtp was going to a backup server that 
> works) that one of my postfix instances cannot send mail (smtp doesn't work, 
> postscreen and smtpd work fine).
> # submission (587)
> submission inet n - n - - smtpd
> -o smtpd_tls_security_level=encrypt
> -o smtpd_sasl_auth_enable=yes
> -o smtpd_tls_auth_only=yes
> -o syslog_name=submission
> 990 inet n - n - - smtpd
> -o smtpd_tls_security_level=encrypt
> -o smtpd_sasl_auth_enable=yes
> -o smtpd_tls_auth_only=yes
> -o syslog_name=submission_haproxy
> -o smtpd_upstream_proxy_protocol=haproxy
> The one that haproxy sees as down has been recently updated to postfix 3.9
> So, HAproxy sees smtpd as up on postfix 3.9 but it sees smtp as down. In 
> reality, both are up.
> It probably started to behave this when I installed postfix 3.9 on one side, 
> though I cannot exclude that I updated HAproxy too, so I am not 100% certain.
> What should I do? Revert to postfix 3.8? I rather not, I rather would upgrade 
> the other to 3.9 (but if I do that, I probably lose all smtp behind HAproxy 
> for now)
> Gerben Wierda (LinkedIn , Mastodon 
> )
> R IT Strategy

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread Wietse Venema via Postfix-users 
Gerben Wierda via Postfix-users:
> Hmm, I just noticed (all outgoing smtp was going to a backup server that 
> works) that one of my postfix instances cannot send mail (smtp doesn't work, 
> postscreen and smtpd work fine).
> 
> # submission (587)
> submission inet n   -   n   -   -   smtpd
>   -o smtpd_tls_security_level=encrypt
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_tls_auth_only=yes
>   -o syslog_name=submission
> 990 inet n   -   n   -   -   smtpd
>   -o smtpd_tls_security_level=encrypt
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_tls_auth_only=yes
>   -o syslog_name=submission_haproxy
>   -o smtpd_upstream_proxy_protocol=haproxy
> 
> The one that haproxy sees as down has been recently updated to postfix 3.9
> 
> So, HAproxy sees smtpd as up on postfix 3.9 but it sees smtp as down. In 
> reality, both are up.

Postfix logging for failed haproxy connections would be extremely
informative.

There was a change in how Postfix detects clients that talk too early
(smtpd_forbid_unauth_pipelining = yes). This was disabled prior to
Postfix 3.9. Perhaps haproxy falls into this trap.

Wietse

> It probably started to behave this when I installed postfix 3.9 on one side, 
> though I cannot exclude that I updated HAproxy too, so I am not 100% certain.
> 
> What should I do? Revert to postfix 3.8? I rather not, I rather would upgrade 
> the other to 3.9 (but if I do that, I probably lose all smtp behind HAproxy 
> for now)
> 
> Gerben Wierda (LinkedIn , Mastodon 
> )
> R IT Strategy  (main site)
> Book: Chess and the Art of Enterprise?Architecture 
> 
> Book: Mastering ArchiMate 
> YouTube Channel 

> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: HAproxy 4.3 thinks one of my postfixes (3.9) is down on SMTP, but it sees another (3.8.6) as up on SMTP (SMTPD/postscreen are OK on both sides)

2024-05-31 Thread patpro--- via Postfix-users 
Hello,

Any sign of postfix 3.9 blacklisting HAproxy because of SMTP 
errors/abuse/half-baked connections?
May 31, 2024 1:06 PM, "Gerben Wierda via Postfix-users" 
 wrote:
Hmm, I just noticed (all outgoing smtp was going to a backup server that works) 
that one of my postfix instances cannot send mail (smtp doesn't work, 
postscreen and smtpd work fine).
# submission (587)

submission inet n - n - - smtpd

-o smtpd_tls_security_level=encrypt

-o smtpd_sasl_auth_enable=yes

-o smtpd_tls_auth_only=yes

-o syslog_name=submission

990 inet n - n - - smtpd

-o smtpd_tls_security_level=encrypt

-o smtpd_sasl_auth_enable=yes

-o smtpd_tls_auth_only=yes

-o syslog_name=submission_haproxy

-o smtpd_upstream_proxy_protocol=haproxy

The one that haproxy sees as down has been recently updated to postfix 3.9

So, HAproxy sees smtpd as up on postfix 3.9 but it sees smtp as down. In 
reality, both are up.

It probably started to behave this when I installed postfix 3.9 on one side, 
though I cannot exclude that I updated HAproxy too, so I am not 100% certain.

What should I do? Revert to postfix 3.8? I rather not, I rather would upgrade 
the other to 3.9 (but if I do that, I probably lose all smtp behind HAproxy for 
now) 

Gerben Wierda (LinkedIn (https://www.linkedin.com/in/gerbenwierda), Mastodon 
(https://newsie.social/@gctwnl))
R IT Strategy (https://ea.rna.nl/) (main site)
Book: Chess and the Art of Enterprise Architecture (https://ea.rna.nl/the-book/)
Book: Mastering ArchiMate (https://ea.rna.nl/the-book-edition-iii/)
YouTube Channel (http://www.youtube.com/@GerbenWierda)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org