message may be sent more than once

[K bharathan]

Mar 18 17:25:19 relay2 postfix/smtp[21383]: 5470B21265:
to=<41b.4.74998426-6452...@whereverstormy.com>, relay=
mail.WhereverStormy.com[173.46.193.75]:25, delay=418568,
delays=418439/0.46/4.7/123, dsn=4.4.2, status=deferred (lost connection with
mail.WhereverStormy.com[173.46.193.75] while sending end of data -- message
may be sent more than once)

the above is from my relay;
"while sending end of data -- message may be sent more than once)" what does
this mean?

"message may be sent more than once"

[Ralf Hildebrandt]

Sep 11 09:21:22 mail2 postfix/cleanup[23372]: 3cZZKZ2WvdzBt9C: 
message-id=
Sep 11 09:21:22 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: 
from=, size=36991, nrcpt=1 (queue active)
Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: conversation with 
mail.vivantes.de[62.220.2.98] timed out while sending end of data -- message 
may be sent more than once
Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: 
to=, relay=mail1.vivantes.de[62.220.2.99]:25, delay=601, 
delays=0.01/0/601/0.55, dsn=2.0.0, status=sent (250 2.0.0 r8B7VMsm013069 
Message accepted for delivery)
Sep 11 09:31:23 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: removed

It seems the first delivery fails (after 10 minutes), and the
subsequent retry succeeds immediately. It strikes me as odd that the
mail is being retried in the very second the error message ist being
logged.

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: message may be sent more than once

[Ralf Hildebrandt]

* K bharathan :
> Mar 18 17:25:19 relay2 postfix/smtp[21383]: 5470B21265:
> to=<41b.4.74998426-6452...@whereverstormy.com>, relay=
> mail.WhereverStormy.com[173.46.193.75]:25, delay=418568,
> delays=418439/0.46/4.7/123, dsn=4.4.2, status=deferred (lost connection with
> mail.WhereverStormy.com[173.46.193.75] while sending end of data -- message
> may be sent more than once)
> 
> the above is from my relay;
> "while sending end of data -- message may be sent more than once)" what does
> this mean?

It means that the connection to mail.WhereverStormy.com has been lost
while sending end of data and THUS the message may be sent more than
once (it may have been sent now, but Postfix didn't get a
confirmation, thus Postfix will resend)

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
"There are two major products that come out of Berkeley: LSD and UNIX. 
We don't believe this to be a coincidence."  -- Jeremy S. Anderson

Re: message may be sent more than once

[Wietse Venema]

K bharathan:
> Mar 18 17:25:19 relay2 postfix/smtp[21383]: 5470B21265:
> to=<41b.4.74998426-6452...@whereverstormy.com>, relay=
> mail.WhereverStormy.com[173.46.193.75]:25, delay=418568,
> delays=418439/0.46/4.7/123, dsn=4.4.2, status=deferred (lost connection with
> mail.WhereverStormy.com[173.46.193.75] while sending end of data -- message
> may be sent more than once)
> 
> the above is from my relay;
> "while sending end of data -- message may be sent more than once)" what does
> this mean?

You did not read the first part of the error message:

(lost connection with ... while sending end of data -- message may
be sent more than once)

The connection was broken before server confirmed that the mail
has arrived.

Wietse

Re: "message may be sent more than once"

[Paul Hoffman]

On Wed, Sep 11, 2013 at 10:19:19AM +0200, Ralf Hildebrandt wrote:
> Sep 11 09:21:22 mail2 postfix/cleanup[23372]: 3cZZKZ2WvdzBt9C: 
> message-id=
> Sep 11 09:21:22 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: 
> from=, size=36991, nrcpt=1 (queue active)
> Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: conversation with 
> mail.vivantes.de[62.220.2.98] timed out while sending end of data -- message 
> may be sent more than once
> Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: 
> to=, relay=mail1.vivantes.de[62.220.2.99]:25, 
> delay=601, delays=0.01/0/601/0.55, dsn=2.0.0, status=sent (250 2.0.0 
> r8B7VMsm013069 Message accepted for delivery)
> Sep 11 09:31:23 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: removed
> 
> It seems the first delivery fails (after 10 minutes), and the
> subsequent retry succeeds immediately. It strikes me as odd that the
> mail is being retried in the very second the error message ist being
> logged.

The message was immediately accepted by the second MX host for 
vivantes.de after the first MX host timed out.

Paul.

-- 
Paul Hoffman 
Systems Librarian
Fenway Libraries Online
c/o Wentworth Institute of Technology
550 Huntington Ave.
Boston, MA 02115
(617) 442-2384 (FLO main number)

Re: "message may be sent more than once"

[Ralf Hildebrandt]

* Paul Hoffman :
> On Wed, Sep 11, 2013 at 10:19:19AM +0200, Ralf Hildebrandt wrote:
> > Sep 11 09:21:22 mail2 postfix/cleanup[23372]: 3cZZKZ2WvdzBt9C: 
> > message-id=
> > Sep 11 09:21:22 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: 
> > from=, size=36991, nrcpt=1 (queue active)
> > Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: conversation 
> > with mail.vivantes.de[62.220.2.98] timed out while sending end of data -- 
> > message may be sent more than once
> > Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: 
> > to=, relay=mail1.vivantes.de[62.220.2.99]:25, 
> > delay=601, delays=0.01/0/601/0.55, dsn=2.0.0, status=sent (250 2.0.0 
> > r8B7VMsm013069 Message accepted for delivery)
> > Sep 11 09:31:23 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: removed
> > 
> > It seems the first delivery fails (after 10 minutes), and the
> > subsequent retry succeeds immediately. It strikes me as odd that the
> > mail is being retried in the very second the error message ist being
> > logged.
> 
> The message was immediately accepted by the second MX host for 
> vivantes.de after the first MX host timed out.

ARGH. Thanks for the second set of eyes.
I just didn't see mail|mail1 (was expecting mail|mail2 or something
along those lines).

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: "message may be sent more than once"

[Wietse Venema]

Ralf Hildebrandt:
[ Charset UTF-8 unsupported, converting... ]
> Sep 11 09:21:22 mail2 postfix/cleanup[23372]: 3cZZKZ2WvdzBt9C: 
> message-id=
> Sep 11 09:21:22 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: 
> from=, size=36991, nrcpt=1 (queue active)
> Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: conversation with 
> mail.vivantes.de[62.220.2.98] timed out while sending end of data -- message 
> may be sent more than once
> Sep 11 09:31:23 mail2 postfix/smtp[22134]: 3cZZKZ2WvdzBt9C: 
> to=, relay=mail1.vivantes.de[62.220.2.99]:25, 
> delay=601, delays=0.01/0/601/0.55, dsn=2.0.0, status=sent (250 2.0.0 
> r8B7VMsm013069 Message accepted for delivery)
> Sep 11 09:31:23 mail2 postfix/qmgr[10759]: 3cZZKZ2WvdzBt9C: removed
> 
> It seems the first delivery fails (after 10 minutes), and the
> subsequent retry succeeds immediately. It strikes me as odd that the
> mail is being retried in the very second the error message ist being
> logged.

Delivery fails to the primary MX host (mail.vivantes.de) and then
it succeeds to the secondary MX host. Why should Postfix wait when
it switches from primary to secondary MX?

Wietse

Re: "message may be sent more than once"

[Ralf Hildebrandt]

* Wietse Venema :

> Delivery fails to the primary MX host (mail.vivantes.de) and then
> it succeeds to the secondary MX host. Why should Postfix wait when
> it switches from primary to secondary MX?

PEBCAK (on my side here).

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

lost connection while sending end of data -- message may be sent more than once

[Stanisław Findeisen]

Hi

I am getting such errors in the log:

Feb 24 10:03:21 * postfix/smtp[9203]: C2EFF1823C1: lost connection with
ASPMX.L.GOOGLE.COM[74.125.43.27] while sending end of data -- message
may be sent more than once

This happens many times a day with various servers --- not just
google.com. Otherwise everything works fine.

What do you think the problem is?

The puzzling thing is also this one:

$ host 74.125.43.27
27.43.125.74.in-addr.arpa domain name pointer bw-in-f27.1e100.net.

??

-- 
Eisenbits - proven software solutions: http://www.eisenbits.com/
OpenPGP: DFD9 0146 3794 9CF6 17EA  D63F DBF5 8AA8 3B31 FE8A



signature.asc
Description: OpenPGP digital signature

Timed out while sending end of data -- message may be sent more than once

[Thomas Kristensen]

Hey

I got this strange problem with postfix 3.1.0.
I got this one server that doesn't get all the mails, queued for it. Some mails 
gets the error in subject. 
And if I do a tcpdump on the tcp stream I see this everytime:

(the content has been wiped for some information)

220 [794178adb94846f8975ac93c9a320e4a] SMTP Version: 1.3.1.34773 21:18:18 12. 
august 2018
EHLO (removed)
250 [794178adb94846f8975ac93c9a320e4a] OK
MAIL FROM: (removed)
250 [794178adb94846f8975ac93c9a320e4a] OK
RCPT TO: (removed)
250 [794178adb94846f8975ac93c9a320e4a] OK
DATA
354 [794178adb94846f8975ac93c9a320e4a] Start mail input; end with .
Received: from Server (unknown [(removed)])
by Server (Postfix) with ESMTP id 41pBtg5rKGzqYnC
for (removed); Sun, 12 Aug 2018 10:32:27 +0200 (CEST)
MIMEVersion: 1.0
MIME-Version: 1.0
From: (removed)
To: (removed)
Date: 12 Aug 2018 10:32:27 +0200
Subject: (removed)
Content-Type: multipart/mixed;
 boundary=--boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059


boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable


boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
Content-Type: application
.


As you can see the stream stops before the last mimepart is done. This is 
captured with tcpdump on the server with postfix 3.1.0
We use SMTP protocol to transport small files, so the body of the mail is not 
importen but the attachment is the actual message we want to transport.

This is only a problem with about 5-10% of the delivers to this one host. We 
got about 40-5 messages in and out, to many servers, everyday and only see 
this problem with one host we try to deliver to. 

If I do a postqueue -I on the queueid It still fails, but if I make the sender 
resend it, so it is a whole new message in postfix, it goes fine without 
problems. 
So the question is, why does some of the mails fail on the server, with no 
option to requeue them? 

Med venlig hilsen
Thomas Kristensen

Storhaven 12 - 7100 Vejle
Tlf: 75 72 54 99 - Fax: 75 72 65 33
E-mail: t...@multimed.dk

Re: lost connection while sending end of data -- message may be sent more than once

[Wietse Venema]

Stanis??aw Findeisen:
> Hi
> 
> I am getting such errors in the log:
> 
> Feb 24 10:03:21 * postfix/smtp[9203]: C2EFF1823C1: lost connection with
> ASPMX.L.GOOGLE.COM[74.125.43.27] while sending end of data -- message
> may be sent more than once
> 
> This happens many times a day with various servers --- not just
> google.com. Otherwise everything works fine.
> 
> What do you think the problem is?

Don't speculate, measure. Look at packet traces with tcpdump,
and see at what point the connection breaks.

- Does the problem only happen with connections that use TCP window
scaling? If so, you have a borked firewall that mis-implements TCP.

- Does the problem go away with small messages? If so, you have a
broken IP path MTU problem.

And so on, there are many reasons why TCP can break.

Wietse

> The puzzling thing is also this one:
> 
> $ host 74.125.43.27
> 27.43.125.74.in-addr.arpa domain name pointer bw-in-f27.1e100.net.
> 
> ??
> 
> -- 
> Eisenbits - proven software solutions: http://www.eisenbits.com/
> OpenPGP: DFD9 0146 3794 9CF6 17EA  D63F DBF5 8AA8 3B31 FE8A
> 
-- End of PGP section, PGP failed!

Re: lost connection while sending end of data -- message may be sent more than once

[Victor Duchovni]

On Thu, Feb 24, 2011 at 10:29:50AM +0100, Stanis??aw Findeisen wrote:

> The puzzling thing is also this one:
> 
> $ host 74.125.43.27
> 27.43.125.74.in-addr.arpa domain name pointer bw-in-f27.1e100.net.

Lets guess what "1e100.net" is: looks like "scientific notation" for 1
times 10 to the 100th power, which is sometimes called a "googol"...

-- 
Viktor.

Re: lost connection while sending end of data -- message may be sent more than once

[Stanisław Findeisen]

On 2011-02-24 13:09, Wietse Venema wrote:
> Stanisław Findeisen:
>> Hi
>>
>> I am getting such errors in the log:
>>
>> Feb 24 10:03:21 * postfix/smtp[9203]: C2EFF1823C1: lost connection with
>> ASPMX.L.GOOGLE.COM[74.125.43.27] while sending end of data -- message
>> may be sent more than once
>>
>> This happens many times a day with various servers --- not just
>> google.com. Otherwise everything works fine.
>>
>> What do you think the problem is?
> 
> Don't speculate, measure. Look at packet traces with tcpdump,
> and see at what point the connection breaks.
> 
> - Does the problem only happen with connections that use TCP window
> scaling? If so, you have a borked firewall that mis-implements TCP.
> 
> - Does the problem go away with small messages? If so, you have a
> broken IP path MTU problem.
> 
> And so on, there are many reasons why TCP can break.
> 
>   Wietse

Thank you, Wietse. Indeed the problem was with too big (1500) IPv4
packets going out. It is not clear yet why is the MTU calculation broken.

Some hop must really be broken there, because there is DF flag set on
outgoing IPv4 packets and we're not getting any ICMP Fragmentation
Needed back (we're just getting nothing).

Anyway this doesn't seem to be Postfix related.

-- 
Eisenbits - proven software solutions: http://www.eisenbits.com/
OpenPGP: DFD9 0146 3794 9CF6 17EA  D63F DBF5 8AA8 3B31 FE8A



signature.asc
Description: OpenPGP digital signature

PIX & timed out while sending end of data -- message may be sent more than once

[John Allen]

I am getting the following message/errors

Oct  5 00:00:10 myhost postfix/qmgr[18862]: 125BC2400A7: 
from=, size=2760, nrcpt=1 (queue active)
Oct  5 00:00:10 myhost postfix/smtp[28713]: 125BC2400A7: enabling PIX 
workarounds: disable_esmtp delay_dotcrlf for 
mail.abc.tld[123.456.789.123]:25
Oct  5 00:10:22 myhost postfix/smtp[28713]: 125BC2400A7: 
to=, relay=mail.abc.tld[123.456.789.123]:25, 
delay=187500, delays=186888/0.01/0.16/612, dsn=4.4.2, status=deferred 
(conversation with mail.abc.tld[123.456.789.123] timed out while sending 
end of data -- message may be sent more than once)

.
.
Oct  5 01:20:10 myhost postfix/qmgr[18862]: 125BC2400A7: 
from=, size=2760, nrcpt=1 (queue active)
Oct  5 01:20:10 myhost postfix/smtp[30509]: 125BC2400A7: enabling PIX 
workarounds: disable_esmtp delay_dotcrlf for 
mail.abc.tld[123.456.789.123]:25
Oct  5 01:30:20 myhost postfix/smtp[30509]: 125BC2400A7: 
to=, relay=mail.abc.tld[123.156.789.123]:25, 
delay=192299, delays=191688/0.01/0.23/610, dsn=4.4.2, status=deferred


My understanding is that the first message is the original, and the 
second and subsequent are postfix retrying the delivery.

my postconf -n output follows below,
1) Am I misconfigured?
2) How can I handle this sort of problem in future?

TIA
JohnA

===postconf output
alias_database = $alias_maps
alias_maps = hash:/etc/aliases
allow_untrusted_routing = no
biff = no
body_checks = regexp:/etc/postfix/maps/body_checks
bounce_size_limit = 65536
broken_sasl_auth_clients = yes
command_directory = /usr/sbin
config_directory = /etc/postfix
content_filter = amavisfeed:[127.0.0.1]:10024
daemon_directory = /usr/lib/postfix
data_directory = /var/lib/postfix
default_privs = nobody
default_process_limit = 20
delay_warning_time = 12
disable_vrfy_command = yes
header_checks = regexp:/etc/postfix/maps/header_checks
header_size_limit = 32768
home_mailbox = Maildir/
html_directory = /usr/share/doc/postfix/html
inet_interfaces = all
inet_protocols = all
local_destination_concurrency_limit = 5
mail_owner = postfix
mailbox_command = /usr/lib/dovecot/deliver
mailbox_transport = dovecot
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
message_size_limit = 32768000
mydestination = localhost, localhost.localdomain, localdomain
mydomain = linderly.com
myhostname = mail.$mydomain
mynetworks = 127.0.0.0/8, 192.168.0.0/16, 74.116.186.176/28, [::1]/128, 
[2001:470:1f10:6bf::]/64

myorigin = $mydomain
newaliases_path = /usr/bin/newaliases.postfix
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix
recipient_delimiter = +
relocated_maps = hash:/etc/postfix/maps/relocated
sample_directory = /usr/share/doc/postfix-2.5.6/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_cert_file = /etc/ssl/certs/Linderly_Mail_SSL.crt
smtp_tls_enforce_peername = no
smtp_tls_key_file = /etc/ssl/private/Linderly_Mail_SSL.key
smtp_tls_note_starttls_offer = yes
smtp_tls_security_level = may
smtpd_banner = $myhostname ESMTP
smtpd_data_restrictions = reject_multi_recipient_bounce,
reject_unauth_pipelining,permit

smtpd_delay_reject = yes
smtpd_error_sleep_time = 5s
smtpd_hard_error_limit = 20
smtpd_helo_required = yes
smtpd_recipient_limit = 128
smtpd_recipient_restrictions = reject_invalid_hostname,
reject_non_fqdn_hostname,reject_non_fqdn_sender,
reject_non_fqdn_recipient,reject_unknown_sender_domain,
reject_unknown_recipient_domain,permit_mynetworks, 
permit_sasl_authenticated,reject_unauth_destination,
check_recipient_access pcre:/etc/postfix/maps/recipient_checks.pcre,
check_helo_access pcre:/etc/postfix/maps/helo_checks.pcre,
check_helo_access hash:/etc/postfix/maps/helo_checks,
check_sender_access hash:/etc/postfix/maps/sender_checks,
check_client_access hash:/etc/postfix/maps/client_checks,
check_client_access cidr:/etc/postfix/maps/client_checks.cidr,
reject_rbl_client zen.spamhaus.org,check_policy_service 
inet:127.0.0.1:10023,permit

smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_path = private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_soft_error_limit = 10
smtpd_tls_ask_ccert = yes
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/ssl/certs/Linderly_Mail_SSL.crt
smtpd_tls_key_file = /etc/ssl/private/Linderly_Mail_SSL.key
smtpd_tls_loglevel = 1
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_mandatory_protocols = SSLv3, TLSv1
smtpd_tls_received_header = yes
smtpd_tls_req_ccert = no
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_session_cache_timeout = 3600s
soft_bounce = no
strict_rfc821_envelopes = yes
tls_random_source = dev:/dev/urandom
virtual_alias_maps = hash:/etc/postfix/maps/valiases
virtual_gid_maps

Re: Timed out while sending end of data -- message may be sent more than once

[Benny Pedersen]

Invalid mimetype?

milter out of mem or temp storage?



boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
Content-Type: application
.

Re: Timed out while sending end of data -- message may be sent more than once

[Viktor Dukhovni]

On Sun, Aug 12, 2018 at 08:50:19PM +, Thomas Kristensen wrote:

> DATA
> 354 [794178adb94846f8975ac93c9a320e4a] Start mail input; end with 
> .
> Received: from Server (unknown [(removed)])
>   by Server (Postfix) with ESMTP id 41pBtg5rKGzqYnC
>   for (removed); Sun, 12 Aug 2018 10:32:27 +0200 (CEST)
> MIMEVersion: 1.0
> MIME-Version: 1.0
> From: (removed)
> To: (removed)
> Date: 12 Aug 2018 10:32:27 +0200
> Subject: (removed)
> Content-Type: multipart/mixed;
>  boundary=--boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
> 
> 
> boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
> Content-Type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: quoted-printable
> 
> 
> boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
> Content-Type: application
> .
> 
> As you can see the stream stops before the last mimepart is done.

Perhaps this is the end of the queued message? If you send truncated
MIME into Postfix, it will forward truncated MIME.  You have not
said anything about the content of the queue file with the deferred
message.  How big is the queued message.  Where are the logs for
the message arriving? ...

-- 
Viktor.

Re: Timed out while sending end of data -- message may be sent more than once

[Matus UHLAR - fantomas]

On 12.08.18 20:50, Thomas Kristensen wrote:

Subject: Timed out while sending end of data -- message may be sent more
than once



I got this strange problem with postfix 3.1.0.
I got this one server that doesn't get all the mails, queued for it. Some mails 
gets the error in subject.
And if I do a tcpdump on the tcp stream I see this everytime:

[...]


As you can see the stream stops before the last mimepart is done. This is 
captured with tcpdump on the server with postfix 3.1.0

This is only a problem with about 5-10% of the delivers to this one host.
We got about 40-5 messages in and out, to many servers, everyday and
only see this problem with one host we try to deliver to.


the host may have problem, or just scans the mail for too long.

Is the smtp_data_done_timeout default 600s?
what are the timestamps in the tcpdump? does the timeout happen after those
600 seconds?

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety. -- Benjamin Franklin, 1759

Re: Timed out while sending end of data -- message may be sent more than once

[Bill Cole]

On 12 Aug 2018, at 16:50, Thomas Kristensen wrote:


Hey

I got this strange problem with postfix 3.1.0.
I got this one server that doesn't get all the mails, queued for it. 
Some mails gets the error in subject.

And if I do a tcpdump on the tcp stream I see this everytime:

(the content has been wiped for some information)



[...]



boundary_274246_f400b577-4e93-4ffd-b5ec-355c7a0b5059
Content-Type: application
.


As you can see the stream stops before the last mimepart is done.


Which *should not* make any difference, since the terminating 
.CRLF> is present. Unfortunately, the bogus Content-Type and 
unterminated MIME part may be the cause.



This is captured with tcpdump on the server with postfix 3.1.0
We use SMTP protocol to transport small files, so the body of the mail 
is not importen but the attachment is the actual message we want to 
transport.


This is only a problem with about 5-10% of the delivers to this one 
host. We got about 40-5 messages in and out, to many servers, 
everyday and only see this problem with one host we try to deliver to.


So, the receiving host is broken. It is not answering the submitted data 
for over 10 minutes (unless you've shortened the smtp_data_done_timeout 
value? Don't do that...) so presumably it is doing some sort of content 
filtering or delivery procedure which is hanging. A badly-designed 
content filter or heavyweight delivery agent (e.g. Exchange) could make 
the mistake of demanding that MIME parts have parseable headers and/or 
non-null bodies and/or closing boundaries before they accept them as 
complete and process them, ignoring the fact that a misconstructed 
message like this one might be sent and terminated correctly at the SMTP 
level with the last part being a useless stub as shown.



If I do a postqueue -I on the queueid It still fails, but if I make 
the sender resend it, so it is a whole new message in postfix, it goes 
fine without problems.
So the question is, why does some of the mails fail on the server, 
with no option to requeue them?


You would probably need to provide postconf -n output and logging of all 
activity regarding a particular message to get a useful answer to that.

conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

Today I found that some sites behind a PIX/ASA firewall with "smtp
protocol fixup" would not accept DKIM signed mails.

Solution:
=

master.cf:
nodkimunix  -   -   -   -   -   smtp -o 
smtp_header_checks=pcre:/etc/postfix/no_dkim.pcre

main.cf:
transport_maps = cdb:/etc/postfix/transport

and in /etc/postfix/transport:
mrnaz.com   nodkim:

/etc/postfix/no_dkim.pcre contains:
/^DKIM-Signature:/  IGNORE
# this strips a DKIM Signature


-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: PIX & timed out while sending end of data -- message may be sent more than once

[Noel Jones]

On 10/5/2011 6:23 PM, John Allen wrote:
> I am getting the following message/errors
> 
> Oct  5 00:00:10 myhost postfix/qmgr[18862]: 125BC2400A7:
> from=, size=2760, nrcpt=1 (queue active)
> Oct  5 00:00:10 myhost postfix/smtp[28713]: 125BC2400A7: enabling
> PIX workarounds: disable_esmtp delay_dotcrlf for
> mail.abc.tld[123.456.789.123]:25
> Oct  5 00:10:22 myhost postfix/smtp[28713]: 125BC2400A7:
> to=, relay=mail.abc.tld[123.456.789.123]:25,
> delay=187500, delays=186888/0.01/0.16/612, dsn=4.4.2,
> status=deferred (conversation with mail.abc.tld[123.456.789.123]
> timed out while sending end of data -- message may be sent more than
> once)
> .
> .
> Oct  5 01:20:10 myhost postfix/qmgr[18862]: 125BC2400A7:
> from=, size=2760, nrcpt=1 (queue active)
> Oct  5 01:20:10 myhost postfix/smtp[30509]: 125BC2400A7: enabling
> PIX workarounds: disable_esmtp delay_dotcrlf for
> mail.abc.tld[123.456.789.123]:25
> Oct  5 01:30:20 myhost postfix/smtp[30509]: 125BC2400A7:
> to=, relay=mail.abc.tld[123.156.789.123]:25,
> delay=192299, delays=191688/0.01/0.23/610, dsn=4.4.2, status=deferred
> 
> My understanding is that the first message is 

first message is the first delivery attempt.  The attempt timed out;
it's impossible to know if the delivery actually succeeded or not.
Postfix assumes "not" in order to not lose mail.


> the original, and the
> second and subsequent are postfix retrying the delivery.

Yes.

> my postconf -n output follows below,
> 1) Am I misconfigured?

Nothing here that would cause this problem.  Some unrelated nit
picks that I'll ignore.  Maybe someone else will bite.

> 2) How can I handle this sort of problem in future?

Wait for the recipient to fix their system?  If you're really
interested, you can capture a TCP recording of the session and find
out if there's anything dodgy going on with the network -- but more
than likely the other end just quits talking.
http://www.postfix.org/DEBUG_README.html#sniffer

If you want to twiddle knobs, you can disable the pix delay_dotcrlf
and see if it makes any difference.  doubtful.
http://www.postfix.org/postconf.5.html#smtp_pix_workarounds
# main.cf
smtp_pix_workarounds = disable_esmtp



  -- Noel Jones

Re: PIX & timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

John,

> Oct  5 00:10:22 myhost postfix/smtp[28713]: 125BC2400A7: 
> to=, relay=mail.abc.tld[123.456.789.123]:25, 
> delay=187500, delays=186888/0.01/0.16/612, dsn=4.4.2, status=deferred 
> (conversation with mail.abc.tld[123.456.789.123] timed out while sending 
> end of data -- message may be sent more than once)
> 
> My understanding is that the first message is the original, and the 
> second and subsequent are postfix retrying the delivery.
> my postconf -n output follows below,

Your message probably hit one of several Cisco PIX/ASA bugs
related to mail header section parsing:

http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml

> 1) Am I misconfigured?

Probably not.

> 2) How can I handle this sort of problem in future?

If it is your PIX or ASA or you can establish a contact with their
hostmaster, the box should be upgraded, or the smtp protocol fixup
(mis)feature turned off - best to do both.

If it is entirely out of control, removing a DKIM signature
header field for mail to such site will probably help.
Ralf posted some workaround here some time ago.

  Mark

Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Jørn Odberg]

Ok, I've seen a couple of this on the mailinglist earlier. But as I 
understand, each case is pretty much unique.



The case is: We're hosting a lot of linux-boxes around Norway, spread on 
different locations, working as library servers. Everyone running 
postfix. And we seldom have anything to do with the rest of the network 
or the Internet connection...


We're running postfix 2.4.10 at every location.

In this case, I'm pretty sure the problem lies at the gateway/firewall 
at one of the ends. But I need to know which end, and preferably also 
exactly what is wrong. Because I will have to tell some IT-dude at one 
of the locations what they need to change, and where. Or if I could 
change something in postfix/linux which may solve the problem.


I'm not good at reading tcpdump output, so that's why I write to you 
guys... Hoping someone will spot the error. :)




This problem is when a mail is sent from the server called NotBib, to 
the server called BamBib, located at two different places. Email from 
and to other locations seems to work ok, but I haven't tried too many. 
Small emails is sent correctly, but larger mails (about 1500 bytes and 
upwards, I think?) will not deliver... It takes a minute or two, and 
then I receive "conversation with NotBib(..and the rest of the domain) 
timed out while sending end of data -- message may be sent more than once".


I have tried sending a mail with a size of 1589 from NotBib to BamBib, 
with debugging turned on inside main.cf on both sides. I have also 
recorded the transmission with tcpdump, as specified on www.postfix.org.



Didn't want to risk anything by attaching the logs to this mail. ;)

All logs can be found at the URL: 
http://postfix.jorno.net/2009_04_22-BamBib-NotBib/


Output from sender:
NotBib-mail.log  (the relevant portion from mail.log)
NotBib.tcpdump (output from "tcpdump -w 
NotBib.DUMP.postfix.from.BamBib.tcpdump -s 0 -i eth0 host 159.130.66.44")

NotBib.postconf (output from "postconf -n")

Output from receiver:
BamBib-mail.log  (the relevant portion from mail.log)
BamBib.tcpdump (output from "tcpdump -w 
BamBib.DUMP.postfix.from.NotBib.tcpdump -s 0 -i eth0 host 
notweb.notodden.folkebibl.no")

BamBib.postconf (output from "postconf -n")


Please shout out if I need to include more documentation or logging. I 
will provide whatever necessary :)



Kind regards from Norway,
Jørn Odberg
--
_
Jørn Odberg, Bibliotek-Systemer AS, N-3271 Larvik, Norway
LPIC-1 ID#: LPI000164485

Tlf: +47 33 11 68 00
Fax: +47 33 11 68 22
j...@bibsyst.no <mailto:j...@bibsyst.no>

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Noel Jones]

On 6/14/2011 8:34 AM, Ralf Hildebrandt wrote:

Today I found that some sites behind a PIX/ASA firewall with "smtp
protocol fixup" would not accept DKIM signed mails.

Solution:
=

master.cf:
nodkimunix  -   -   -   -   -   smtp -o 
smtp_header_checks=pcre:/etc/postfix/no_dkim.pcre

main.cf:
transport_maps = cdb:/etc/postfix/transport

and in /etc/postfix/transport:
mrnaz.com   nodkim:

/etc/postfix/no_dkim.pcre contains:
/^DKIM-Signature:/  IGNORE
# this strips a DKIM Signature





I think I posted something almost exactly like this a while 
ago (year+?).  Anyway, I can confirm that I've had this same 
problem and came up with the same workaround, still in place.



  -- Noel Jones

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Noel Jones :

> I think I posted something almost exactly like this a while ago
> (year+?).  Anyway, I can confirm that I've had this same problem and
> came up with the same workaround, still in place.

Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Victor Duchovni]

On Tue, Jun 14, 2011 at 07:48:54PM +0200, Ralf Hildebrandt wrote:

> * Noel Jones :
> 
> > I think I posted something almost exactly like this a while ago
> > (year+?).  Anyway, I can confirm that I've had this same problem and
> > came up with the same workaround, still in place.
> 
> Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!

I guess you'd like:

smtp_pix_header_checks = ...

this feature would be rather a large concession to a problem that needs
to be fixed at the receiving system...

-- 
Viktor.

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

Ralf Hildebrandt:
> * Noel Jones :
> 
> > I think I posted something almost exactly like this a while ago
> > (year+?).  Anyway, I can confirm that I've had this same problem and
> > came up with the same workaround, still in place.
> 
> Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!

How does an SMTP client recognize an ASA box before it breaks email? 

Wietse

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Wietse Venema :

> > Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!
> 
> How does an SMTP client recognize an ASA box before it breaks email? 

Only from the /^[02 *]+$/ banner.

# telnet mx.interfree.it 25
Trying 213.158.72.46...
Connected to mx.interfree.it.
Escape character is '^]'.
220 **

# telnet mailamir.com 25
Trying 114.31.73.44...
Connected to mailamir.com.
Escape character is '^]'.
220 **

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

Ralf Hildebrandt:
> * Wietse Venema :
> 
> > > Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!
> > 
> > How does an SMTP client recognize an ASA box before it breaks email? 
> 
> Only from the /^[02 *]+$/ banner.
> 
> # telnet mx.interfree.it 25
> Trying 213.158.72.46...
> Connected to mx.interfree.it.
> Escape character is '^]'.
> 220 **
> 
> # telnet mailamir.com 25
> Trying 114.31.73.44...
> Connected to mailamir.com.
> Escape character is '^]'.
> 220 **

Hmm...

% telnet mailamir.com 25
Trying 114.31.73.44...
Connected to mailamir.com.
Escape character is '^]'.
220 **
help
502 5.5.2 Error: command not recognized

Wietse

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Victor Duchovni]

On Tue, Jun 14, 2011 at 02:18:43PM -0400, Wietse Venema wrote:

> > # telnet mailamir.com 25
> > Trying 114.31.73.44...
> > Connected to mailamir.com.
> > Escape character is '^]'.
> > 220 **
> 
> Hmm...
> 
> % telnet mailamir.com 25
> Trying 114.31.73.44...
> Connected to mailamir.com.
> Escape character is '^]'.
> 220 **
> help
> 502 5.5.2 Error: command not recognized

A Postfix system with a PIX in front of it and STARTTLS censored as
"XXXA" (same length).

Connected to mailamir.com[114.31.73.44]:25
< 220 **
> EHLO amnesiac.example.com
< 250-mailamir.com
< 250-PIPELINING
< 250-SIZE 2048
< 250-VRFY
< 250-ETRN
< 250-XXXA
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250 DSN

-- 
Viktor.

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

Ralf wrote:
> Today I found that some sites behind a PIX/ASA firewall with "smtp
> protocol fixup" would not accept DKIM signed mails.

But you already knew that!  :)

ASA bug CSCsy28792 and a couple of related header-parsing bugs,
triggered by encountering a "content-type" or "content-transfer-encoding"
in a header field body of some unrelated header field, such as an 'h' tag
of a DKIM signature:

  http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml


Mark

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Victor Duchovni :

> A Postfix system with a PIX in front of it and STARTTLS censored as
> "XXXA" (same length).

Yes, thought so too.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Mark Martinec :
> Ralf wrote:
> > Today I found that some sites behind a PIX/ASA firewall with "smtp
> > protocol fixup" would not accept DKIM signed mails.
> 
> But you already knew that!  :)

Yes I know.

> ASA bug CSCsy28792 and a couple of related header-parsing bugs,
> triggered by encountering a "content-type" or "content-transfer-encoding"
> in a header field body of some unrelated header field, such as an 'h' tag
> of a DKIM signature:
> 
>   http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml

Back then I didn't know the workaround!

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

> > How does an SMTP client recognize an ASA box before it breaks email? 
> 
> Only from the /^[02 *]+$/ banner.
> # telnet mx.interfree.it 25
> 220 **

I think the newer versions of ASA can be configured to let ESMTP pass through
without censoring the greeting, while still exhibiting one of the header
parsing bugs - which can lead to dropping the TCP session without
a RST (but with a message in the log ... which noone reads).

  Mark

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Mark Martinec :

> I think the newer versions of ASA can be configured to let ESMTP pass
> through without censoring the greeting, while still exhibiting one of
> the header parsing bugs - which can lead to dropping the TCP session
> without a RST (but with a message in the log ... which noone reads).

:(

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Robert Schetterer]

Am 14.06.2011 15:34, schrieb Ralf Hildebrandt:
> Today I found that some sites behind a PIX/ASA firewall with "smtp
> protocol fixup" would not accept DKIM signed mails.
> 
> Solution:
> =
> 
> master.cf:
> nodkimunix  -   -   -   -   -   smtp -o 
> smtp_header_checks=pcre:/etc/postfix/no_dkim.pcre
> 
> main.cf:
> transport_maps = cdb:/etc/postfix/transport
> 
> and in /etc/postfix/transport:
> mrnaz.com   nodkim:
> 
> /etc/postfix/no_dkim.pcre contains:
> /^DKIM-Signature:/  IGNORE
> # this strips a DKIM Signature
> 
> 
yes there a few of them out there

-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Robert Schetterer]

Am 14.06.2011 20:48, schrieb Ralf Hildebrandt:
> * Mark Martinec :
> 
>> I think the newer versions of ASA can be configured to let ESMTP pass
>> through without censoring the greeting, while still exhibiting one of
>> the header parsing bugs - which can lead to dropping the TCP session
>> without a RST (but with a message in the log ... which noone reads).
> 
> :(
> 
make it more public , firewall admins may awake, in germany heise
postings help sometimes *g


-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

Wietse Venema:
> Hmm...
> 
> % telnet mailamir.com 25
> Trying 114.31.73.44...
> Connected to mailamir.com.
> Escape character is '^]'.
> 220 **
> help
> 502 5.5.2 Error: command not recognized

FYI, this is how I quickly identify Postfix MTAs.

Wietse

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Robert Schetterer :

> make it more public , firewall admins may awake, in germany heise
> postings help sometimes *g

For that one would need large scale statistics.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Benny Pedersen]

On Tue, 14 Jun 2011 19:48:54 +0200, Ralf Hildebrandt wrote:

* Noel Jones :


I think I posted something almost exactly like this a while ago
(year+?).  Anyway, I can confirm that I've had this same problem and
came up with the same workaround, still in place.


Yeah. Maybe it would make a cool addition to smtp_pix_workarounds!


or list bad domains as rfc-ignorant if there is a rfc for this

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Noel Jones]

On 6/14/2011 5:49 PM, Benny Pedersen wrote:

On Tue, 14 Jun 2011 19:48:54 +0200, Ralf Hildebrandt wrote:

* Noel Jones :


I think I posted something almost exactly like this a while
ago
(year+?). Anyway, I can confirm that I've had this same
problem and
came up with the same workaround, still in place.


Yeah. Maybe it would make a cool addition to
smtp_pix_workarounds!


or list bad domains as rfc-ignorant if there is a rfc for this



No, there is no RFC that says "you must receive my properly 
formatted email even if your software chokes on it".


I was thinking along the lines of a smtp_pix_workarounds 
keyword like "removeDKIM" or, more general and more complex, 
"removeheaders" with a matching smtp_pix_removeheaders list of 
header names to remove.


the choices I see are

A) single-purpose workaround:
smtp_pix_workarounds = removeDKIM ...

B) general anti-choke workaround
smtp_pix_workarounds = removeheaders
smtp_pix_removeheaders = DKIM, X-foo, Bar

(proposed docs available if there is any interest)

C) use existing smtp_header_checks solution.


For me, the existing workaround (master.cf dumbpix transport 
with -o smtp_header_checks) is sufficient.  I currently have 
only two domains on the dumbpix transport -- apparently 
unrelated government agencies, a school system in one city, 
police in another.




  -- Noel Jones

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Benny Pedersen]

On Tue, 14 Jun 2011 19:32:39 -0500, Noel Jones wrote:


C) use existing smtp_header_checks solution.


extend to smtp_header_checks_maps, and then use any maps postfix 
support


is smtp_header_checks already pr recipients server ?

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Noel Jones]

On 6/14/2011 7:42 PM, Benny Pedersen wrote:

On Tue, 14 Jun 2011 19:32:39 -0500, Noel Jones wrote:


C) use existing smtp_header_checks solution.


extend to smtp_header_checks_maps, and then use any maps
postfix support


That's an interesting idea in itself, but in the scope of pix 
workarounds it's not a huge improvement since it still 
requires manual intervention per server/domain.


anyway, don't think it's possible.  I think all possible 
tables would need to be loaded before postfix knew which one 
to use, or postfix would need to wastefully launch a new smtp 
for each delivery.





is smtp_header_checks already pr recipients server ?



No, currently either a global setting or custom transports 
with -o smtp_header_checks option.


I was thinking a setting integrated with smtp_pix_workarounds 
would be more automatic, with little maintenance once configured.



  -- Noel Jones

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Victor Duchovni]

On Tue, Jun 14, 2011 at 08:05:24PM -0500, Noel Jones wrote:

> I was thinking a setting integrated with smtp_pix_workarounds would be more 
> automatic, with little maintenance once configured.

Given that the banner detection is incomplete (some pixen are not
obviously such) one still needs manual configuration for some cases,
so I am not convinced that any new feature is warranted, the receiving
systems need to be incented to fix their bug.

-- 
Viktor.

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Benny Pedersen]

On Tue, 14 Jun 2011 20:05:24 -0500, Noel Jones wrote:


That's an interesting idea in itself, but in the scope of pix
workarounds it's not a huge improvement since it still requires 
manual

intervention per server/domain.


fail2ban could be ones friend if postfix have this

fail2ban then just grep logs for outgoing mails that failed pr ip, and 
add this header ignore pr cidr maps



anyway, don't think it's possible.  I think all possible tables would
need to be loaded before postfix knew which one to use, or postfix
would need to wastefully launch a new smtp for each delivery.


as is now its not, but i think it could be solved :-)


is smtp_header_checks already pr recipients server ?



No, currently either a global setting or custom transports with -o
smtp_header_checks option.


okay


I was thinking a setting integrated with smtp_pix_workarounds would
be more automatic, with little maintenance once configured.


suggest pfsense is out of the question for hosts that runs cisco 
hardware

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Noel Jones]

On 6/14/2011 8:22 PM, Victor Duchovni wrote:

On Tue, Jun 14, 2011 at 08:05:24PM -0500, Noel Jones wrote:


I was thinking a setting integrated with smtp_pix_workarounds would be more
automatic, with little maintenance once configured.


Given that the banner detection is incomplete (some pixen are not
obviously such) one still needs manual configuration for some cases,
so I am not convinced that any new feature is warranted, the receiving
systems need to be incented to fix their bug.



OTOH, the current pix detection and workarounds are not 
useless, so extending/improving them is worth discussing -- 
even if not necessarily worth doing.


At this time I'm inclined to set this aside.  The DKIM bug 
doesn't seem to be widespread; there is no compelling case to 
add a new workaround right now.


Maybe an example of the current smtp_header_checks workaround 
(Ralf's was fine) could be added to the docs somewhere rather 
than a feature change.



  -- Noel Jones

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Ralf Hildebrandt]

* Benny Pedersen :

> fail2ban could be ones friend if postfix have this
> 
> fail2ban then just grep logs for outgoing mails that failed pr ip,
> and add this header ignore pr cidr maps

Yeah, that's a great idea!

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Benny Pedersen]

On Wed, 15 Jun 2011 08:39:11 +0200, Ralf Hildebrandt wrote:

* Benny Pedersen :


fail2ban could be ones friend if postfix have this

fail2ban then just grep logs for outgoing mails that failed pr ip,
and add this header ignore pr cidr maps


Yeah, that's a great idea!


it is ?, oh thanks :-)

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Robert Schetterer]

Am 15.06.2011 08:39, schrieb Ralf Hildebrandt:
> * Benny Pedersen :
> 
>> fail2ban could be ones friend if postfix have this
>>
>> fail2ban then just grep logs for outgoing mails that failed pr ip,
>> and add this header ignore pr cidr maps
> 
> Yeah, that's a great idea!
> 
but what if there are other reasons, ok it doesnt hurt to much
to remove dkim sigs, but the admin should be informed
to this, or maybe it should expire, as far i remember this can be done
with fail2ban too ( but i may fail here )

-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

On Wednesday June 15 2011 05:42:36 Noel Jones wrote:
> At this time I'm inclined to set this aside.  The DKIM bug
> doesn't seem to be widespread; there is no compelling case to
> add a new workaround right now.

Indeed the situation has much improved in the past year or two.

Many sites have turned off smtp fixups or upgraded their ASA
firmware or both. It also helps to send mail to postmasters of
affected sites with a pointer to Ralf's web page and the Heise
article, and suggest turning off the (mis)feature.

Perhaps the incentive was when they started missing some of the
mail from gmail.com and the like.

  Mark

Re: conversation with ... timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

Victor Duchovni:
> On Tue, Jun 14, 2011 at 08:05:24PM -0500, Noel Jones wrote:
> 
> > I was thinking a setting integrated with smtp_pix_workarounds would be more 
> > automatic, with little maintenance once configured.
> 
> Given that the banner detection is incomplete (some pixen are not
> obviously such) one still needs manual configuration for some cases,
> so I am not convinced that any new feature is warranted, the receiving
> systems need to be incented to fix their bug.

If enough "big mailers" sign their email (gmail, yahoo, etc.)
then that will provide the incentive.

Wietse

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Jørn Odberg]

Would I need to do this at the sender or the receiver? Or both ends?


Thanks for the reply, Wietse. And thanks for Postfix.  :-)

Kind regards from Norway,
Jørn Odberg



Wietse Venema skrev:

Turn off TCP window scaling.
http://www.google.com/search?q=tcp+window+scaling

Wietse
  


--
_
Jørn Odberg, Bibliotek-Systemer AS, N-3271 Larvik, Norway
LPIC-1 ID#: LPI000164485

Tlf: +47 33 11 68 00
Fax: +47 33 11 68 22
j...@bibsyst.no 

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

Turn off TCP window scaling.
http://www.google.com/search?q=tcp+window+scaling

Wietse

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Sahil Tandon]

On Wed, 22 Apr 2009, Jørn Odberg wrote:

> Would I need to do this at the sender or the receiver? Or both ends?

Do it on your end, which is what you control.

-- 
Sahil Tandon 

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Jørn Odberg]

Hello Sahil, and thanks for your reply.


As I said in the first email, I control both ends (both the sender- and 
the receiver-server). But I do not control neither network-connectivity 
or Internet-connectivity at either sites.


I did try turning of Window Scaling at both ends, but it did not help at 
all. It still won't deliver.



I know there is a Cisco PIX at the senders side. They have already 
turned of the ESMTP Fixup (or fuckup if you'd like) "feature" by an 
earlier request from me, and that solved some problems. But it seems 
like the damn PIX has some other "features" which fucks up some more... 
Just don't know what.



Kind regards,
Jørn


Sahil Tandon skrev:

On Wed, 22 Apr 2009, Jørn Odberg wrote:

  

Would I need to do this at the sender or the receiver? Or both ends?



Do it on your end, which is what you control.

  


--
_
Jørn Odberg, Bibliotek-Systemer AS, N-3271 Larvik, Norway
LPIC-1 ID#: LPI000164485

Tlf: +47 33 11 68 00
Fax: +47 33 11 68 22
j...@bibsyst.no 

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Barney Desmond]

2009/4/22 Jørn Odberg :
> As I said in the first email, I control both ends (both the sender- and the
> receiver-server). But I do not control neither network-connectivity or
> Internet-connectivity at either sites.
>
> I did try turning of Window Scaling at both ends, but it did not help at
> all. It still won't deliver.

Unless I'm mistaken, disabling window-scaling at one end *should* be
enough (it's a "negotiated" feature, right?). You should be able to
confirm whether it's on or off by looking at a fresh tcpdump, which
would be advisable now.

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Victor Duchovni]

On Wed, Apr 22, 2009 at 11:57:30PM +1000, Barney Desmond wrote:

> > As I said in the first email, I control both ends (both the sender- and the
> > receiver-server). But I do not control neither network-connectivity or
> > Internet-connectivity at either sites.
> >
> > I did try turning of Window Scaling at both ends, but it did not help at
> > all. It still won't deliver.
> 
> Unless I'm mistaken, disabling window-scaling at one end *should* be
> enough (it's a "negotiated" feature, right?).

No, each side advertises a scale, and turning it "off" typically means
advertising a scale of 2^0 (i.e. 1) on your end, but the other side is
still free to use scaling.

> You should be able to
> confirm whether it's on or off by looking at a fresh tcpdump, which
> would be advisable now.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:


If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Wietse Venema]

J?rn Odberg:
> Would I need to do this at the sender or the receiver? Or both ends?

You can run tcpdump at one end first. If we can't figure out what
is happening, then we may also need the other end to see if
something is messing around with the packets.

Some "firewalls" have incomplete TCP implementations.

Wietse

> Thanks for the reply, Wietse. And thanks for Postfix.  :-)
> 
> Kind regards from Norway,
> J?rn Odberg
> 
> 
> 
> Wietse Venema skrev:
> > Turn off TCP window scaling.
> > http://www.google.com/search?q=tcp+window+scaling
> >
> > Wietse
> >   
> 
> -- 
> _
> J?rn Odberg, Bibliotek-Systemer AS, N-3271 Larvik, Norway
> LPIC-1 ID#: LPI000164485
> 
> Tlf: +47 33 11 68 00
> Fax: +47 33 11 68 22
> j...@bibsyst.no 

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

Jørn,

> As I said in the first email, I control both ends (both the sender- and
> the receiver-server). But I do not control neither network-connectivity
> or Internet-connectivity at either sites.
>
> I did try turning of Window Scaling at both ends, but it did not help at
> all. It still won't deliver.
>
> I know there is a Cisco PIX at the senders side. They have already
> turned of the ESMTP Fixup (or fuckup if you'd like) "feature" by an
> earlier request from me, and that solved some problems. But it seems
> like the damn PIX has some other "features" which fucks up some more...
> Just don't know what.

It is strange that the sending side transmitted the large data packet
(with mail contents) with a DF (don't fragment) bit turned on,
yet it is captured as two fragments on the transmitting side.

Of these two fragments only the second (smaller) reaches the receiver.
Looks like something is forcefully breaking packets despite a DF,
and I don't find it unusual that a receiving side reluctantly
discards a fragment.

  Mark

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Jørn Odberg]

Hello again.

I can now see that the recieving side has an ESTABLISHED connection from 
the sender, even after the sender tell me it has lost the connection 
with the reciever. So it seems like something in the middle is forcing 
the connection to a close...


I have now captured some more tcpdumping from both sides.

http://postfix.jorno.net/2009_04_23-BamBib-NotBib/


// Jørn


Mark Martinec skrev:

Jørn,

  

As I said in the first email, I control both ends (both the sender- and
the receiver-server). But I do not control neither network-connectivity
or Internet-connectivity at either sites.

I did try turning of Window Scaling at both ends, but it did not help at
all. It still won't deliver.

I know there is a Cisco PIX at the senders side. They have already
turned of the ESMTP Fixup (or fuckup if you'd like) "feature" by an
earlier request from me, and that solved some problems. But it seems
like the damn PIX has some other "features" which fucks up some more...
Just don't know what.



It is strange that the sending side transmitted the large data packet
(with mail contents) with a DF (don't fragment) bit turned on,
yet it is captured as two fragments on the transmitting side.

Of these two fragments only the second (smaller) reaches the receiver.
Looks like something is forcefully breaking packets despite a DF,
and I don't find it unusual that a receiving side reluctantly
discards a fragment.

  Mark
  


--
_
Jørn Odberg, Bibliotek-Systemer AS, N-3271 Larvik, Norway
LPIC-1 ID#: LPI000164485

Tlf: +47 33 11 68 00
Fax: +47 33 11 68 22
j...@bibsyst.no 

Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once

[Mark Martinec]

On Thursday 23 April 2009 10:02:29 Jørn Odberg wrote:
> I can now see that the recieving side has an ESTABLISHED connection from
> the sender, even after the sender tell me it has lost the connection
> with the reciever. So it seems like something in the middle is forcing
> the connection to a close...
>
> I have now captured some more tcpdumping from both sides.
> http://postfix.jorno.net/2009_04_23-BamBib-NotBib/

The root of evil are some misguided firewall configurations which
block ICMP type 3 packets. As a misguided attempt to work around
the first problem, some firewalls or routers intentionally ignore
the DF flag (don't fragment), and fragment a long packet anyway,
instead of sending an ICMP notification and dropping a packet.
And some receiving firewalls drop fragments of a packet which
has a DF flag set.

A workaround is sometimes to force a smaller MTU at your mailer.
Or to turn off MTU discovery (= not to set a DF flag).

A fix is to disable blocking of ICMP type 3 packets in firewalls
(your outgoing, or recipient's incoming), and turn off the second
mentioned misfeature.

  Mark