Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Matt Corallo]

On 5/11/22 8:09 AM, Gedalya wrote:

I'm a little dazzled by the variety of crashes I've seen so far: smtp_setup_conn > 
tls_client_start > verify_certificate, and during ARC signing, but it could be 
just noise so I'll leave it alone for now.



As a passer-by might I suggest valgrind or building with address/undefined-behavior sanitizer? I 
don't see any mention of it in this issue, and "it keeps crashing in random places that may or may 
not be related" screams "memory corruption".


Matt

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

Package: exim4
Version: 4.95-3
Severity: serious

When sending a mail:

[...]
2022-02-01 14:23:22 1nEt2b-0008jG-97 H=mx2.mailchannels.net [34.214.163.59]: 
SMTP error from remote mail server after DATA: 450 [S09] Try Again Later
2022-02-01 14:23:22 1nEt2b-0008jG-97 H=mx2.mailchannels.net [34.214.163.59] TLS 
error on connection (recv): The TLS connection was non-properly terminated.
2022-02-01 14:23:23 1nEt2b-0008jG-97 SIGSEGV (maybe attempt to write to 
immutable memory)
2022-02-01 14:23:23 1nEt2b-0008jG-97 Delivery status for xxx@yyy: got 0 of 7 
bytes (pipeheader) from transport process 35015 for transport smtp

2022-02-01 14:23:23 1nEt2b-0008jG-97 == xxx@yyy R=dnslookup T=remote_smtp defer 
(-1): smtp transport process returned non-zero status 0x000b: terminated by 
signal 11
2022-02-01 14:23:23 1nEt7z-00096o-IX <= <> R=1nEt2b-0008jG-97 U=Debian-exim 
P=local S=783
2022-02-01 14:23:23 1nEt2b-0008jG-97 Frozen
2022-02-01 14:23:23 1nEt7z-00096o-IX => vlefevre 
 R=local_user T=maildir_home
2022-02-01 14:23:23 1nEt7z-00096o-IX Completed
2022-02-01 14:23:23 End queue run: pid=35012
2022-02-01 14:28:19 Start queue run: pid=35460
2022-02-01 14:28:19 1nEt2b-0008jG-97 Message is frozen
2022-02-01 14:28:19 End queue run: pid=35460
[...]

The consequence is that the mail is not sent, there are no retries,
and the end user is not warned. So, from the end user point of view,
the message is *silently lost*.

-- Package-specific info:
Exim version 4.95 #2 built 16-Dec-2021 18:26:32
Copyright (c) University of Cambridge, 1995 - 2018
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2020
Berkeley DB: Berkeley DB 5.3.28: (September  9, 2013)
Support for: crypteq iconv() IPv6 GnuTLS TLS_resume move_frozen_messages DANE 
DKIM DNSSEC Event I18N OCSP PIPE_CONNECT PRDR Experimental_Queue_Ramp SOCKS SRS 
TCP_Fast_Open
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz 
dbmnz dnsdb dsearch nis nis0 passwd
Authenticators: cram_md5 external plaintext
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Configuration file search path is 
/etc/exim4/exim4.conf:/var/lib/exim4/config.autogenerated
Configuration file is /var/lib/exim4/config.autogenerated
# /etc/exim4/update-exim4.conf.conf
#
# Edit this file and /etc/mailname by hand and execute update-exim4.conf
# yourself or use 'dpkg-reconfigure exim4-config'
#
# Please note that this is _not_ a dpkg-conffile and that automatic changes
# to this file might happen. The code handling this will honor your local
# changes, so this is usually fine, but will break local schemes that mess
# around with multiple versions of the file.
#
# update-exim4.conf uses this file to determine variable values to generate
# exim configuration macros for the configuration file.
#
# Most settings found in here do have corresponding questions in the
# Debconf configuration, but not all of them.
#
# This is a Debian specific file

dc_eximconfig_configtype='internet'
dc_other_hostnames='cventin.lip.ens-lyon.fr'
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost=''
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname=''
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'
mailname:cventin.lip.ens-lyon.fr
# /etc/default/exim4
EX4DEF_VERSION=''

# 'combined' -   one daemon running queue and listening on SMTP port
# 'no'   -   no daemon running the queue
# 'separate' -   two separate daemons
# 'ppp'  -   only run queue with /etc/ppp/ip-up.d/exim4.
# 'nodaemon' - no daemon is started at all.
# 'queueonly' - only a queue running daemon is started, no SMTP listener.
# setting this to 'no' will also disable queueruns from /etc/ppp/ip-up.d/exim4
QUEUERUNNER='combined'
# how often should we run the queue
QUEUEINTERVAL='5m'
# options common to quez-runner and listening daemon
COMMONOPTIONS=''
# more options for the daemon/process running the queue (applies to the one
# started in /etc/ppp/ip-up.d/exim4, too.
QUEUERUNNEROPTIONS=''
# special flags given to exim directly after the -q. See exim(8)
QFLAGS=''
# Options for the SMTP listener daemon. By default, it is listening on
# port 25 only. To listen on more ports, it is recommended to use
# -oX 25:587:10025 -oP /var/run/exim4/exim.pid
SMTPLISTENEROPTIONS=''

-- System Information:
Debian Release: bookworm/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 
'stable-security'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 
'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.15.0-3-amd64 (SMP w/12 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, 
TAINT_UNSIGNED_MODULE
Locale: LANG=POSIX, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked t

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-01 14:44:21 +0100, Vincent Lefevre wrote:
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 SIGSEGV (maybe attempt to write to 
> immutable memory)
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 Delivery status for xxx@yyy: got 0 of 7 
> bytes (pipeheader) from transport process 35015 for transport smtp
> 
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 == xxx@yyy R=dnslookup T=remote_smtp 
> defer (-1): smtp transport process returned non-zero status 0x000b: 
> terminated by signal 11
> 2022-02-01 14:23:23 1nEt7z-00096o-IX <= <> R=1nEt2b-0008jG-97 U=Debian-exim 
> P=local S=783
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 Frozen
> 2022-02-01 14:23:23 1nEt7z-00096o-IX => vlefevre 
>  R=local_user T=maildir_home
> 2022-02-01 14:23:23 1nEt7z-00096o-IX Completed
> 2022-02-01 14:23:23 End queue run: pid=35012
> 2022-02-01 14:28:19 Start queue run: pid=35460
> 2022-02-01 14:28:19 1nEt2b-0008jG-97 Message is frozen
> 2022-02-01 14:28:19 End queue run: pid=35460
> [...]
> 
> The consequence is that the mail is not sent, there are no retries,
> and the end user is not warned. So, from the end user point of view,
> the message is *silently lost*.

Some clarification: by "end user", I mean the author of the message.
There is an error message sent to postmaster, but not to the address
of the author of the message.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-01 14:44:21 +0100, Vincent Lefevre wrote:
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 SIGSEGV (maybe attempt to write to 
> immutable memory)
> 2022-02-01 14:23:23 1nEt2b-0008jG-97 Delivery status for xxx@yyy: got 0 of 7 
> bytes (pipeheader) from transport process 35015 for transport smtp

And at midnight, r...@cventin.lip.ens-lyon.fr got a mail with

  exim paniclog /var/log/exim4/paniclog on cventin.lip.ens-lyon.fr
  has non-zero size, mail system might be broken. Up to 10 lines
  are quoted below.

and the above two lines.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Andreas Metzler]

On 2022-02-01 Vincent Lefevre  wrote:
> On 2022-02-01 14:44:21 +0100, Vincent Lefevre wrote:
> > 2022-02-01 14:23:23 1nEt2b-0008jG-97 SIGSEGV (maybe attempt to write to 
> > immutable memory)
> > 2022-02-01 14:23:23 1nEt2b-0008jG-97 Delivery status for xxx@yyy: got 0 of 
> > 7 bytes (pipeheader) from transport process 35015 for transport smtp
> > 
> > 2022-02-01 14:23:23 1nEt2b-0008jG-97 == xxx@yyy R=dnslookup T=remote_smtp 
> > defer (-1): smtp transport process returned non-zero status 0x000b: 
> > terminated by signal 11
> > 2022-02-01 14:23:23 1nEt7z-00096o-IX <= <> R=1nEt2b-0008jG-97 U=Debian-exim 
> > P=local S=783
> > 2022-02-01 14:23:23 1nEt2b-0008jG-97 Frozen
> > 2022-02-01 14:23:23 1nEt7z-00096o-IX => vlefevre 
> >  R=local_user T=maildir_home
> > 2022-02-01 14:23:23 1nEt7z-00096o-IX Completed
> > 2022-02-01 14:23:23 End queue run: pid=35012
> > 2022-02-01 14:28:19 Start queue run: pid=35460
> > 2022-02-01 14:28:19 1nEt2b-0008jG-97 Message is frozen
> > 2022-02-01 14:28:19 End queue run: pid=35460
> > [...]
> > 
> > The consequence is that the mail is not sent, there are no retries,
> > and the end user is not warned. So, from the end user point of view,
> > the message is *silently lost*.

> Some clarification: by "end user", I mean the author of the message.
> There is an error message sent to postmaster, but not to the address
> of the author of the message.

Yeah from exim's POV this workes out as designed. Something very much
unexpected happened, let's freeze the message and tell the admin. The
message is not lost. It is still in queue.

Is this reproducible, happening with a specific host? Any chance of
getting a coredump?

cu Andreas

-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-02 18:20:48 +0100, Andreas Metzler wrote:
> Yeah from exim's POV this workes out as designed. Something very much
> unexpected happened, let's freeze the message and tell the admin. The
> message is not lost. It is still in queue.

Yes, once the issue occurs, exim works as designed. But this makes
the issue particularly problematic. Note that on a multi-user machine,
the user typically doesn't have access to the status of the queue,
so doesn't know that the mail has not been sent, and will probably
never know the reason.

> Is this reproducible, happening with a specific host?

Not reproduced yet. I wonder whether this is due to the "SMTP error
from remote mail server after DATA: 450 [S09] Try Again Later".

> Any chance of getting a coredump?

exim didn't leave a coredump. I could see that Mutt disables coredumps
after some time. I don't know under which condition.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-03 03:29:24 +0100, Vincent Lefevre wrote:
> On 2022-02-02 18:20:48 +0100, Andreas Metzler wrote:
> > Any chance of getting a coredump?
> 
> exim didn't leave a coredump. I could see that Mutt disables coredumps
> after some time. I don't know under which condition.

I've eventually found that coredumps are disabled when I start Mutt
from my window manager (FVWM), but not from a terminal (actually,
that's the same wrapper in both cases). The reason seems that
disabled coredumps is the default (according to /proc/.../limits),
but I re-enable them in my shell.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Marc Haber]

On Thu, Feb 03, 2022 at 03:46:22AM +0100, Vincent Lefevre wrote:
> On 2022-02-03 03:29:24 +0100, Vincent Lefevre wrote:
> > On 2022-02-02 18:20:48 +0100, Andreas Metzler wrote:
> > > Any chance of getting a coredump?
> > 
> > exim didn't leave a coredump. I could see that Mutt disables coredumps
> > after some time. I don't know under which condition.
> 
> I've eventually found that coredumps are disabled when I start Mutt
> from my window manager (FVWM), but not from a terminal (actually,
> that's the same wrapper in both cases). The reason seems that
> disabled coredumps is the default (according to /proc/.../limits),
> but I re-enable them in my shell.

Is the bug repeatable by unfreezing the message and forcing a delivery
attempt? If so, you might be able to get a core dump from the delivering
exim.

If the 450 is casued by greylisting on the remote side, that debugging
attempt must be close to the first time of sending the message so that
the 450 is still there.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-03 14:21:58 +0100, Marc Haber wrote:
> Is the bug repeatable by unfreezing the message and forcing a delivery
> attempt? If so, you might be able to get a core dump from the delivering
> exim.

The delivery was successful. There was still a "TLS error on
connection (recv): The TLS connection was non-properly terminated."
error, but no 450:

2022-02-02 13:42:28 Start queue run: pid=150965 -qff
2022-02-02 13:42:28 1nEt2b-0008jG-97 Unfrozen by forced delivery
2022-02-02 13:42:31 1nEt2b-0008jG-97 H=mx1.mailchannels.net [54.189.39.249] TLS 
error on connection (recv): The TLS connection was non-properly terminated.
2022-02-02 13:42:31 1nEt2b-0008jG-97 => xxx@yyy R=dnslookup T=remote_smtp 
H=mx1.mailchannels.net [54.189.39.249] 
X=TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes 
DN="CN=*.mailchannels.net" C="250 2.0.0 Ok: queued as 6D9D5201EF"
2022-02-02 13:42:31 1nEt2b-0008jG-97 Completed
2022-02-02 13:42:31 End queue run: pid=150965 -qff

> If the 450 is casued by greylisting on the remote side, that debugging
> attempt must be close to the first time of sending the message so that
> the 450 is still there.

Unfortunately, I can't control that.

Perhaps the bug is due to the combination of both a "TLS error on
connection (recv): The TLS connection was non-properly terminated."
error and a 450, e.g. in case exim does something valid only if
there were no errors. A potential security issue?

If I understand correctly, the error comes from

  record_io_error(state, (int)inbytes, US"recv", NULL);

in src/tls-gnu.c, tls_read().

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Marc Haber]

On Thu, Feb 03, 2022 at 02:51:10PM +0100, Vincent Lefevre wrote:
> On 2022-02-03 14:21:58 +0100, Marc Haber wrote:
> > Is the bug repeatable by unfreezing the message and forcing a delivery
> > attempt? If so, you might be able to get a core dump from the delivering
> > exim.
> 
> The delivery was successful. There was still a "TLS error on
> connection (recv): The TLS connection was non-properly terminated."
> error, but no 450:
> 
> 2022-02-02 13:42:28 Start queue run: pid=150965 -qff
> 2022-02-02 13:42:28 1nEt2b-0008jG-97 Unfrozen by forced delivery
> 2022-02-02 13:42:31 1nEt2b-0008jG-97 H=mx1.mailchannels.net [54.189.39.249] 
> TLS error on connection (recv): The TLS connection was non-properly 
> terminated.

Just for the protocol: gnutls-cli --insecure -s -p 25 54.189.39.249
allows me to simulate an EHLO/STARTTLS/EHLO/QUIT session that gets
properly terminated. Andreas as GnuTLS maintainer might make more sense
from that finding.

> 2022-02-02 13:42:31 1nEt2b-0008jG-97 => xxx@yyy R=dnslookup T=remote_smtp 
> H=mx1.mailchannels.net [54.189.39.249] 
> X=TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes 
> DN="CN=*.mailchannels.net" C="250 2.0.0 Ok: queued as 6D9D5201EF"
> 2022-02-02 13:42:31 1nEt2b-0008jG-97 Completed
> 2022-02-02 13:42:31 End queue run: pid=150965 -qff

That doesn't help though, we neeed the error to happen ;-)

Do you see this TLS error with other remotes?

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Vincent Lefevre]

On 2022-02-03 17:30:28 +0100, Marc Haber wrote:
> Do you see this TLS error with other remotes?

No, but the machine only has 11 days of logs, while I don't use it
much in these times. So that's almost nothing.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Processed: Re: Bug#1004740: exim4: SIGSEGV (maybe attempt to write to immutable memory) when sending a mail; message frozen

[Debian Bug Tracking System]

Processing commands for cont...@bugs.debian.org:

> forwarded 1004740 https://bugs.exim.org/show_bug.cgi?id=2886
Bug #1004740 [exim4] exim4: SIGSEGV (maybe attempt to write to immutable 
memory) when sending a mail; message frozen
Set Bug forwarded-to-address to 'https://bugs.exim.org/show_bug.cgi?id=2886'.
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
1004740: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004740
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems