Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[Mike Constabel via dovecot]

Hello,

telsch via dovecot schrieb am Tuesday, den 10. September 2019:

> >Fix here:
> >https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d
> >
> 
> Could this also apply to this Timo? We also using attachment_dir with
> plugin zlib
> 
> https://dovecot.org/list/dovecot/2019-September/117007.html

That would be nice...

Error: Corrupted record in index cache file 
/mailspool1/public-folders/mdbox/mailboxes/WW/dbox-Mails/dovecot.index.cache: 
UID 685: Broken virtual size in mailbox public/WW: 
read(attachments-connector(zlib(/mailspool1/public-folders/mdbox/storage/m.2137))):
 FETCH BODY[] got too little data: 701113 vs 701327


Regards,
Constabel

Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[telsch via dovecot]

Actually the mail isn't saved corrupted. The bug is when reading the
mail. So any existing corrupted mails become fixed after upgrading.

Fix here:
https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d



Could this also apply to this Timo? We also using attachment_dir with
plugin zlib

https://dovecot.org/list/dovecot/2019-September/117007.html

Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[Timo Sirainen via dovecot]

On 19 Jul 2019, at 17.52, Patrick Cernko via dovecot  
wrote:
> 
> Hello list, hello Dovecot developers,
> 
> this week, I discovered a serious bug in Dovecot, that lead to several broken 
> mails on our servers. The bug corrupts the first few characters of the mail 
> header during saving. On our setup, it was almost always only the very first 
> line of text, that was corrupted.
> 
> The bug seems to be triggered by a bad "interaction" of attachment_dir option 
> and zlib plugin. If you use both, you most likely are affected, too, except 
> you only use zlib plugin for reading previously compressed stored mails. 
> That's also the workaround we use now: zlib plugin only enabled in 
> mail_plugins but no plugin/zlib_save set.

Actually the mail isn't saved corrupted. The bug is when reading the mail. So 
any existing corrupted mails become fixed after upgrading.

Fix here: 
https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d 

Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[Timo Sirainen via dovecot]

On 19 Jul 2019, at 17.52, Patrick Cernko via dovecot  
wrote:
> 
> Hello list, hello Dovecot developers,
> 
> this week, I discovered a serious bug in Dovecot, that lead to several broken 
> mails on our servers. The bug corrupts the first few characters of the mail 
> header during saving. On our setup, it was almost always only the very first 
> line of text, that was corrupted.
..
> The bug occurs on very specific mails. Due to privacy reasons I could not 
> provide sample mails here. Storing such mails seems to trigger the bug 
> reproducible.
> 
> 
> I attached a very minimal doveconf -n config, that can be used to trigger the 
> bug. If one of the developers is interested, I can try to generate an 
> "anonymized" version of such a specific mail that still causes the issue. I 
> discovered the bug on our productive systems, running latest Dovecot 2.2 
> release, but the latest 2.3 I used during debugging is affected, too.

Getting such a mail that would allow reproducing would be helpful. I can't seem 
to be able to reproduce this with stress testing.

https://dovecot.org/tools/  has a couple of scripts 
that can obfuscate emails in a bit different ways. For example 
https://dovecot.org/tools/maildir-obfuscate.pl 
 might work.

I'm also wondering if Stephan's recent base64 code changes will fix this 
(everything is not merged yet).

WARNING: using attachment_dir with plugin zlib can corrupt mails

[Patrick Cernko via dovecot]

Hello list, hello Dovecot developers,

this week, I discovered a serious bug in Dovecot, that lead to several 
broken mails on our servers. The bug corrupts the first few characters 
of the mail header during saving. On our setup, it was almost always 
only the very first line of text, that was corrupted.


Depending on the IMAP client (they seem to request different header 
fields, ... during mail access), the bug causes the imap process to hang 
up the TCP connection and log errors like this:



imap(USERNAME)<4767>: Error: Corrupted record in index cache 
file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: 
UID 489113: Broken fields in mailbox Trash: 
read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH 
BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122


In our case that finally grabbed my attention, the client was the users 
iphone that did not display any new messages but his Thunderbird did.


The bug seems to be triggered by a bad "interaction" of attachment_dir 
option and zlib plugin. If you use both, you most likely are affected, 
too, except you only use zlib plugin for reading previously compressed 
stored mails. That's also the workaround we use now: zlib plugin only 
enabled in mail_plugins but no plugin/zlib_save set.


The bug occurs on very specific mails. Due to privacy reasons I could 
not provide sample mails here. Storing such mails seems to trigger the 
bug reproducible.



I attached a very minimal doveconf -n config, that can be used to 
trigger the bug. If one of the developers is interested, I can try to 
generate an "anonymized" version of such a specific mail that still 
causes the issue. I discovered the bug on our productive systems, 
running latest Dovecot 2.2 release, but the latest 2.3 I used during 
debugging is affected, too.


During debugging, I also found one hint, that might help find the bug: 
If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and 
then disable the zlib plugin in mail_plugins, you can call


doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress

on test's mailbox with only that one broken mail.
This will display the beginning of the rfc822 mail text until gzip 
terminates with "gzip: stdin: unexpected end of file", approximately 
after twice the length of the mail HEADER. This might indicate, that 
dovecot stores the uncompressed size of the header in it's data 
structures although the mail is stored compressed.



I also found a very efficient way to find all affected mails in our setup:

doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \
  grep -a "^[^ ]+ user=" | \
  grep -avF ' hdr=Return-path: ' | \
  grep -av '.* hdr=[[:print:][:space:]]*$'
(runtime for ~6M mails on our servers was 20-30min)

This can be even more optimized if you have a powerful storage system 
with GNU parallel:

doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid 
seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* 
hdr=[[:print:][:space:]]*$' || true"

(runtime for ~6M mails on our servers was ~4min)

The command will give you a list of mails that possibly are affected, 
check the full output of


doveadm fetch -u USERNAME hdr guid GUID | less

to verify that the header is really broken.

On our systems I found 39 mails within ~12M mails.

I was able to recover these mails "manually" by reconstructing the 
Return-Path header line, importing the fixed mails and expunging the 
corrupt ones. Before importing, I had to disable zlib_save option obviously.


Best regards,
--
Patrick Cernko  +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
# 2.3.6.1 (d124cc84b): /etc/dovecot/dovecot.conf
# OS: Linux 4.14.127.1.amd64-smp x86_64 Debian 9.9 
# Hostname: adove.mpi-klsb.mpg.de
listen = *
mail_attachment_dir = /var/vmail/attachments
mail_attachment_fs = posix
mail_gid = nogroup
mail_home = /var/vmail/%u
mail_location = mdbox:~/mdbox
mail_plugins = " zlib"
mail_uid = nobody
passdb {
  args = /etc/dovecot/userdb
  driver = passwd-file
}
plugin {
  zlib_save = gz
}
protocols = imap
userdb {
  args = /etc/dovecot/userdb
  driver = passwd-file
}


smime.p7s
Description: S/MIME Cryptographic Signature