sis deduplication broken from 2.2.16 upwards
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, sis attachment deduplication is broken in 2.2.16 upwards. It is caused by this commit. https://github.com/dovecot/core/commit/664bf3e236c214aee86294483c379e4fa66c2e63 in src/lib-fs/fs-sis.c function fs_sis_try_link() is comparation of inodes of hash files. Because fs_stat() after that commit use fstat() on open fd of temporary file instead of stat on filename. But that temporary file has differnt inode. It not cause any corruption but it will not save any space. Because every duplicate attachment will be in separate file. - -- [ Ohodnotte kvalitu mailu: https://www.nicereply.com/websupport/4afafd34 ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJW3XkgAAoJEJDF0QA0DLajmJ0QAIZTFTkN0KMJDT3I/rft5Dpp QYM6kyTVdr/FNeR200LLPjXsHXHF8hhoxkQbpfq1D9ceQO0ERDVaL6knKK7/j5Zj EffO+ercXUkcJbE8iYRNcbdpXltUjgYYzgOha7ULiCk0VxROxcujThNTGCDHY8g/ zHXre2aW7hQ8o1yjjhITIOU5WsENSI8PbFktvvWF6OEMVwv8EnaEckJTRGwEl2fu FEPSV+eQg6jHTs/fgiMxAbd4DSzRa2tkeOrw4l4oprtIuthU5hAi9G42Dk+IFvXk k6imnsI6QMPhlrHCm+8Ym+8FwCv7S75JJ/iRp/sS2zHuEwvfwFcVi8pBSjeV50kZ Z0Xglm35tFrHNLAFFb1CVnEw6ftQpuMEE7o/IYrLMVVdPB0C/1IP0s9ucKK0ccQ0 Ld9RULBo6jLgxsVodEntgtMABkfvxybut/tF3O4mm/iN0rDzlOZHMeQ+lsgcdrZd J7h3ow/J0OYosR7CfRFLA875ue2g4GsF+FFP9/oKx/1TkHXJNXpKAhhjOCnSgxHe h44bDrpJyYKlrCaLcOjQM4iUO0upBPpepOnJu50Fde1hNmyr4NaiHLTo23OGX/v3 kEA6xEgmTsXyr3kjhg51/0tdradqKvih2yIptmhpw8P+RWp+ceMoDX2hF95EGB8q FGklrxdxCIoGPAZTOVyf =imfX -END PGP SIGNATURE-
Re: Remove all messages in a mailbox
Hi, we use namespace with prefix = INBOX. When we need to remove all messages in mailbox we need to run this commands: doveadm expunge -u user@domain mailbox INBOX ALL doveadm expunge -u user@domain mailbox INBOX.* ALL The first remove mailboxes in inbox and the second in other subfolders. And in case you use dbox format: doveadm purge -u user@domain On Wed, 25 Jun 2014 17:57:21 +0800 "Murray Trainer" wrote: > Hi All, > > I am having trouble removing all messages in a mailbox. The > command below doesn't remove all the messages: > > doveadm expunge -u user@domain mailbox '*' all > > Do I need to add .* or other? > > THanks > > Murray > -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
Re: [Dovecot] attachment sis + EMLINK (too many links) = segfault bug (2.2.12)
Hi,
attached patch fix problem with segfault when EMLINK happend when file
has too many hard link.
On Fri, 30 May 2014 17:02:47 +0200 Pavel Stano
wrote:
> Hi,
>
> we use attachment dedup with lots of emails (still migrating to it
> from maildir).
> We use netapp storage with wafl filesystem over nfs.
> Problem is that netapp has hard limit of 100k hardlinks to one file.
> And we encountered it.
>
> Problem is that dovecot start do segfault (lmtp,dsync,pop3 etc) when
> it happend when tried to deliver new emails with that attachment.
> Here is strace of dsync:
>
> 6740
> link("/nfsmnt/mailatch1/f9/10/hashes/f9108ddaa156ac15738e41ed3bedec1eda50175d",
> "/nfsmnt/mailatch1/f9/10/f9108ddaa156ac15738e41ed3bedec1eda50175d-7bb7a20ddb598853541a28db4a9f")
> = -1 EMLINK (Too many links) 6740 --- SIGSEGV (Segmentation fault) @
> 0 (0) ---
>
> ls -lh:
> -rw--- 10 vmail vmail 4.7K Apr 28
> 16:54 /nfsmnt/mailatch1/f9/10/hashes/f9108ddaa156ac15738e41ed3bedec1eda50175d
>
> We were using mail_attachment_min_size=4kb, we solve it by increasing
> it to 8kb.
>
> It would be nice to somehow fix this problem. Like not crash when
> EMLINK happend and maybe do not deduplicate attachments but deliver
> email without dedup.
> Or create second file in hashes/ and start hardlinking it instead of
> original.
>
> AFAIK ext4 has also hard-link limit 64k
> (http://en.wikipedia.org/wiki/Hard_link#Limitations_of_hard_links)
> So this can happen to anyone with lots of emails.
>
> Thanks
--
[ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ]
Pavel Stano | Troubleshooter
http://WebSupport.sk
*** BERTE A VYCHUTNAVAJTE ***
diff -ru dovecot-2.2.13/src/lib-fs/fs-sis.c dovecot-2.2.13.fixed/src/lib-fs/fs-sis.c
--- dovecot-2.2.13/src/lib-fs/fs-sis.c 2014-02-04 22:28:38.0 +0100
+++ dovecot-2.2.13.fixed/src/lib-fs/fs-sis.c 2014-06-05 16:05:30.768976354 +0200
@@ -395,7 +395,9 @@
return 1;
}
}
- o_stream_unref(&_file->output);
+if (_file->output != NULL) {
+ o_stream_unref(&_file->output);
+}
if (fs_write_stream_finish(file->super, &file->fs_output) < 0) {
fs_sis_file_copy_error(file);
signature.asc
Description: PGP signature
[Dovecot] attachment sis + EMLINK (too many links) = segfault bug (2.2.12)
Hi,
we use attachment dedup with lots of emails (still migrating to it
from maildir).
We use netapp storage with wafl filesystem over nfs.
Problem is that netapp has hard limit of 100k hardlinks to one file.
And we encountered it.
Problem is that dovecot start do segfault (lmtp,dsync,pop3 etc) when it
happend when tried to deliver new emails with that attachment.
Here is strace of dsync:
6740
link("/nfsmnt/mailatch1/f9/10/hashes/f9108ddaa156ac15738e41ed3bedec1eda50175d",
"/nfsmnt/mailatch1/f9/10/f9108ddaa156ac15738e41ed3bedec1eda50175d-7bb7a20ddb598853541a28db4a9f")
= -1 EMLINK (Too many links) 6740 --- SIGSEGV (Segmentation fault) @ 0
(0) ---
ls -lh:
-rw--- 10 vmail vmail 4.7K Apr 28
16:54 /nfsmnt/mailatch1/f9/10/hashes/f9108ddaa156ac15738e41ed3bedec1eda50175d
We were using mail_attachment_min_size=4kb, we solve it by increasing it
to 8kb.
It would be nice to somehow fix this problem. Like not crash when
EMLINK happend and maybe do not deduplicate attachments but deliver
email without dedup.
Or create second file in hashes/ and start hardlinking it instead of
original.
AFAIK ext4 has also hard-link limit 64k
(http://en.wikipedia.org/wiki/Hard_link#Limitations_of_hard_links)
So this can happen to anyone with lots of emails.
Thanks
--
[ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ]
Pavel Stano | Troubleshooter
http://WebSupport.sk
*** BERTE A VYCHUTNAVAJTE ***
signature.asc
Description: PGP signature
[Dovecot] one way replication
Hi,
anyone using one way (or master/slave) replication instead of
master/master ?
There is no info in documentation but in sources
(src/doveadm/dsync/doveadm-dsync.c) i found some flags which looks like
for setting up master/slave
switch (c) {
case '1':
ctx->oneway = TRUE;
ctx->backup = TRUE;
break;
case 'R':
ctx->reverse_backup = TRUE;
break;
So on master i add -1 :
replication_dsync_parameters = -1 -d -n INBOX -l 30 -U
and on slave -1 -R:
replication_dsync_parameters = -1 -R -d -n INBOX -l 30 -U
It looks it works, if i but question is if it is production ready.
--
[ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ]
Pavel Stano | Troubleshooter
http://WebSupport.sk
*** BERTE A VYCHUTNAVAJTE ***
signature.asc
Description: PGP signature
Re: [Dovecot] replication + attachment sis + zlib bug ? (HEAD version from xi.rename-it.nl)
Yes, exactly. I look at the code and find out there is functionality for that. Dovecot store attachments in base64 format without decoding if there are longer lines with more base64 blocks (4chars). So these attachments get deduplicated but are not decoded. But it don't work if last line is buggy. I think i fixed it. It looks like it works, but i test only one mailbox. Patch attached On Fri, 11 Apr 2014 09:29:53 +0200 Gilles Chauvin wrote: > On 10/04/2014 18:21, Pavel Stano wrote: > > Hi, > > > > yes its the same problem. > > I can confirm that it is caused by last line in base64 > > attachment which is longer than 72 chars in original message. > > > > > > Hi, > > I don't know what is the current status for this issue since Timo > never responded to my two mails. A safe way to workaround this > problem could be Dovecot skipping deduplication when it encounters a > malformed base64 attachment ? > > > > Regards, > Gilles. -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** attachment_extractor_base64_last_line_fix Description: Binary data signature.asc Description: PGP signature
Re: [Dovecot] replication + attachment sis + zlib bug ? (HEAD version from xi.rename-it.nl)
Hi, yes its the same problem. I can confirm that it is caused by last line in base64 attachment which is longer than 72 chars in original message. On Thu, 10 Apr 2014 14:26:25 +0200 Gilles Chauvin wrote: > On 10/04/2014 13:38, Pavel Stano wrote: > > Hi, > > > > i have setup with mail_attachment single instance store + > > replication + zlib and got this bug when i try to replicate one > > test mailbox: > > > > On master1 in mail.log: > > Apr 10 13:25:22 master1 dovecot: > > dsync-local(z...@blabla666.sk): Error: > > read(/nfsmnt/mailnfs1/attachments1/6b/57/6b57ad34cf6c414662233d833a7801fde4e1cdcb-92b5052558774653a72813e2b982[base64:18 > > b/l]) failed: Stream is larger than expected (97824 > 97823, eof=1) > > Apr 10 13:25:22 master1 dovecot: dsync-local(z...@blabla666.sk): > > Error: dsync(master2): > > read(attachments-connector(zlib(/nfsmnt/mailnfs1/b/l/blabla666.sk/z...@blabla666.sk/mdbox/storage/m.9))) > > failed: > > read(/nfsmnt/mailnfs1/attachments1/6b/57/6b57ad34cf6c414662233d833a7801fde4e1cdcb-92b5052558774653a72813e2b982[base64:18 > > b/l]) failed: Stream is larger than expected (97824 > 97823, eof=1) > > > > > > This is on master2 in mail.log > > Apr 10 13:32:21 master2 dovecot: dsync-server(z...@blabla666.sk): > > Error: dsync(master1): read() failed: read(10.10.30.2) failed: > > dot-input stream ends without '.' line > > > > Hi, > Your problem looks quite similar to the one I reported 2 months ago. > → http://markmail.org/message/tt4jpjnpsa6lmlz2 > > > Regards, > Gilles -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
Re: [Dovecot] dovecot: disable ssl compression
Sorry, i replied to wrong thread On Thu, 10 Apr 2014 18:08:05 +0200 Pavel Stano wrote: > Hi, > > yes its the same problem. > I can confirm that it is caused by last line in base64 > attachment which is longer than 72 chars in original message. > > On Thu, 10 Apr 2014 16:41:38 +0200 Reindl Harald > wrote: > > > > > > > Am 10.04.2014 15:04, schrieb Andreas Schulze: > > > Our "it-security" department asked me about Qualys warnings like > > >-> SSL/TLS Compression Algorithm Information Leakage > > > Vulnerability > > > > > > As far as I learned it's compression inside ssl. > > > postfix-2.11 knows 'tls_ssl_options = no_compression' > > > ( see http://www.postfix.org/postconf.5.html#tls_ssl_options ) > > > > > > is the something comparable in dovecot too? > > > > > > Looks like most extensions in ssl exist only to be disabled :-/ > > > > that attacks are not relevant for email because they > > rely on the way a webbrowser works which is not the > > case for a mail client - you can't trigger XSS and > > Ajax in a MUA > > > > https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls > > > > >> This year, it's CRIME, a practical attack against how TLS is > > >> used in browsers. In a wider sense, the same attack conceptually > > >> applies to any encrypted protocol where the attacker controls > > >> what is being communicated > > > > > -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
Re: [Dovecot] dovecot: disable ssl compression
Hi, yes its the same problem. I can confirm that it is caused by last line in base64 attachment which is longer than 72 chars in original message. On Thu, 10 Apr 2014 16:41:38 +0200 Reindl Harald wrote: > > > Am 10.04.2014 15:04, schrieb Andreas Schulze: > > Our "it-security" department asked me about Qualys warnings like > >-> SSL/TLS Compression Algorithm Information Leakage > > Vulnerability > > > > As far as I learned it's compression inside ssl. > > postfix-2.11 knows 'tls_ssl_options = no_compression' > > ( see http://www.postfix.org/postconf.5.html#tls_ssl_options ) > > > > is the something comparable in dovecot too? > > > > Looks like most extensions in ssl exist only to be disabled :-/ > > that attacks are not relevant for email because they > rely on the way a webbrowser works which is not the > case for a mail client - you can't trigger XSS and > Ajax in a MUA > > https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls > > >> This year, it's CRIME, a practical attack against how TLS is > >> used in browsers. In a wider sense, the same attack conceptually > >> applies to any encrypted protocol where the attacker controls > >> what is being communicated > -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
[Dovecot] replication + attachment sis + zlib bug ? (HEAD version from xi.rename-it.nl)
Hi, i have setup with mail_attachment single instance store + replication + zlib and got this bug when i try to replicate one test mailbox: On master1 in mail.log: Apr 10 13:25:22 master1 dovecot: dsync-local(z...@blabla666.sk): Error: read(/nfsmnt/mailnfs1/attachments1/6b/57/6b57ad34cf6c414662233d833a7801fde4e1cdcb-92b5052558774653a72813e2b982[base64:18 b/l]) failed: Stream is larger than expected (97824 > 97823, eof=1) Apr 10 13:25:22 master1 dovecot: dsync-local(z...@blabla666.sk): Error: dsync(master2): read(attachments-connector(zlib(/nfsmnt/mailnfs1/b/l/blabla666.sk/z...@blabla666.sk/mdbox/storage/m.9))) failed: read(/nfsmnt/mailnfs1/attachments1/6b/57/6b57ad34cf6c414662233d833a7801fde4e1cdcb-92b5052558774653a72813e2b982[base64:18 b/l]) failed: Stream is larger than expected (97824 > 97823, eof=1) This is on master2 in mail.log Apr 10 13:32:21 master2 dovecot: dsync-server(z...@blabla666.sk): Error: dsync(master1): read() failed: read(10.10.30.2) failed: dot-input stream ends without '.' line I tried to rm -Rf mailbox on master2 and run on master1 doveadm replicator remove z...@blabla666.sk doveadm replicator replicate z...@blabla666.sk but same problem happen. Thanks -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
Re: [Dovecot] mdbox with mail_attachment_dir per user in 2.2.12
Hello Alex, overriding it from user database works perfectly. Thanks On Fri, 4 Apr 2014 11:22:37 +0200 Axel Luttgens wrote: > Le 3 avr. 2014 à 15:12, Pavel Stano a écrit : > > > Hello, > > > > we have 2 netapp arrays with 2 volumes and we need to store our mail > > data in one or other volume for every user. > > We want keep data for one user in one volume only. > > > > So we would like to have 2 separate mail_attachment_dir, one on > > every volume. > > But in latest version 2.2.12 it look like it is only possible to set > > one global mail_attachment_dir. > > > > I have an idea with 2 separate dovecot instances per machine with > > different mail_attachment_dir. Or it is possible to somehow specify > > it per user ? > > Hello Pavel, > > I don't know whether that setting may be customized or if must have a > single constant value. > > Anyway, have you tried to make use of variables for setting its value? > Or, if more relevant in your case, by overriding it from your user > database (see section "Overriding settings" at > http://wiki2.dovecot.org/UserDatabase/ExtraFields). > > Axel -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
[Dovecot] mdbox with mail_attachment_dir per user in 2.2.12
Hello, we have 2 netapp arrays with 2 volumes and we need to store our mail data in one or other volume for every user. We want keep data for one user in one volume only. So we would like to have 2 separate mail_attachment_dir, one on every volume. But in latest version 2.2.12 it look like it is only possible to set one global mail_attachment_dir. I have an idea with 2 separate dovecot instances per machine with different mail_attachment_dir. Or it is possible to somehow specify it per user ? Thanks -- [ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/Stano/ ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** signature.asc Description: PGP signature
[Dovecot] [PATCH] Pop3/imap ordering in courier-dovecot-migrate.pl
Hi,
current version of courier migration script courier-dovecot-migrate.pl
preserve order of POP3 UIDLs, but it keep IMAP UIDs not in order which
cause problems with clients using IMAP along with POP3 in some cases.
Attached patch use MAIL_FETCH_POP3_ORDER feature added in version 2.0.13
to keep POP3 and IMAP UID[L]s in order.
--
[ Ohodnotte kvalitu mailu: http://nicereply.com/websupport/4afafd34/ ]
Pavel Stano | admin team
http://WebSupport.sk
*** TAKE AND ENJOY ***
--- courier-dovecot-migrate.pl.orig 2012-03-04 15:08:08.0 +0100
+++ courier-dovecot-migrate.pl 2012-07-18 16:34:55.0 +0200
@@ -311,12 +311,13 @@
}
# POP3 clients may want to get POP3 UIDLs in the same order always.
-# Preserve the order even if it causes IMAP UIDs to change.
+# Preserve the order with O flag supported in dovecot 2.0.13+ and dont change order of IMAP UIDs.
my %order_map;
foreach my $fname ( keys %{$filename_map} ) {
my $order = $filename_map->{$fname}->[4];
-$order_map{$order} = $fname if ($order != 0);
+ $filename_map->{$fname}->[5]=$order;
}
+
my $prev_uid = 0;
foreach my $order ( sort { $a <=> $b } keys %order_map ) {
my $file_ar = $filename_map->{ $order_map{$order} };
@@ -332,20 +333,14 @@
}
}
-# Sort POP3 messages by their Courier-POP3 order.
-my @pop3 = sort {
-$filename_map->{$a}[4] <=> $filename_map->{$b}[4]
-} grep { $filename_map->{$_}[4] } keys %$filename_map;
-
-# Sort non-POP3 messages by their UID.
-# Those that have UID==0 come last since they have been marked to get a new UID.
-my @imaponly = sort {
-($filename_map->{$a}[0] || 'inf') <=> ($filename_map->{$b}[0] || 'inf')
-} grep { ! $filename_map->{$_}[4] } keys %$filename_map;
+# sort all messages by their imap UID
+my @all = sort {
+ ($filename_map->{$a}[0] || 'inf') <=> ($filename_map->{$b}[0] || 'inf')
+} keys %$filename_map;
$prev_uid = 0;
my %uidlist_map;
-foreach my $fname ( @pop3, @imaponly ) {
+foreach my $fname ( @all ) {
my $file_ar = $filename_map->{ $fname };
my $uid = $file_ar->[0];
if ($uid == 0 # only in pop3 list or marked for new UID
@@ -365,6 +360,9 @@
foreach my $uid ( sort { $a <=> $b } keys %uidlist_map ) {
my $file_ar = $filename_map->{ $uidlist_map{$uid} };
print $dovecot_uidlist_fh "$uid ";
+ if ($file_ar->[5]) {
+ print $dovecot_uidlist_fh "O".$file_ar->[5]." ";
+ }
print $dovecot_uidlist_fh 'P' . $file_ar->[1] . ' ' if ( $file_ar->[1] ne "" );
print $dovecot_uidlist_fh 'W' . $file_ar->[2] . ' ' if ( $file_ar->[2] > 0 );
print $dovecot_uidlist_fh ':' . $file_ar->[3] . "\n";
