Re: [Dovecot] zlib plugin question
On 2013-12-19 16:07, Przemysław Orzechowski wrote: Is it possible to compress incoming mails delivered via dovecots LDA when using dovecot --version 1.2.9 or do i have to compress them via cron? There is a zlib plugin for Dovecot 1.x you could use. See http://wiki1.dovecot.org/Plugins/Zlib -- Frerich Raabe - ra...@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Re: [Dovecot] zlib plugin question
On 2013-12-19 17:51, Frerich Raabe wrote: On 2013-12-19 16:07, Przemysław Orzechowski wrote: Is it possible to compress incoming mails delivered via dovecots LDA when using dovecot --version 1.2.9 or do i have to compress them via cron? There is a zlib plugin for Dovecot 1.x you could use. See http://wiki1.dovecot.org/Plugins/Zlib Sorry, I only now noticed that this plugin only allows reading compressed mail files - apparently you would need Dovecot v2.x if you want to compress incoming mail. -- Frerich Raabe - ra...@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Re: [Dovecot] Zlib plugin - when does it make sense?
I have a relatively big installation here in the university, 1 mail server for 10.000 daily active accounts. I converted from maildir to mdbox + zlib in march, and these are my considerations: 1. CPU is not a problem, even delivering more than 400 messages per minute (blade HS21 with 2xQuad E5430 from 2008) 2. I used dsync to convert from maildir to mdbox, with dovecot 2.1 at the time. The biggest win is the file number reduction, from 27 million to 2 million. Em 25/11/2013 06:53, Frerich Raabe escreveu: Hi, I run a small IMAP server for a dozen guys in the office, serving about 55GB of Maildir. I recently became aware of the Zlib plugin ( http://wiki2.dovecot.org/Plugins/Zlib ) and wondered 1. given that there is about zero CPU load on my IMAP server, is enabling the plugin a no-brainer or are there other things (except CPU load) to consider? 2. For enabling the plugin, I suppose you compress all the existing mail just once and then add 'zlib' to mail_plugins in order to have all future incoming mail saved? Any insight by people familiar with the plugin would be much appreciated - thanks!
Re: [Dovecot] Zlib plugin - when does it make sense?
Am 25.11.2013 14:12, schrieb Everton Foscarini: I have a relatively big installation here in the university, 1 mail server for 10.000 daily active accounts. I converted from maildir to mdbox + zlib in march, and these are my considerations: 1. CPU is not a problem, even delivering more than 400 messages per minute (blade HS21 with 2xQuad E5430 from 2008) 2. I used dsync to convert from maildir to mdbox, with dovecot 2.1 at the time. The biggest win is the file number reduction, from 27 million to 2 million. But this is due to the conversion to mdbox and not the compression, isn't it? - René
Re: [Dovecot] Zlib plugin - when does it make sense?
On Mon, Nov 25, 2013 at 09:53:14AM +0100, Frerich Raabe wrote: I run a small IMAP server for a dozen guys in the office, serving about 55GB of Maildir. I recently became aware of the Zlib plugin ( http://wiki2.dovecot.org/Plugins/Zlib ) and wondered 1. given that there is about zero CPU load on my IMAP server, is enabling the plugin a no-brainer or are there other things (except CPU load) to consider? Yes, it's a no-brainer. I can't remember how the cpuload was before we enabled zlib, but our cpus are running 80% idle (6 servers, mix of IBM x3550 and x346, serving 15TB mdbox, but was serving maildir with zlib a year ago). 2. For enabling the plugin, I suppose you compress all the existing mail just once and then add 'zlib' to mail_plugins in order to have all future incoming mail saved? You don't strictly need to compress existing mail. It should handle a mix of compressed and non-compressed messages in the same maildir. -jf
Re: [Dovecot] Zlib plugin - when does it make sense?
On 2013-11-25 14:35, Jan-Frode Myklebust wrote: On Mon, Nov 25, 2013 at 09:53:14AM +0100, Frerich Raabe wrote: I run a small IMAP server for a dozen guys in the office, serving about 55GB of Maildir. I recently became aware of the Zlib plugin ( http://wiki2.dovecot.org/Plugins/Zlib ) and wondered 1. given that there is about zero CPU load on my IMAP server, is enabling the plugin a no-brainer or are there other things (except CPU load) to consider? Yes, it's a no-brainer. I can't remember how the cpuload was before we enabled zlib, but our cpus are running 80% idle (6 servers, mix of IBM x3550 and x346, serving 15TB mdbox, but was serving maildir with zlib a year ago). Interesting! What zlib compression level did you use? I figure even low levels would work rather well for plain text. Now that I think about plain text: I also have the fts_solr plugin enabled to speed up the occasional full-text search - does the indexing still work as before when the mail is compressed, i.e. is the reading of the mail centralized so that the individual plugins don't actually know or care? Or would I need to make sure I use 'zlib-aware' plugins? -- Frerich Raabe - ra...@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Re: [Dovecot] Zlib plugin - when does it make sense?
On Mon, Nov 25, 2013 at 02:47:33PM +0100, Frerich Raabe wrote:
Interesting! What zlib compression level did you use? I figure even low
levels would work rather well for plain text.
plugin {
zlib_save_level = 6 # 1..9
zlib_save = gz # or bz2
}
Now that I think about plain text: I also have the fts_solr plugin
enabled to speed up the occasional full-text search - does the indexing
still work as before when the mail is compressed, i.e. is the
reading of the mail centralized so that the individual plugins don't
actually know or care? Or would I need to make sure I use
'zlib-aware' plugins?
I don't have fts (yet).
-jf
Re: [Dovecot] zlib plugin issues
Just realized the imap plugin must be imap_zlib...
It works now.
Thanks
Vijay
On Mon, Aug 26, 2013 at 10:45 PM, Vijay Rajah vijayra...@gmail.com wrote:
Hello,
I'm running dovecot 2.2.5 in Centos 6.4 on EXT4 fs.
I have activated zlib plugin. The incoming mails is compressed. However,
when i try to retrieve the mail using imap I'm getting this error
Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@mydomain.tld): Error:
Cached message size smaller than expected (1464 2452)
Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@mydomain.tld): Error:
Maildir filename has wrong S value, renamed the file from /mail/mailbox/
rvijay.me/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
to
/mail/mailbox/mydomain.tld/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@mydomain.tld): Error:
Corrupted index cache file /mail/mailbox/mydomain.tld/me/Mai
ldir/.vijay_gmail/dovecot.index.cache: Broken physical size for mail UID 12
$ file
/mail/mailbox/mydomain.tld/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5\,S\=1464\:2\,S
/mail/mailbox/mydomain.tld/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,S:
bzip2 compressed data, block size = 900k
Am I missing anything? any ideas is much appreciated.
Thanks
Vijay
###doveconf -n
Here is my doveconf -n
# 2.2.5: /mail/dovecot/2.2.5/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-358.14.1.el6.x86_64 x86_64 CentOS release 6.4 (Final)
ext4
auth_mechanisms = plain login digest-md5 cram-md5
auth_socket_path = /mail/var/dovecot/run/auth-userdb
base_dir = /mail/var/dovecot/run/
debug_log_path = /var/log/dovecot-debug.log
hostname = ma...@rvijay.me
mail_location = maildir:/mail/mailbox/%d/%n/Maildir
mail_plugin_dir = /mail/dovecot/2.2.5/lib/dovecot/
mail_plugins = virtual quota zlib
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox Sent Messages {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
passdb {
args = /mail/dovecot/2.2.5/etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid box msgid size
quota = maildir:User quota
quota_grace = 10%%
quota_rule = *:storage=1GB
quota_rule2 = Trash:storage=+10%%
quota_rule3 = Spam:storage=+20%%
quota_status_nouser = DUNNO
quota_status_overquota = 450 4.2.2 Mailbox is full. Try again later
quota_status_success = DUNNO
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
quota_warning3 = -storage=100%% quota-warning below %u
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /mail/var/dovecot/sieve
zlib_save = bz2
zlib_save_level = 9
}
protocols = imap lmtp sieve
service auth {
unix_listener /mail/var/postfix/dovecot-auth {
group = postfix
mode = 0660
user = postfix
}
}
service imap-login {
inet_listener imap {
port = 143
}
}
service lmtp {
unix_listener /mail/var/postfix/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
service_count = 10
vsz_limit = 64 M
}
service managesieve {
process_limit = 1024
}
service quota-status {
client_limit = 1
executable = quota-status -p postfix
inet_listener {
port = 12340
}
}
service quota-warning {
executable = script
/mail/dovecot/2.2.5/etc/dovecot/conf.d/quota-warning.sh
user = vmail
}
shutdown_clients = no
ssl_ca = /mail/dovecot/2.2.5/etc/dovecot/ssl/mayca.crt.pem
ssl_cert = /mail/dovecot/2.2.5/etc/dovecot/ssl/mail1.rvijay.me.pem
ssl_key = /mail/dovecot/2.2.5/etc/dovecot/ssl/mail1.rvijay.me.key
ssl_parameters_regenerate = 30 hours
submission_host = 127.0.0.1:587
userdb {
driver = prefetch
}
userdb {
args = /mail/dovecot/2.2.5/etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
protocol lda {
mail_plugins = virtual quota sieve zlib
}
protocol imap {
mail_plugins = virtual quota imap_quota zlib
}
protocol lmtp {
mail_plugins = virtual quota sieve zlib
}
Re: [Dovecot] zlib plugin undefined symbol: imap_module_register
Le 19 juil. 2013 à 15:43, Alexander Weber a écrit :
Hi all,
i've tried to enable the zlib plugin within die LDA, but i've got some
strange errors
dovecot: lda: Error:
dlopen(/usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so) failed:
/usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so: undefined symbol:
imap_module_register
dovecot: lda: Fatal: Couldn't load required plugins
[...]
protocol lda {
deliver_log_format = msgid=%m: %$
mail_plugins = sieve, zlib, acl, imap_zlib
postmaster_address = postmaster
quota_full_tempfail = yes
rejection_reason = Your message to %t was automatically rejected:%n%r
}
Hello Alexander,
Should the imap_zlib plugin be listed in the above?
Axel
Re: [Dovecot] Zlib plugin: Compress emails for one folder only
Thanks Timo, I will see. If I have some time, I may have a look at the source code. Kind regards, André. On 20 March 2013 17:58, Timo Sirainen t...@iki.fi wrote: On 16.3.2013, at 14.43, Andre Rodier andre.rod...@gmail.com wrote: Hello everyone, Is it possible to configure the zlib plugin to compress the emails in one folder only? I would like to compress the emails when they are stored in my 'Archives' folders and sub folders. Nope. With hardcoding it would be a simple one line of code addition. Some day in future there will be namespace-specific settings. Then you could create Archive namespace and enable zlib compression only for that namespace. -- André Rodier
Re: [Dovecot] Zlib plugin: Compress emails for one folder only
On 16.3.2013, at 14.43, Andre Rodier andre.rod...@gmail.com wrote: Hello everyone, Is it possible to configure the zlib plugin to compress the emails in one folder only? I would like to compress the emails when they are stored in my 'Archives' folders and sub folders. Nope. With hardcoding it would be a simple one line of code addition. Some day in future there will be namespace-specific settings. Then you could create Archive namespace and enable zlib compression only for that namespace.
Re: [Dovecot] zlib plugin bug?
El 08/03/13 14:13, Jan Phillip Greimann escribió: Hi there, got a problem with the zlib plugin, just wanted to test it on a test-mailserver, configured like in the wiki (http://wiki2.dovecot.org/Plugins/Zlib) and restarted the server. I've send an email to a blank maildir++ mailbox, the file is saved as gzip compressed file, but the Z flag is missing in the filename. zlib plugin doesn't put any flag in messages. The Z flag mentioned in the wiki is one you could put if your are compressing an existing mailbox in order to know which files you have previously compressed. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 86337
Re: [Dovecot] zlib plugin bug?
Am 11.03.2013 08:05, schrieb Angel L. Mateo: zlib plugin doesn't put any flag in messages. The Z flag mentioned in the wiki is one you could put if your are compressing an existing mailbox in order to know which files you have previously compressed. Thank you. I noticed later that there is no Z-flag by default for compressed mails. I think it should be added to the plugin, but maybe it isn't possible. Greetings, Jan
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?
On 2.8.2012, at 10.09, Robert Schetterer wrote: Hi Timo, most of the Problems are fixed now many thx for your script , but how to avoid that in future ( Maildir filename has wrong S value ) You'd need to find out why it started happening in the first place. What software wrote those filenames? Are new filenames still creating with wrong S value? Does your dovecot.conf use zlib plugin globally (instead of only in some protocol sections, which might cause this kind of trouble)? you released a patch diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.cWed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.cWed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream-mail; enum mail_lookup_abort orig_lookup_abort; + return FALSE; if (mstream-expected_size != (uoff_t)-1) return TRUE; would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ? It's an ugly workaround for the wrong S=size problem when zlib is used. Preferably you wouldn't ever need to use it.
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?
Am 03.08.2012 15:56, schrieb Timo Sirainen: On 2.8.2012, at 10.09, Robert Schetterer wrote: Hi Timo, most of the Problems are fixed now many thx for your script , but how to avoid that in future ( Maildir filename has wrong S value ) You'd need to find out why it started happening in the first place. the problem is/was getting maildirs fixed, no time to debug on production system in deed i have no idea why this happend, i found out some broken were old maildirs, so for them i bet coruption by migration 2.0 -2.1 other cuted out attachments imap online with serveral clients on = user fault in my meaning left the cases which i reproduced myself, or some brand new mailboxes it was about compressed mail and nocompressed with wrong message size in one folder What software wrote those filenames? Are new filenames still creating with wrong S value? Does your dovecot.conf use zlib plugin globally (instead of only in some protocol sections, which might cause this kind of trouble)? 10-mail.conf:mail_plugins = $mail_plugins zlib expire virtual fts fts_lucene mail_log autocreate notify acl listescape 20-imap.conf: mail_plugins = imap_acl acl fts fts_lucene mail_log autocreate notify expire virtual quota imap_quota zlib imap_zlib listescape 90-plugin.conf:zlib_save_level = 6 90-plugin.conf:zlib_save = gz should be ok ? you released a patch diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Wed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.c Wed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream-mail; enum mail_lookup_abort orig_lookup_abort; +return FALSE; if (mstream-expected_size != (uoff_t)-1) return TRUE; would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ? It's an ugly workaround for the wrong S=size problem when zlib is used. Preferably you wouldn't ever need to use it. ok , i fixed all mailboxes with your script and one from ralf with compressed maildirs, updated to 2.1.9, using lucene now, disabled stats ( for kernel bugs ), and for now all problems are gone if i cant find any better reason, i go for shit happens, and learned how to fix it, however it were only 6 maildirs from 200 which got broken going trough lot of updates in years , so this is ok in my mind good time for users learn to build some mail archive from time to time however no mail got lost -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?
Am 01.08.2012 14:58, schrieb Timo Sirainen: On 1.8.2012, at 15.43, Robert Schetterer wrote: If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those. i will try things couldn not get more worst ./maildir-size-check.sh -f maildir is an unknown option...? Oh, looks like I forgot to re-upload it after adding that option. It's there now. Hi Timo, most of the Problems are fixed now many thx for your script , but how to avoid that in future ( Maildir filename has wrong S value ) with Ralf you released a patch diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Wed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.c Wed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream-mail; enum mail_lookup_abort orig_lookup_abort; + return FALSE; if (mstream-expected_size != (uoff_t)-1) return TRUE; would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ? -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 01.08.2012 07:36, schrieb Robert Schetterer: Am 31.07.2012 23:44, schrieb Timo Sirainen: On 1.8.2012, at 0.29, Robert Schetterer wrote: Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file in dove 2.1.8 i can reproduce the problem with tb 14 with 2 nat machines connected to the same server the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client sometime this happens at once , sometime after a few copy actions I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them? at my recent onwledge mixing compressed new mails from my sent folder, and older uncompressed ones ( which are shown nice both in tb ), with some copy action in a new folder from another client leads to wrong S and tb does not display them any longer sometime, perhaps a double zip happens? It would help to know how to reproduce this on an entirely empty mail account. i will try to test, but first i have to deal with now broken maildirs i try disable zlip plugin now ,as workaround hopefully this does not lead to more problems on old maildirs , and fix the problem with new mail hi Timo, rolled back all stuff you recommended offlist so situation is a bit better now, however why, i had to handle cause more mailldirs get broken i guess iam dealing with equal Problems described here http://www.mail-archive.com/dovecot@dovecot.org/msg43986.html conversation with ralf, looks like your patch wasnt enough i have a few mailboxes and folders now i cant fix compressed files stay unreadable after move copy by Maildir filename has wrong S value i need some kind of mass repair actions etc , or some new patch -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 01.08.2012 14:06, schrieb Timo Sirainen: See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh tons of i.e Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835 -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
On 1.8.2012, at 15.12, Robert Schetterer wrote: Am 01.08.2012 14:06, schrieb Timo Sirainen: See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh tons of i.e Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835 If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those. (There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.)
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 01.08.2012 14:17, schrieb Timo Sirainen: On 1.8.2012, at 15.12, Robert Schetterer wrote: Am 01.08.2012 14:06, schrieb Timo Sirainen: See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh tons of i.e Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835 If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those. i will try things couldn not get more worst (There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails jep compressed is a must have and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.) i will report next using script -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 01.08.2012 14:25, schrieb Robert Schetterer: Am 01.08.2012 14:17, schrieb Timo Sirainen: On 1.8.2012, at 15.12, Robert Schetterer wrote: Am 01.08.2012 14:06, schrieb Timo Sirainen: See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh tons of i.e Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835 If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those. i will try things couldn not get more worst (There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails jep compressed is a must have and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.) i will report next using script ./maildir-size-check.sh -f maildir is an unknown option...? -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 01.08.2012 14:58, schrieb Timo Sirainen: On 1.8.2012, at 15.43, Robert Schetterer wrote: If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those. i will try things couldn not get more worst ./maildir-size-check.sh -f maildir is an unknown option...? Oh, looks like I forgot to re-upload it after adding that option. It's there now. looks like working fine, but it has some problems with pathnames with special char and spaces -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
On 1.8.2012, at 0.29, Robert Schetterer wrote: Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file in dove 2.1.8 i can reproduce the problem with tb 14 with 2 nat machines connected to the same server the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client sometime this happens at once , sometime after a few copy actions I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them? It would help to know how to reproduce this on an entirely empty mail account.
Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Am 31.07.2012 23:44, schrieb Timo Sirainen: On 1.8.2012, at 0.29, Robert Schetterer wrote: Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file in dove 2.1.8 i can reproduce the problem with tb 14 with 2 nat machines connected to the same server the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client sometime this happens at once , sometime after a few copy actions I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them? at my recent onwledge mixing compressed new mails from my sent folder, and older uncompressed ones ( which are shown nice both in tb ), with some copy action in a new folder from another client leads to wrong S and tb does not display them any longer sometime, perhaps a double zip happens? It would help to know how to reproduce this on an entirely empty mail account. i will try to test, but first i have to deal with now broken maildirs i try disable zlip plugin now ,as workaround hopefully this does not lead to more problems on old maildirs , and fix the problem with new mail -- Best Regards MfG Robert Schetterer
Re: [Dovecot] zlib plugin doesn't read concatenated .gz mailbox
On Fri, 2011-07-29 at 20:00 +0300, Dmitry Nezhevenko wrote: I've observed that dovecot doesn't displays all mails from compressed via gzip mailbox (mbox). At the same time mutt -f mailbox.gz displays it correctly with all messages. Fixed: http://hg.dovecot.org/dovecot-2.0/rev/35e4a547231c
Re: [Dovecot] zlib plugin weirdness
On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote: I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 05:06 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote: I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 No, that causes a rather ugly panic: Error: zlib.read(/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz): unexpected EOF at 1764046 Panic: file istream-zlib.c: line 204 (i_stream_zlib_read): assertion failed: (zstream-eof_offset == high_offset) Error: Raw backtrace: /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 [0xb76769e1] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 [0xb7676a5f] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_error+0) [0xb7676d18] - /tmp/dovecottesting/lib/dovecot/lib20_zlib_plugin.so [0xb74aaf7a] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_stream_read+0x7c) [0xb767d50c] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb7730cc1] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(istream_raw_mbox_get_header_offset+0x78) [0xb77316a8] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb773bb99] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb773e154] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_sync+0x4b) [0xb773f35b] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_storage_sync_init+0x6d) [0xb773f4cd] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x39) [0xb76d0949] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x44) [0xb76d11f4] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](cmd_select_full+0x1d2) [0x8053962] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](cmd_select+0x19) [0x80543d9] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x805673c] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x80567d9] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](client_handle_input+0x2d) [0x805694d] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](client_input+0x5f) [0x805733f] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xf5) [0xb7683275] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) [0xb7682250] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(master_service_run+0x2a) [0xb766f72a] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](main+0x2b5) [0x805ffb5] - /lib/i686/cmov/libc.so.6(__libc_start_main+0xe5) [0xb74e3455] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x804eac1] master: Error: service(imap): child 15613 killed with signal 6 (core dumped) ~Kyle -- Why, this Satan's drink is so delicious that it would be a pity to let the infidels have exclusive use of it. We shall fool Satan by baptizing it, and making it a truly Christian beverage. -- Pope Clement VII, referring to coffee signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 05:46 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 11:21 -0500, Kyle Wheeler wrote: On Friday, October 1 at 05:06 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote: I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 No, that causes a rather ugly panic: What about the above + http://hg.dovecot.org/dovecot-2.0/rev/e7768ec9d3de That does it for gz-compressed files; it works! However, for bz2-compressed files, I get: Error: Next message unexpectedly corrupted in mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: Unexpectedly lost From-line from mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: i_stream_get_size() failed with mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: No such file or directory ~Kyle -- Time is an illusion. Lunchtime doubly so. -- Douglas Adams signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 11:51 AM, quoth Kyle Wheeler: That does it for gz-compressed files; it works! However, for bz2-compressed files, I get: Error: Next message unexpectedly corrupted in mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: Unexpectedly lost From-line from mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: i_stream_get_size() failed with mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: No such file or directory I made the change suggested by http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 to istream-bzlib.c, and things now work perfectly for bz2-compressed files as well. ~Kyle -- He who joyfully marches in rank and file has already earned my contempt. He has been given a large brain by mistake, since for him the spinal cord would suffice. -- Albert Einstein signature.asc Description: Digital signature
Re: [Dovecot] Zlib plugin dovecot 2.0beta4
Mostrar romanização
The extra line added to the header is to maintain compatibility with
SMTP(share library code), which already works well. Besides having other
benefits that are due to the hash in the header.
I tried accessing the same way that istream is done in the zlib (
zlib_permail_get_stream ), where the reference to mail- uid == 0, (nonzero
is to read the file - dovecot).
if (imail-data.stream != NULL || (_mail-uid == 0 zuser-disable_uis ==
1)) { //disable_uis ir read like the zlib_save variable , if is not present
, we don't need to get stream, and works like no modifications.
return zmail-super.get_stream(_mail, hdr_size, body_size, stream_r);
}else if (imail-data.stream != NULL || _mail-uid == 0 ) { // OK, its a
istream from received email.
input = imail-data.stream;
imail-data.stream = i_stream_create_emx(input);
}
But I do not know if it is correct.
If you can tell me where is exactly the way or the point that i must catch
the istream input, i do the rest.
Tks Timo !
2010/5/26 Timo Sirainen t...@iki.fi
On Tue, 2010-05-11 at 16:41 -0300, Alex Baule wrote:
But i need to get / change some values to create the link from header
to
body.
When a email is saved, have the S an W flags, that is the size of email.
There is a way to increase this Flags ? I need to put one more line in
the
header, to have some link to body.
So you want to add one header and have the S/W increase the number? The
problem isn't only the S/W, it's also that Dovecot saves the message
size to cache file. The only good way to solve this is to change your
input stream to return the header, so that all of Dovecot's internal
code sees it and internally adds them to the S/W/cache.
But do you really even need a header for the link? Maybe you could just
use IMAP UID number? Like if you have ~/Maildir/.foo/cur/abcd:2, file
with UID=200, its body could be in e.g. ~/Maildir/.foo/body/200.
Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream
Hi Timo... I made the plugin like you say. The header part is saved by dovecot, with o_stream_send , and the body with my function. This implementation, works fine. But i need to get / change some values to create the link from header to body. When a email is saved, have the S an W flags, that is the size of email. There is a way to increase this Flags ? I need to put one more line in the header, to have some link to body. The second is, the name of file. I use the o_stream_get_name, passing the output ostream, but the return is empty. The name is avaliable when ? Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi You should write the message header to the ostream that you get, and message body to some other file that you create. Use the o_stream_send() function to write the message header to the ostream (which does basically the same as write(fd)). I don't see how getting the stream's fd would help with anything. On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote: Because my plugin will separate the body of the message header. With the FD I have control over the message to separate it. Do you have any suggestions for this separation? I do not quite understand how the struct ostream works. Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote: There is a way to get the file descriptor from output used in ostream zlib plugin ? No. Why do you need it? You should be writing to ostream, not to its fd.
Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream
Hi Timo I'm doing the plugin based on what you said. But I'll add a line in the message header, so the S and W (size of email) flags, on the file name has to be changed ... There is a way to do that in the plugin? 2010/4/20 Alex Baule alexwba...@gmail.com If I had the file descriptor, I will reuse my functions to do this, made in a separate library, that i use in the MTA. So no need to rewrite again the part of split the message. 2010/4/20 Timo Sirainen t...@iki.fi You should write the message header to the ostream that you get, and message body to some other file that you create. Use the o_stream_send() function to write the message header to the ostream (which does basically the same as write(fd)). I don't see how getting the stream's fd would help with anything. On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote: Because my plugin will separate the body of the message header. With the FD I have control over the message to separate it. Do you have any suggestions for this separation? I do not quite understand how the struct ostream works. Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote: There is a way to get the file descriptor from output used in ostream zlib plugin ? No. Why do you need it? You should be writing to ostream, not to its fd.
Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream
Because my plugin will separate the body of the message header. With the FD I have control over the message to separate it. Do you have any suggestions for this separation? I do not quite understand how the struct ostream works. Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote: There is a way to get the file descriptor from output used in ostream zlib plugin ? No. Why do you need it? You should be writing to ostream, not to its fd.
Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream
You should write the message header to the ostream that you get, and message body to some other file that you create. Use the o_stream_send() function to write the message header to the ostream (which does basically the same as write(fd)). I don't see how getting the stream's fd would help with anything. On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote: Because my plugin will separate the body of the message header. With the FD I have control over the message to separate it. Do you have any suggestions for this separation? I do not quite understand how the struct ostream works. Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote: There is a way to get the file descriptor from output used in ostream zlib plugin ? No. Why do you need it? You should be writing to ostream, not to its fd. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream
If I had the file descriptor, I will reuse my functions to do this, made in a separate library, that i use in the MTA. So no need to rewrite again the part of split the message. 2010/4/20 Timo Sirainen t...@iki.fi You should write the message header to the ostream that you get, and message body to some other file that you create. Use the o_stream_send() function to write the message header to the ostream (which does basically the same as write(fd)). I don't see how getting the stream's fd would help with anything. On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote: Because my plugin will separate the body of the message header. With the FD I have control over the message to separate it. Do you have any suggestions for this separation? I do not quite understand how the struct ostream works. Tks Timo. 2010/4/20 Timo Sirainen t...@iki.fi On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote: There is a way to get the file descriptor from output used in ostream zlib plugin ? No. Why do you need it? You should be writing to ostream, not to its fd.
Re: [Dovecot] zlib Plugin Dovecot 2.0
On Fri, 2010-04-09 at 11:42 -0300, Alex Baule wrote: I developed a plugin based on the zlib plugin, but want to port it to dovecot 2.0. There are some differences in the 2.0 plugin, which I did not quite understand how it works for me to readjust it in my plugin. I don't think you need to do (almost?) any changes. v2.0's zlib plugin is very different from v1.x, but that's because I'm using zlib library in a different way. Not because of some istream changes. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib Plugin Dovecot 2.0
Thanks Timo, I have another question about the plugin. Part of reading I already migrated, only that the piece of writing (the new part in the plugin) I need to understand some things about the operation in order to migrate and redo the same functions with the separation of body / header Can you explain to me ? Tks again ! 2010/4/16 Timo Sirainen t...@iki.fi On Fri, 2010-04-09 at 11:42 -0300, Alex Baule wrote: I developed a plugin based on the zlib plugin, but want to port it to dovecot 2.0. There are some differences in the 2.0 plugin, which I did not quite understand how it works for me to readjust it in my plugin. I don't think you need to do (almost?) any changes. v2.0's zlib plugin is very different from v1.x, but that's because I'm using zlib library in a different way. Not because of some istream changes.
Re: [Dovecot] zlib Plugin Dovecot 2.0
On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote: Part of reading I already migrated, only that the piece of writing (the new part in the plugin) I need to understand some things about the operation in order to migrate and redo the same functions with the separation of body / header So you also want to add support for the split operating directly during saving new mails? I guess you can copypaste most of the code from zlib-plugin.c and just implement a new ostream. ostream-bzlib.c should give a good example. You just need to implement: - sendv(): this works similar to writev() syscall. Here you need to keep read the input bytes and track when the message header ends and write to different files before/after that. - flush(): If you do any buffering, flush it here. That's about it. Much simpler than istreams! signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib Plugin Dovecot 2.0
On Fri, 2010-04-16 at 18:34 +0300, Timo Sirainen wrote: On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote: Part of reading I already migrated, only that the piece of writing (the new part in the plugin) I need to understand some things about the operation in order to migrate and redo the same functions with the separation of body / header So you also want to add support for the split operating directly during saving new mails? I guess you can copypaste most of the code from zlib-plugin.c and just implement a new ostream. ostream-bzlib.c should give a good example. You just need to implement: Oh, and the output ostream that you get from create() is the one that ends up in the maildir. So I guess you should be writing headers there, and write the body to another file some other way. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib Plugin Dovecot 2.0
Ok !
That's easy.
In the sendv rewrite, have this:
for (i = 0; i iov_count; i++) {
if (o_stream_emexis_send_chunk(emexis_stream, iov[i].iov_base,
iov[i].iov_len) 0)
return -1;
bytes += iov[i].iov_len;
}
This iov is every line from the new emaill to save ?
Or this is a email block with X bytes ?
In my function to write email, i split the email by \n, to save it splited,
and if the iov is splited by \n, is good for me.
2010/4/16 Timo Sirainen t...@iki.fi
On Fri, 2010-04-16 at 18:34 +0300, Timo Sirainen wrote:
On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote:
Part of reading I already migrated, only that the piece of writing (the
new
part in the plugin) I need to understand some things about the
operation in
order to migrate and redo the same functions with the separation of
body /
header
So you also want to add support for the split operating directly during
saving new mails?
I guess you can copypaste most of the code from zlib-plugin.c and just
implement a new ostream. ostream-bzlib.c should give a good example. You
just need to implement:
Oh, and the output ostream that you get from create() is the one that
ends up in the maildir. So I guess you should be writing headers there,
and write the body to another file some other way.
Re: [Dovecot] zlib Plugin Dovecot 2.0
On 16.4.2010, at 19.15, Alex Baule wrote: This iov is every line from the new emaill to save ? Or this is a email block with X bytes ? In my function to write email, i split the email by \n, to save it splited, and if the iov is splited by \n, is good for me. You can't assume anything about iov. iov_count is typically 1, but can be much more. Each block's length can be from 1 byte to anything (typically less than 8kB I guess).
Re: [Dovecot] zlib Plugin Dovecot 2.0
I Dont want the configuration. I Want know how is implemented. Explain, the functions, the hooks to get write a email stuffs like this. 2010/4/8 Alex Baule alexwba...@gmail.com Hi Everyone Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ?? In zlib from 2.0 beta4, Which part of the plugin is responsible for write the compressed message ?
Re: [Dovecot] zlib Plugin Dovecot 2.0
On 04/09/2010 02:45 PM Alex Baule wrote: I Dont want the configuration. I Want know how is implemented. Explain, the functions, the hooks to get write a email stuffs like this. How it is implemented: http://hg.dovecot.org/dovecot-2.0/file/tip/src/plugins/zlib Regards, Pascal -- The trapper recommends today: defaced.1009...@localdomain.org
Re: [Dovecot] zlib Plugin Dovecot 2.0
Hi Paschal You do not understand what I want. I developed a plugin based on the zlib plugin, but want to port it to dovecot 2.0. There are some differences in the 2.0 plugin, which I did not quite understand how it works for me to readjust it in my plugin. As in the source code there are not many comments, I do not quite know what each function does, if they are all necessary in my adaptation. In developing the plugin in version 1.4, Timo helped me a lot. What I need to know is how the hooks on write emails do this, and what the differences in the read function, since the code is different from version 1.4 to 2.0.
Re: [Dovecot] zlib Plugin Dovecot 2.0
Hi Paschal You do not understand what I want. I developed a plugin based on the zlib plugin, but want to port it to dovecot 2.0. There are some differences in the 2.0 plugin, which I did not quite understand how it works for me to readjust it in my plugin. As in the source code there are not many comments, I do not quite know what each function does, if they are all necessary in my adaptation. In developing the plugin in version 1.4, Timo helped me a lot. What I need to know is how the hooks on write emails do this, and what the differences in the read function, since the code is different from version 1.4 to 2.0. 2010/4/9 Alex Baule alexwba...@gmail.com I Dont want the configuration. I Want know how is implemented. Explain, the functions, the hooks to get write a email stuffs like this. 2010/4/8 Alex Baule alexwba...@gmail.com Hi Everyone Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ?? In zlib from 2.0 beta4, Which part of the plugin is responsible for write the compressed message ?
Re: [Dovecot] zlib plugin trouble
On Tue, 2010-02-16 at 15:30 +0100, Stéphane Cottin wrote: Hi, I have error with dovecot 1.2.10 + compress on deliver patch and zlib plugin. logfile sample: [...] mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC Yeah, bzip2 code is broken in v1.2. I fixed it in v2.0 by basically rewriting the whole thing. Maybe I should just disable bzip2 support in v1.2. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib plugin trouble
Em 16/02/2010 12:30, Stéphane Cottin escreveu: Hi, I have error with dovecot 1.2.10 + compress on deliver patch and zlib plugin. logfile sample: [...] mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC [...] i have already reported some similar errors when messages are compressed with bzip. I couldnt reproduce with gzip, which you appears to be using: zlib_save: gz do you used some script for compressing your existing messages, those who were delivered before you getting LDA to do that automatically ? If yes, is there any chance that you used bzip2 instead of gzip on that script ? -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it
Re: [Dovecot] zlib plugin trouble
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Le 16 févr. 2010 à 16:27, Timo Sirainen a écrit : On Tue, 2010-02-16 at 15:30 +0100, Stéphane Cottin wrote: Hi, I have error with dovecot 1.2.10 + compress on deliver patch and zlib plugin. logfile sample: [...] mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC Yeah, bzip2 code is broken in v1.2. I fixed it in v2.0 by basically rewriting the whole thing. Maybe I should just disable bzip2 support in v1.2. I've first tried bzip2 but these are errors with gzip. -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAkt6uswACgkQUeIu/wInUewfbgCffU7QKSaZ/cirfmuJZdCTi8Vx QMAAn0X+oKT/Kp0nwEwAdHbm3Wb+z8B3 =3/7y -END PGP SIGNATURE-
Re: [Dovecot] zlib plugin trouble
Em 16/02/2010 13:27, Timo Sirainen escreveu: Yeah, bzip2 code is broken in v1.2. I fixed it in v2.0 by basically rewriting the whole thing. Maybe I should just disable bzip2 support in v1.2. will dovecot 2.0 be able to handling mixed maildirs with gzip and bzip2 compressed messages, or all messages should use just one or the other ? -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it
Re: [Dovecot] zlib plugin trouble
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Le 16 févr. 2010 à 16:28, Leonardo Rodrigues a écrit : Em 16/02/2010 12:30, Stéphane Cottin escreveu: Hi, I have error with dovecot 1.2.10 + compress on deliver patch and zlib plugin. logfile sample: [...] mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:25 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): zlib_istream.seek() failed: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): FETCH for mailbox Meursault UID 101 failed to read message input: Invalid argument mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC [...] i have already reported some similar errors when messages are compressed with bzip. I couldnt reproduce with gzip, which you appears to be using: zlib_save: gz do you used some script for compressing your existing messages, those who were delivered before you getting LDA to do that automatically ? If yes, is there any chance that you used bzip2 instead of gzip on that script ? I was testing bzip2, but as timo said, it's broken in 1.2. I made a simple script to gzip uncompressed messages and convert bzip2 to gzip (I can share it if anyone cares), There is no bzip2 compressed files left. I will activate mail_debug to get more details and be certain the bug occurs with a good gzipped file. -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAkt6vCMACgkQUeIu/wInUezHBACfV2VM2Nwcolj0LMOQjmq5NqgL GCIAniRICtoOHuIfCj2uJ++/cgxqpqQK =LQGB -END PGP SIGNATURE-
Re: [Dovecot] zlib plugin trouble
On 16.2.2010, at 17.39, Stéphane Cottin wrote: mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() failed: DATA_ERROR_MAGIC I made a simple script to gzip uncompressed messages and convert bzip2 to gzip (I can share it if anyone cares), There is no bzip2 compressed files left. The above error message string is coming from bzip2 library. % grep DATA_ERROR_MAGIC /usr/lib/libz.so % grep DATA_ERROR_MAGIC /usr/lib/libbz2.so Binary file /usr/lib/libbz2.so matches So I think you do have some bzip2 compressed files left.
Re: [Dovecot] zlib plugin
On 281, 10 07, 2008 at 03:10:47PM +0300, Timo Sirainen wrote: On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote: On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote: On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load. But is that 1.8 in bytes or disk blocks? Bytes. Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message? Yes, for small messages it's not very usefull, but multimegabyte mails are quite common nowadays and for them compression is a clear win. Also I remember that load average decreased when I started to use message compression. I have no real numbers now, but it was noticeable. -- Andrey Panin| Linux and UNIX system administrator [EMAIL PROTECTED] | PGP key: wwwkeys.pgp.net signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin
On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote: On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load. -- Andrey Panin| Linux and UNIX system administrator [EMAIL PROTECTED] | PGP key: wwwkeys.pgp.net signature.asc Description: Digital signature
Re: [Dovecot] Zlib plugin question
Ralf Hildebrandt wrote: http://wiki.dovecot.org/Plugins/Zlib says pop3 and imap can use this plugin. Does deliver also use it? No, the zlib plugin is only for reading compressed messages/mailboxes. You have to compress them manually. -- calc Knghtbrd: irc doesn't compile c code very well ;) Eduardo M KALINOWSKI [EMAIL PROTECTED] http://move.to/hpkb
Re: [Dovecot] zlib plugin
On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote: On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote: On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load. But is that 1.8 in bytes or disk blocks? Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Zlib plugin question
* Eduardo M KALINOWSKI [EMAIL PROTECTED]: Ralf Hildebrandt wrote: http://wiki.dovecot.org/Plugins/Zlib says pop3 and imap can use this plugin. Does deliver also use it? No, the zlib plugin is only for reading compressed messages/mailboxes. You have to compress them manually. awww, too bad. -- Ralf Hildebrandt ([EMAIL PROTECTED]) [EMAIL PROTECTED] Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de I'm looking for a job Applying computer technology is simply finding the right wrench to pound in the correct screw.
Re: [Dovecot] zlib plugin
Timo Sirainen wrote: On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote: On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote: On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. I use zlib compressed maildirs on my primary server (12000+ users) and it achieves average compression ratio of 1.8, so if your system is io-bound you can spend some cpu cycles to significantly decrease disk load. But is that 1.8 in bytes or disk blocks? Or does it make much of a difference either way? Does 2x less space per message even matter with small messages since most disk I/O probably goes to seeking, so reading a single 4 kB message takes pretty much the same time as a 8 kB message? Just throwing in my 0.02 currency units... ... obviously, which messages you choose to compress is up to you, and should be tuned depending on your system. Certainly, I wouldn't bother compressing anything smaller than 1 allocation block (whatever that is on your FS), and even 2, unless your CPU is really idle. The larger the message, the greater the potential rewards. PS: Timo... 2x smaller? ... somehow, I expected better of you :) -- Curtis Maloney [EMAIL PROTECTED]
Re: [Dovecot] zlib plugin
On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: hi, Could anyone give some info on this plugin? When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still left with some questions... I use Maildir to store my mails, if i enable this zlib-plugin, should i 'manually' compress all messages? or wil the uncompressed messages still be accessible by dovecot? wil new messages be compressed automagically? Dovecot can read both compressed and uncompressed messages. how should the compressed messages be named? It doesn't matter from Dovecot's point of view since it detects the compression by reading the first few bytes of the file. But since you'll be compressing them manually you should mark them in some way so you don't compress anything twice (and that would break Dovecot too). a message now has the name '1223224520.2783_1.opensuse:2,S' how would a compressed message be named? ['1223224520.2783_1.opensuse:2,SZ' This is OK. or '1223224520.2783_1.opensuse:2,S,Z' This isn't a valid maildir file name. Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib plugin
Timo Sirainen schreef: On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote: hi, Could anyone give some info on this plugin? When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still left with some questions... I use Maildir to store my mails, if i enable this zlib-plugin, should i 'manually' compress all messages? or wil the uncompressed messages still be accessible by dovecot? wil new messages be compressed automagically? Dovecot can read both compressed and uncompressed messages. how should the compressed messages be named? It doesn't matter from Dovecot's point of view since it detects the compression by reading the first few bytes of the file. But since you'll be compressing them manually you should mark them in some way so you don't compress anything twice (and that would break Dovecot too). a message now has the name '1223224520.2783_1.opensuse:2,S' how would a compressed message be named? ['1223224520.2783_1.opensuse:2,SZ' This is OK. or '1223224520.2783_1.opensuse:2,S,Z' This isn't a valid maildir file name. Last but not least, what gain will i have by using this plugin in combination with Maildir, or will it have more impact when mails are stored in mbox. I guess it depends on the size of your mails. With mboxes it can be used only with read-only archives, so it's not all that useful. With maildir it can be used transparently with read-write mailboxes. I've no idea what the performance difference or space improvements are. I guess that'll also depend on your system. ok, thank you, in the near furure i will try to install this zlib-plugin (on a test-system) and see what happens... -- Luuk $ dovecot --version dovecot version 1.1.1 $ uname -srpmio Linux 2.6.18.8-0.10-default x86_64 x86_64 x86_64 GNU/Linux
Re: [Dovecot] zlib plugin and moving mails, bug?
Dnia piątek, 18 lipca 2008, Nuno Lopes napisał: This was a bug in version 1.0 which is now corrected in version 1.1.rc6 by looking at the file header instead of the Z flag. Yet another reason to upgrade to 1.1... -- Mateusz Kijowski Biuro Informatyki DFQS mail: [EMAIL PROTECTED] | tel: 058 7668 644 AA58 B5B3 4A94 997F 78E9 A28C 886D 4C58 F1A5 4F2B *** Dom Finansowy QS Sp. z o.o. z siedzibą w Gdańsku przy ul. Piastowska 11, wpisany do Rejestru Przedsiębiorców Krajowego Rejestru Sądowego przez Sąd Rejonowy Gdańsk - Północ w Gdańsku pod numerem KRS 66674, z kapitałem zakładowym w wysokości 50.000,00 PLN, NIP 957-08-11-591. signature.asc Description: This is a digitally signed message part.
Re: [Dovecot] zlib plugin nad moving mails, bug?
On Fri, 2008-07-18 at 16:14 +0200, Mateusz Kijowski wrote: Recently I gzipped mails older than 30 days in our users' maildirs. I used find and gzip. Gzip added 'Z' at the end of the filename and everything seemed to work fine, users did not notice any difference in performance (yet :-), and I saved about 30% of our mail storage space. There seems to be a problem when moving gzipped messages from one folder to another though. Dovecot drops the Z flag, moved messages are not processed by the zlib plugin and email clients end up showing garbage. Is there something wrong with my config, or did I encounter a bug? .. maildir_copy_with_hardlinks: yes Hardlink copying is broken with zlib plugin. v1.1 works differently and it works there (although non-hardlink copying was broken there, fixed only in hg). signature.asc Description: This is a digitally signed message part
Re: [Dovecot] zlib plugin nad moving mails, bug?
This was a bug in version 1.0 which is now corrected in version 1.1.rc6 by looking at the file header instead of the Z flag. http://dovecot.org/pipermail/dovecot-news/2008-May/69.html Regards, -- Nuno Mateusz Kijowski wrote: Hi, Recently I gzipped mails older than 30 days in our users' maildirs. I used find and gzip. Gzip added 'Z' at the end of the filename and everything seemed to work fine, users did not notice any difference in performance (yet :-), and I saved about 30% of our mail storage space. There seems to be a problem when moving gzipped messages from one folder to another though. Dovecot drops the Z flag, moved messages are not processed by the zlib plugin and email clients end up showing garbage. Is there something wrong with my config, or did I encounter a bug? dovecot -n follows: # 1.0.13: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log protocols: imap imaps pop3s ssl_ca_file: /etc/certs/certum-chain.crt ssl_cert_file: /etc/certs/wild.dfqs.pl.crt ssl_key_file: /etc/certs/wild.dfqs.pl.key ssl_cipher_list: ALL:!LOW:!MEDIUM login_dir: /var/run/dovecot/login login_executable(default): /usr/lib64/dovecot/imap-login login_executable(imap): /usr/lib64/dovecot/imap-login login_executable(pop3): /usr/lib64/dovecot/pop3-login login_greeting: imap.dfqs.pl ready. login_process_per_connection: no login_processes_count: 5 login_max_processes_count: 32 max_mail_processes: 4096 verbose_proctitle: yes mail_location: maildir:~/mail:INDEX=/var/lib/dovecot/indexes/%Ln mail_debug: yes maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_executable(default): /usr/local/lib64/dovecot-post-login-imap.sh mail_executable(imap): /usr/local/lib64/dovecot-post-login-imap.sh mail_executable(pop3): /usr/local/lib64/dovecot-post-login-pop3.sh mail_plugins(default): imap_quota trash quota acl zlib mail_plugins(imap): imap_quota trash quota acl zlib mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib64/dovecot/plugins/imap mail_plugin_dir(imap): /usr/lib64/dovecot/plugins/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/plugins/pop3 mail_log_max_lines_per_sec: 0 imap_client_workarounds(default): outlook-idle tb-extra-mailbox-sep imap_client_workarounds(imap): outlook-idle tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh namespace: type: private separator: / inbox: yes namespace: type: public separator: / prefix: Public/ location: maildir:/var/vmail/Public:CONTROL=~/Public/control:INDEX=~/Public/index namespace: type: shared separator: / prefix: Shared/ location: maildir:/var/vmail/Shared:CONTROL=~/Shared/control:INDEX=~/Shared/index auth default: verbose: yes debug: yes worker_max_count: 512 passdb: driver: pam args: dovecot userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 user: root group: vmail master: path: /var/run/dovecot/auth-master mode: 432 user: root group: vmail plugin: quota: maildir:storage=12582912:ignore=Trash acl: vfile: trash: /etc/dovecot/dovecot-trash.conf
Re: [Dovecot] zlib plugin
On 7.4.2007, at 3.08, guenther wrote: Well, here at least. ;) I recently found out by accident about support for gzip compressed mbox files. While this is great for archives to keep wasting of space down to a minimum, unfortunately it is read only -- no support for rw access. Is implementing write access by any chance planned for future releases? Nope. Not planned at all. It would require changing quite a lot of code. PGP.sig Description: This is a digitally signed message part
