Re: [Dovecot] zlib maildir reindex broken

[Robert Schetterer]

Am 03.07.2014 16:55, schrieb Christian Rohmann:
> Hey Timo,
> 
> On 02.05.2014 12:37, Timo Sirainen wrote:
>>> During a reindex maildir_mail_get_received_date() does an i_stream_stat on 
>>> the the i_stream_seekable stream created in zlib_mail_cache_open, but this 
>>> istream does not know about the original maildir message file and always 
>>> returns the current time for the file's stat times.
>>> This is also broken on initial index, but if mail is indexed when it's 
>>> received (ours isn't) it coincidentally gets the right time.
>>
>> Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/9f1460682295
> 
> It seems this is unfortunately also broken in 2.1.17.
> 
> If possible could you also provide a fix for the 2.1.x tree, as we are
> still happily running your code from that era. Your fix for 2.2 does
> sadly not apply without modifications and I don't want to destroy things
> with incompetent attempts to do production grade C coding ;-)
> 
> 
> 
> With regards from Cologne,
> 
> 
> Christian
> 

I strongly vote for a 2.1.18 including such fix


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] zlib maildir reindex broken

[Christian Rohmann]

Hey Timo,

On 02.05.2014 12:37, Timo Sirainen wrote:
>> During a reindex maildir_mail_get_received_date() does an i_stream_stat on 
>> the the i_stream_seekable stream created in zlib_mail_cache_open, but this 
>> istream does not know about the original maildir message file and always 
>> returns the current time for the file's stat times.
>> This is also broken on initial index, but if mail is indexed when it's 
>> received (ours isn't) it coincidentally gets the right time.
> 
> Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/9f1460682295

It seems this is unfortunately also broken in 2.1.17.

If possible could you also provide a fix for the 2.1.x tree, as we are
still happily running your code from that era. Your fix for 2.2 does
sadly not apply without modifications and I don't want to destroy things
with incompetent attempts to do production grade C coding ;-)



With regards from Cologne,


Christian

Re: [Dovecot] zlib maildir reindex broken

[Timo Sirainen]

On 24.4.2014, at 0.51, Richard Platel  wrote:

> I posted to the list about this a while ago but never got a response, I have 
> a bit more information now.
> 
> Dovecot 2.2.12 and other 2.2 versions are broken when using zlib and maildir. 
>  If messages are re-indexed, the INTERNALDATE of all messages is set to the 
> time the re-index is done.
> 
> The problem seems to be in src/plugins/zlib/zlib-plugin.c in the function 
> zlib_mail_cache_open.
> 
> During a reindex maildir_mail_get_received_date() does an i_stream_stat on 
> the the i_stream_seekable stream created in zlib_mail_cache_open, but this 
> istream does not know about the original maildir message file and always 
> returns the current time for the file's stat times.
> 
> This is also broken on initial index, but if mail is indexed when it's 
> received (ours isn't) it coincidentally gets the right time.

Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/9f1460682295

[Dovecot] zlib maildir reindex broken

[Richard Platel]

I posted to the list about this a while ago but never got a response, I have a 
bit more information now.

Dovecot 2.2.12 and other 2.2 versions are broken when using zlib and maildir.  
If messages are re-indexed, the INTERNALDATE of all messages is set to the time 
the re-index is done.
  
The problem seems to be in src/plugins/zlib/zlib-plugin.c in the function 
zlib_mail_cache_open.

During a reindex maildir_mail_get_received_date() does an i_stream_stat on the 
the i_stream_seekable stream created in zlib_mail_cache_open, but this istream 
does not know about the original maildir message file and always returns the 
current time for the file's stat times.

This is also broken on initial index, but if mail is indexed when it's received 
(ours isn't) it coincidentally gets the right time.

Re: [Dovecot] zlib unexpected EOF

[Robert Schetterer]

Am 04.02.2014 10:16, schrieb Anand Kumria:
> Hi,
> 
> I am seeing occasional log entries like:
> 
> Feb  4 06:20:01 mail1 dovecot: imap(u...@example.com): Error: read( client>) failed: zlib.read(): unexpected EOF at 894774
> Feb  4 06:20:01 mail1 dovecot: imap(u...@example.com): Disconnected in
> APPEND (1 msgs, 201 secs, 0/5180550 bytes) in=894774 out=643701
> 
> I've been able to correlate this to users - using Thunderbird - complaining
> that they have been unable to "send email".
> 
> Is the problem on the Dovecot side, their side or something else?
> 
> If on the Dovecot any pointers and what I should look at?
> 
> Thanks,
> Anand
> 

let us see your conf


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

[Dovecot] zlib unexpected EOF

[Anand Kumria]

Hi,

I am seeing occasional log entries like:

Feb  4 06:20:01 mail1 dovecot: imap(u...@example.com): Error: read() failed: zlib.read(): unexpected EOF at 894774
Feb  4 06:20:01 mail1 dovecot: imap(u...@example.com): Disconnected in
APPEND (1 msgs, 201 secs, 0/5180550 bytes) in=894774 out=643701

I've been able to correlate this to users - using Thunderbird - complaining
that they have been unable to "send email".

Is the problem on the Dovecot side, their side or something else?

If on the Dovecot any pointers and what I should look at?

Thanks,
Anand

Re: [Dovecot] zlib plugin question

[Frerich Raabe]

On 2013-12-19 17:51, Frerich Raabe wrote:

On 2013-12-19 16:07, Przemysław Orzechowski wrote:
Is it possible to compress incoming mails delivered via dovecots LDA 
when using

dovecot --version
1.2.9
or do i have to compress them via cron?


There is a zlib plugin for Dovecot 1.x you could use. See

  http://wiki1.dovecot.org/Plugins/Zlib


Sorry, I only now noticed that this plugin only allows reading 
compressed mail
files - apparently you would need Dovecot v2.x if you want to compress 
incoming

mail.

--
Frerich Raabe - ra...@froglogic.com
www.froglogic.com - Multi-Platform GUI Testing

Re: [Dovecot] zlib plugin question

[Frerich Raabe]

On 2013-12-19 16:07, Przemysław Orzechowski wrote:
Is it possible to compress incoming mails delivered via dovecots LDA 
when using

dovecot --version
1.2.9
or do i have to compress them via cron?


There is a zlib plugin for Dovecot 1.x you could use. See

  http://wiki1.dovecot.org/Plugins/Zlib

--
Frerich Raabe - ra...@froglogic.com
www.froglogic.com - Multi-Platform GUI Testing

[Dovecot] zlib plugin question

[Przemysław Orzechowski]

Hi

Is it possible to compress incoming mails delivered via dovecots LDA 
when using

dovecot --version
1.2.9
or do i have to compress them via cron?

Re: [Dovecot] zlib config questions

[Timo Sirainen]

On 6.12.2013, at 21.28, Sven Hartge  wrote:

> And, please anybody correct me if I am wrong, if you use imap_zlib then
> the IMAP connection is compressed as well which may allow Dovecot to
> skip the decompressing state completely and just deliver the compressed
> stream directly. (Althought I don't know anymore where I read this and
> may be totally wrong and bogus. Please confirm with anyone knowing the
> code. Thanks.)

No, that’s not done. I don’t think it would be a good idea either.

Re: [Dovecot] zlib config questions

[Sven Hartge]

Terry Barnum  wrote:

> I'm curious how many people here use the zlib plugin. Is it common or
> is drive space cheap enough that it's not worth it to run the risk of
> potential problems?

HDD space may be cheap, but is also slow, therefor IOPS are very
expensive. 

I compress the mail using zlib, because even a very slight compression
rate as 3 reduces the IOPS by more than 50% in my measurements and in my
environment. As long as you don't plan to use a Raspberry Pi as your
Dovecot server, todays CPUs are easily able to cope.

And, please anybody correct me if I am wrong, if you use imap_zlib then
the IMAP connection is compressed as well which may allow Dovecot to
skip the decompressing state completely and just deliver the compressed
stream directly. (Althought I don't know anymore where I read this and
may be totally wrong and bogus. Please confirm with anyone knowing the
code. Thanks.)

Also reducing the needed space is a welcome bonus for me. 

Grüße,
Sven,.

-- 
Sigmentation fault. Core dumped.

Re: [Dovecot] zlib config questions

[Robert Schetterer]

Am 06.12.2013 20:18, schrieb Robert Schetterer:
> Am 06.12.2013 20:01, schrieb Terry Barnum:
>> Once you enable zlib, does this mean you must use it from that point on? 
>> Once email is compressed, if you disable the plugin does dovecot lose the 
>> ability to access those compressed emails?
> 
> you may disable the plugin anytime, but you then have to decompress all
> mails, which should easy work with dsync
> 
> 
> http://wiki2.dovecot.org/Tools/Dsync
> 
> specially look at -o options
> 
> 

forgot this

http://sys4.de/de/blog/2013/07/20/fallstricke-mit-acl-und-quota-bei-backup-mit-dsync-dovecot/

> 
> sorry german only
> 
> Best Regards
> MfG Robert Schetterer
> 



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] zlib config questions

[Robert Schetterer]

Am 06.12.2013 20:01, schrieb Terry Barnum:
> Once you enable zlib, does this mean you must use it from that point on? Once 
> email is compressed, if you disable the plugin does dovecot lose the ability 
> to access those compressed emails?

you may disable the plugin anytime, but you then have to decompress all
mails, which should easy work with dsync


http://wiki2.dovecot.org/Tools/Dsync

specially look at -o options



sorry german only

Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] zlib config questions

[Terry Barnum]

On Dec 6, 2013, at 8:23 AM, Robert Schetterer  wrote:

> Am 06.12.2013 17:01, schrieb Charles Marcus:
>> On 2013-12-06 10:55 AM, Richard Platel  wrote:
>>> One gotcha if you're using maildir is, if you're planning to compress
>>> old mail, make sure that the S= and W= sizes in the filenames are
>>> correct,
>> 
>> And how do you make sure of this? Is there a script? Or do you just pray?
> 
> this is only if you manual convert the whole store i.e with dsync to get
> more free space , if you simple enable the plugin you have not be afraid
> cause only new mail will may get compressed
> but however zlib and maildir had bugs in some dove releases or version
> updates, see list archive how people fixed it


Thank you Richard, Charles and Robert for your insight and suggestions.

I'm curious how many people here use the zlib plugin. Is it common or is drive 
space cheap enough that it's not worth it to run the risk of potential problems?

I am using Maildir and did read the threads from last year of problems with the 
manually compressed mail S and W values being incorrect as well as other 
filename problems. Have these been fixed in dovecot >= v2.2.5?

Once you enable zlib, does this mean you must use it from that point on? Once 
email is compressed, if you disable the plugin does dovecot lose the ability to 
access those compressed emails?

Thanks,
-Terry

Terry Barnum
digital OutPost
http://www.dop.com

Re: [Dovecot] zlib config questions

[Robert Schetterer]

Am 06.12.2013 17:01, schrieb Charles Marcus:
> On 2013-12-06 10:55 AM, Richard Platel  wrote:
>> One gotcha if you're using maildir is, if you're planning to compress
>> old mail, make sure that the S= and W= sizes in the filenames are
>> correct,
> 
> And how do you make sure of this? Is there a script? Or do you just pray?
> 

this is only if you manual convert the whole store i.e with dsync to get
more free space , if you simple enable the plugin you have not be afraid
cause only new mail will may get compressed
but however zlib and maildir had bugs in some dove releases or version
updates, see list archive how people fixed it


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] zlib config questions

[Charles Marcus]

On 2013-12-06 10:55 AM, Richard Platel  wrote:

One gotcha if you're using maildir is, if you're planning to compress old mail, 
make sure that the S= and W= sizes in the filenames are correct,


And how do you make sure of this? Is there a script? Or do you just pray?

--

Best regards,

*/Charles/*

Re: [Dovecot] zlib config questions

[Richard Platel]

You only need to add the zlib plugin to mail_plugins once.

As far as I know, there's no indication in the logs that mail is being 
compressed, and the filename isn't modified to indicate that it's compressed, 
but, of course, the files are zlib data on disk.

One gotcha if you're using maildir is, if you're planning to compress old mail, 
make sure that the S= and W= sizes in the filenames are correct, otherwise 
dovecot errors and closes the client session, and it can't repair the filename 
on it's own.



On Dec 6, 2013, at 12:51 AM, dovecot-requ...@dovecot.org wrote:

> Message: 6
> Date: Thu, 5 Dec 2013 13:55:40 -0800
> From: Terry Barnum 
> To: Dovecot Mailing List 
> Subject: [Dovecot] zlib config questions
> Message-ID: 
> Content-Type: text/plain; charset=us-ascii
> 
> After nearly running out of space I swapped in larger disks and then saw the 
> recent threads about zlib compression. Unfortunately I'm still confused after 
> reading <http://wiki2.dovecot.org/Plugins/Zlib>.
> 
> In order to compress new email being stored do I only need to change
> 
> 10-mail.conf to this:
> mail_plugins = $mail_plugins zlib
> 
> and 90-plugin.conf to this:
> plugin {
>  zlib_save_level = 6
>  zlib_save = gz
>  ...
> 
> Or do I need to instead (or also?) add it to the list of plugins in 
> 20-imap.conf?
> 
> A simple 'sudo doveadm reload' to enable?
> 
> Once enabled, is there an indication in the logs that compression is taking 
> place?
> 
> Do new mail files have a suffix like .Z to indicate they've been compressed?
> 
> Any gotchas to be aware of? 
> 
> I'm running macports dovecot 2.2.5.
> 
> Thanks for any help.
> 
> -Terry
> 
> Terry Barnum
> digital OutPost
> http://www.dop.com

[Dovecot] zlib config questions

[Terry Barnum]

After nearly running out of space I swapped in larger disks and then saw the 
recent threads about zlib compression. Unfortunately I'm still confused after 
reading .

In order to compress new email being stored do I only need to change

10-mail.conf to this:
mail_plugins = $mail_plugins zlib

and 90-plugin.conf to this:
plugin {
  zlib_save_level = 6
  zlib_save = gz
  ...

Or do I need to instead (or also?) add it to the list of plugins in 
20-imap.conf?

A simple 'sudo doveadm reload' to enable?

Once enabled, is there an indication in the logs that compression is taking 
place?

Do new mail files have a suffix like .Z to indicate they've been compressed?

Any gotchas to be aware of? 

I'm running macports dovecot 2.2.5.

Thanks for any help.

-Terry

Terry Barnum
digital OutPost
http://www.dop.com

Re: [Dovecot] Zlib plugin - when does it make sense?

[Jan-Frode Myklebust]

On Mon, Nov 25, 2013 at 02:47:33PM +0100, Frerich Raabe wrote:
> 
> Interesting! What zlib compression level did you use? I figure even low
> levels would work rather well for plain text.

plugin {
zlib_save_level = 6 # 1..9
zlib_save = gz # or bz2
}

> Now that I think about "plain text": I also have the fts_solr plugin
> enabled to speed up the occasional full-text search - does the indexing
> still work as before when the mail is compressed, i.e. is the
> reading of the mail centralized so that the individual plugins don't
> actually know or care? Or would I need to make sure I use
> 'zlib-aware' plugins?

I don't have fts (yet).


  -jf

Re: [Dovecot] Zlib plugin - when does it make sense?

[Frerich Raabe]

On 2013-11-25 14:35, Jan-Frode Myklebust wrote:

On Mon, Nov 25, 2013 at 09:53:14AM +0100, Frerich Raabe wrote:


I run a small IMAP server for a dozen guys in the office, serving
about 55GB of Maildir. I recently became aware of the Zlib plugin (
http://wiki2.dovecot.org/Plugins/Zlib ) and wondered

1. given that there is about zero CPU load on my IMAP server, is
enabling the plugin a no-brainer or are there other things (except
CPU load) to consider?


Yes, it's a no-brainer. I can't remember how the cpuload was before 
we
enabled zlib, but our cpus are running 80% idle (6 servers, mix of 
IBM
x3550 and x346, serving 15TB mdbox, but was serving maildir with zlib 
a

year ago).


Interesting! What zlib compression level did you use? I figure even low
levels would work rather well for plain text.

Now that I think about "plain text": I also have the fts_solr plugin
enabled to speed up the occasional full-text search - does the indexing
still work as before when the mail is compressed, i.e. is the
reading of the mail centralized so that the individual plugins don't
actually know or care? Or would I need to make sure I use
'zlib-aware' plugins?

--
Frerich Raabe - ra...@froglogic.com
www.froglogic.com - Multi-Platform GUI Testing

Re: [Dovecot] Zlib plugin - when does it make sense?

[Jan-Frode Myklebust]

On Mon, Nov 25, 2013 at 09:53:14AM +0100, Frerich Raabe wrote:
> 
> I run a small IMAP server for a dozen guys in the office, serving
> about 55GB of Maildir. I recently became aware of the Zlib plugin (
> http://wiki2.dovecot.org/Plugins/Zlib ) and wondered
> 
> 1. given that there is about zero CPU load on my IMAP server, is
> enabling the plugin a no-brainer or are there other things (except
> CPU load) to consider?

Yes, it's a no-brainer. I can't remember how the cpuload was before we
enabled zlib, but our cpus are running 80% idle (6 servers, mix of IBM
x3550 and x346, serving 15TB mdbox, but was serving maildir with zlib a 
year ago).

> 
> 2. For enabling the plugin, I suppose you compress all the existing
> mail just once and then add 'zlib' to mail_plugins in order to have
> all future incoming mail saved?

You don't strictly need to compress existing mail. It should handle a
mix of compressed and non-compressed messages in the same maildir. 



  -jf

Re: [Dovecot] Zlib plugin - when does it make sense?

[René Neumann]

Am 25.11.2013 14:12, schrieb Everton Foscarini:
> I have a relatively big installation here in the university, 1 mail
> server for 10.000 daily active accounts.
> 
> I converted from maildir to mdbox + zlib in march, and these are my
> considerations:
> 
> 1. CPU is not a problem, even delivering more than 400 messages per
> minute (blade HS21 with 2xQuad E5430 from 2008)
> 2. I used dsync to convert from maildir to mdbox, with dovecot 2.1 at
> the time.
> 
> The biggest win is the file number reduction, from 27 million to 2 million.

But this is due to the conversion to mdbox and not the compression,
isn't it?

- René

Re: [Dovecot] Zlib plugin - when does it make sense?

[Everton Foscarini]

I have a relatively big installation here in the university, 1 mail 
server for 10.000 daily active accounts.


I converted from maildir to mdbox + zlib in march, and these are my 
considerations:


1. CPU is not a problem, even delivering more than 400 messages per 
minute (blade HS21 with 2xQuad E5430 from 2008)
2. I used dsync to convert from maildir to mdbox, with dovecot 2.1 at 
the time.


The biggest win is the file number reduction, from 27 million to 2 million.



Em 25/11/2013 06:53, Frerich Raabe escreveu:

Hi,

I run a small IMAP server for a dozen guys in the office, serving 
about 55GB of Maildir. I recently became aware of the Zlib plugin ( 
http://wiki2.dovecot.org/Plugins/Zlib ) and wondered


1. given that there is about zero CPU load on my IMAP server, is 
enabling the plugin a no-brainer or are there other things (except CPU 
load) to consider?


2. For enabling the plugin, I suppose you compress all the existing 
mail just once and then add 'zlib' to mail_plugins in order to have 
all future incoming mail saved?


Any insight by people familiar with the plugin would be much 
appreciated - thanks!

[Dovecot] Zlib plugin - when does it make sense?

[Frerich Raabe]

Hi,

I run a small IMAP server for a dozen guys in the office, serving about 
55GB of Maildir. I recently became aware of the Zlib plugin ( 
http://wiki2.dovecot.org/Plugins/Zlib ) and wondered


1. given that there is about zero CPU load on my IMAP server, is 
enabling the plugin a no-brainer or are there other things (except CPU 
load) to consider?


2. For enabling the plugin, I suppose you compress all the existing 
mail just once and then add 'zlib' to mail_plugins in order to have all 
future incoming mail saved?


Any insight by people familiar with the plugin would be much 
appreciated - thanks!


--
Frerich Raabe - ra...@froglogic.com
www.froglogic.com - Multi-Platform GUI Testing

Re: [Dovecot] zlib plugin issues

[Vijay Rajah]

Just realized the imap plugin must be imap_zlib...

It works now.

Thanks
Vijay


On Mon, Aug 26, 2013 at 10:45 PM, Vijay Rajah  wrote:

> Hello,
>
> I'm running dovecot 2.2.5 in Centos 6.4 on EXT4 fs.
>
> I have activated zlib plugin. The incoming mails is compressed. However,
> when i try to retrieve the mail using imap I'm getting this error
>
> Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error:
> Cached message size smaller than expected (1464 < 2452)
> Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error:
> Maildir filename has wrong S value, renamed the file from /mail/mailbox/
> rvijay.me/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
> to
> /mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
> Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error:
> Corrupted index cache file /mail/mailbox//me/Mai
> ldir/.vijay_gmail/dovecot.index.cache: Broken physical size for mail UID 12
>
>
> $ file
> /mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5\,S\=1464\:2\,S
> /mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,S:
> bzip2 compressed data, block size = 900k
>
> Am I missing anything? any ideas is much appreciated.
>
> Thanks
> Vijay
>
>
> ###doveconf -n 
>
> Here is my doveconf -n
>
> # 2.2.5: /mail/dovecot/2.2.5/etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32-358.14.1.el6.x86_64 x86_64 CentOS release 6.4 (Final)
> ext4
> auth_mechanisms = plain login digest-md5 cram-md5
> auth_socket_path = /mail/var/dovecot/run/auth-userdb
> base_dir = /mail/var/dovecot/run/
> debug_log_path = /var/log/dovecot-debug.log
> hostname = ma...@rvijay.me
> mail_location = maildir:/mail/mailbox/%d/%n/Maildir
> mail_plugin_dir = /mail/dovecot/2.2.5/lib/dovecot/
> mail_plugins = " virtual quota zlib"
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
> copy include variables body enotify environment mailbox date ihave
> namespace inbox {
>   inbox = yes
>   location =
>   mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
>   }
>   mailbox Junk {
> auto = subscribe
> special_use = \Junk
>   }
>   mailbox Sent {
> special_use = \Sent
>   }
>   mailbox "Sent Messages" {
> special_use = \Sent
>   }
>   mailbox Trash {
> auto = subscribe
> special_use = \Trash
>   }
>   prefix =
>   separator = /
>   type = private
> }
> passdb {
>   args = /mail/dovecot/2.2.5/etc/dovecot/conf.d/dovecot-sql.conf.ext
>   driver = sql
> }
> plugin {
>   mail_log_events = delete undelete expunge copy mailbox_delete
> mailbox_rename
>   mail_log_fields = uid box msgid size
>   quota = maildir:User quota
>   quota_grace = 10%%
>   quota_rule = *:storage=1GB
>   quota_rule2 = Trash:storage=+10%%
>   quota_rule3 = Spam:storage=+20%%
>   quota_status_nouser = DUNNO
>   quota_status_overquota = 450 4.2.2 Mailbox is full. Try again later
>   quota_status_success = DUNNO
>   quota_warning = storage=95%% quota-warning 95 %u
>   quota_warning2 = storage=80%% quota-warning 80 %u
>   quota_warning3 = -storage=100%% quota-warning below %u
>   sieve = ~/.dovecot.sieve
>   sieve_dir = ~/sieve
>   sieve_global_dir = /mail/var/dovecot/sieve
>   zlib_save = bz2
>   zlib_save_level = 9
> }
> protocols = imap lmtp sieve
> service auth {
>   unix_listener /mail/var/postfix/dovecot-auth {
> group = postfix
> mode = 0660
> user = postfix
>   }
> }
> service imap-login {
>   inet_listener imap {
> port = 143
>   }
> }
> service lmtp {
>   unix_listener /mail/var/postfix/dovecot-lmtp {
> group = postfix
> mode = 0660
> user = postfix
>   }
> }
> service managesieve-login {
>   inet_listener sieve {
> port = 4190
>   }
>   service_count = 10
>   vsz_limit = 64 M
> }
> service managesieve {
>   process_limit = 1024
> }
> service quota-status {
>   client_limit = 1
>   executable = quota-status -p postfix
>   inet_listener {
> port = 12340
>   }
> }
> service quota-warning {
>   executable = script
> /mail/dovecot/2.2.5/etc/dovecot/conf.d/quota-warning.sh
>   user = vmail
> }
> shutdown_clients = no
> ssl_ca =  ssl_cert =  ssl_key =  ssl_parameters_regenerate = 30 hours
> submission_host = 127.0.0.1:587
> userdb {
>   driver = prefetch
> }
> userdb {
>   args = /mail/dovecot/2.2.5/etc/dovecot/conf.d/dovecot-sql.conf.ext
>   driver = sql
> }
> protocol lda {
>   mail_plugins = " virtual quota sieve zlib"
> }
> protocol imap {
>   mail_plugins = " virtual quota imap_quota zlib"
> }
> protocol lmtp {
>   mail_plugins = " virtual quota sieve zlib"
> }
>
>
>

[Dovecot] zlib plugin issues

[Vijay Rajah]

Hello,

I'm running dovecot 2.2.5 in Centos 6.4 on EXT4 fs.

I have activated zlib plugin. The incoming mails is compressed. However,
when i try to retrieve the mail using imap I'm getting this error

Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error: Cached
message size smaller than expected (1464 < 2452)
Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error:
Maildir filename has wrong S value, renamed the file from /mail/mailbox/
rvijay.me/me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
to
/mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,
Aug 25 22:17:31 vrajah-vps5 dovecot: imap(me@): Error:
Corrupted index cache file /mail/mailbox//me/Mai
ldir/.vijay_gmail/dovecot.index.cache: Broken physical size for mail UID 12


$ file
/mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5\,S\=1464\:2\,S
/mail/mailbox//me/Maildir/.vijay_gmail/cur/1377448276.M960004P32378.vrajah-vps5,S=1464:2,S:
bzip2 compressed data, block size = 900k

Am I missing anything? any ideas is much appreciated.

Thanks
Vijay


###doveconf -n 

Here is my doveconf -n

# 2.2.5: /mail/dovecot/2.2.5/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-358.14.1.el6.x86_64 x86_64 CentOS release 6.4 (Final)
ext4
auth_mechanisms = plain login digest-md5 cram-md5
auth_socket_path = /mail/var/dovecot/run/auth-userdb
base_dir = /mail/var/dovecot/run/
debug_log_path = /var/log/dovecot-debug.log
hostname = ma...@rvijay.me
mail_location = maildir:/mail/mailbox/%d/%n/Maildir
mail_plugin_dir = /mail/dovecot/2.2.5/lib/dovecot/
mail_plugins = " virtual quota zlib"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Junk {
auto = subscribe
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
  type = private
}
passdb {
  args = /mail/dovecot/2.2.5/etc/dovecot/conf.d/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
  mail_log_fields = uid box msgid size
  quota = maildir:User quota
  quota_grace = 10%%
  quota_rule = *:storage=1GB
  quota_rule2 = Trash:storage=+10%%
  quota_rule3 = Spam:storage=+20%%
  quota_status_nouser = DUNNO
  quota_status_overquota = 450 4.2.2 Mailbox is full. Try again later
  quota_status_success = DUNNO
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  quota_warning3 = -storage=100%% quota-warning below %u
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  sieve_global_dir = /mail/var/dovecot/sieve
  zlib_save = bz2
  zlib_save_level = 9
}
protocols = imap lmtp sieve
service auth {
  unix_listener /mail/var/postfix/dovecot-auth {
group = postfix
mode = 0660
user = postfix
  }
}
service imap-login {
  inet_listener imap {
port = 143
  }
}
service lmtp {
  unix_listener /mail/var/postfix/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  service_count = 10
  vsz_limit = 64 M
}
service managesieve {
  process_limit = 1024
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  inet_listener {
port = 12340
  }
}
service quota-warning {
  executable = script
/mail/dovecot/2.2.5/etc/dovecot/conf.d/quota-warning.sh
  user = vmail
}
shutdown_clients = no
ssl_ca = 

Re: [Dovecot] zlib error when running doveadm

[Timo Sirainen]

On 22.7.2013, at 18.01, Michael Long  wrote:

> Upgraded to dovecot 2.2.4 from 2.1.16
> 
> Error I see is:
> 
> doveadm(root): Error: Module is for different ABI version 2.1.16 (we have 
> 2.2.ABIv3(2.2.4)): /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so
> 
> How do I resolve this?

Delete that file. v2.2 no longer has doveadm_zlib plugin, because it's built in 
(to lib-compression).

[Dovecot] zlib error when running doveadm

[Michael Long]

I manually fixed it by copying src/plugins/zlib/lib20_zlib_plugin.la to 
/usr/local/lib/dovecot/doveadm


I do not understand why 'make install' won't do that? A bug perhaps?

[Dovecot] zlib error when running doveadm

[Michael Long]

> Looks like you have two versions installed and are mixing them up.

I checked and only one binary is on the system. I took a look in the 
directory /usr/local/lib/dovecot/doveadm/ and it appears the upgrade 
rebuilt all libraries EXCEPT zlib. Any ideas?


-rw-r--r-- 1 root root 125910 Jul 22 10:58 lib10_doveadm_acl_plugin.a
-rwxr-xr-x 1 root root   1070 Jul 22 10:58 lib10_doveadm_acl_plugin.la
-rwxr-xr-x 1 root root  88190 Jul 22 10:58 lib10_doveadm_acl_plugin.so
-rw-r--r-- 1 root root  54812 Jul 22 10:58 lib10_doveadm_expire_plugin.a
-rwxr-xr-x 1 root root   1091 Jul 22 10:58 lib10_doveadm_expire_plugin.la
-rwxr-xr-x 1 root root  44763 Jul 22 10:58 lib10_doveadm_expire_plugin.so
-rw-r--r-- 1 root root  89116 Jul 22 10:58 lib10_doveadm_quota_plugin.a
-rwxr-xr-x 1 root root   1084 Jul 22 10:58 lib10_doveadm_quota_plugin.la
-rwxr-xr-x 1 root root  61137 Jul 22 10:58 lib10_doveadm_quota_plugin.so
-rw-r--r-- 1 root root 369576 Jul 22 10:47 lib10_doveadm_sieve_plugin.a
-rwxr-xr-x 1 root root   1248 Jul 22 10:47 lib10_doveadm_sieve_plugin.la
-rwxr-xr-x 1 root root 208863 Jul 22 10:47 lib10_doveadm_sieve_plugin.so
-rw-r--r-- 1 root root  26166 Apr  9 14:30 lib10_doveadm_zlib_plugin.a
-rwxr-xr-x 1 root root   1077 Apr  9 14:30 lib10_doveadm_zlib_plugin.la
-rwxr-xr-x 1 root root  25243 Apr  9 14:30 lib10_doveadm_zlib_plugin.so
-rw-r--r-- 1 root root 100120 Jul 22 10:58 lib20_doveadm_fts_plugin.a
-rwxr-xr-x 1 root root   1070 Jul 22 10:58 lib20_doveadm_fts_plugin.la
-rwxr-xr-x 1 root root  65515 Jul 22 10:58 lib20_doveadm_fts_plugin.so


Configure does seem to see it though:

checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking bzlib.h usability... yes
checking bzlib.h presence... yes
checking for bzlib.h... yes
checking for BZ2_bzdopen in -lbz2... yes

Re: [Dovecot] zlib error when running doveadm

[Robert Schetterer]

Am 22.07.2013 17:01, schrieb Michael Long:
> doveadm(root): Error: Module is for different ABI version 2.1.16 (we
> have 2.2.ABIv3(2.2.4)):
> /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so

normally this simple says there is somthing from old dovecot version (
process ) still running, try stop all dove, reinstall again ( might use
force etc )



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] zlib error when running doveadm

[Charles Marcus]

On 2013-07-22 11:01 AM, Michael Long  wrote:

Configuration:
# 2.2.4: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-48-generic x86_64 Ubuntu 12.04.2 LTS
doveconf: Warning: Dovecot was last started using 
/etc/dovecot/dovecot-postfix.conf, but this config is 
/usr/local/etc/dovecot/dovecot.conf 


Looks like you have two versions installed and are mixing them up.

--

Best regards,

Charles Marcus
I.T. Director
Media Brokers International, Inc.
678.514.6224 | 678.514.6299 fax

[Dovecot] zlib error when running doveadm

[Michael Long]

Upgraded to dovecot 2.2.4 from 2.1.16

Error I see is:

doveadm(root): Error: Module is for different ABI version 2.1.16 (we 
have 2.2.ABIv3(2.2.4)): 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so


How do I resolve this?

Configure options:
./configure --with-bzlib --with-ssl --with-mysql --with-zlib --with-pam

Configuration:
# 2.2.4: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-48-generic x86_64 Ubuntu 12.04.2 LTS
doveconf: Warning: Dovecot was last started using 
/etc/dovecot/dovecot-postfix.conf, but this config is 
/usr/local/etc/dovecot/dovecot.conf

auth_mechanisms = plain login cram-md5
disable_plaintext_auth = no
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_access_groups = mail
mail_debug = yes
mail_gid = 8
mail_location = maildir:~/Maildir
mail_plugins = " zlib"
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date

passdb {
  args = /etc/dovecot/passdb
  driver = passwd-file
}
passdb {
  driver = pam
}
plugin {
  fts = squat
  sieve = ~/.dovecot.sieve
  sieve_debug = yes
  sieve_dir = ~/
  sieve_global_dir = /etc/dovecot
  sieve_global_path = /etc/dovecot/dovecot.sieve
  zlib_save = gz
  zlib_save_level = 6
}
protocols = imap pop3 sieve
service auth {
  unix_listener /var/run/dovecot/auth-master {
group = postfix
mode = 0666
user = postfix
  }
  unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
  }
  user = root
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  inet_listener sieve_deprecated {
port = 2000
  }
}
ssl_cert = ssl_cipher_list = 
ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM

ssl_key =  was automatically rejected:%n%r
}

Re: [Dovecot] zlib plugin "undefined symbol: imap_module_register"

[Axel Luttgens]

Le 19 juil. 2013 à 15:43, Alexander Weber a écrit :

> Hi all,
> 
> i've tried to enable the zlib plugin within die LDA, but i've got some 
> strange errors
> 
> dovecot: lda: Error: 
> dlopen(/usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so) failed: 
> /usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so: undefined symbol: 
> imap_module_register
> dovecot: lda: Fatal: Couldn't load required plugins
> 
> 
> [...]
> protocol lda {
>  deliver_log_format = msgid=%m: %$
>  mail_plugins = sieve, zlib, acl, imap_zlib
>  postmaster_address = postmaster
>  quota_full_tempfail = yes
>  rejection_reason = Your message to <%t> was automatically rejected:%n%r
> }

Hello Alexander,

Should the imap_zlib plugin be listed in the above?

Axel

[Dovecot] zlib plugin "undefined symbol: imap_module_register"

[Alexander Weber]

Hi all,

i've tried to enable the zlib plugin within die LDA, but i've got some 
strange errors


dovecot: lda: Error: 
dlopen(/usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so) failed: 
/usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so: undefined symbol: 
imap_module_register

dovecot: lda: Fatal: Couldn't load required plugins


am i missing something within the config?

doveconf -n

# 2.0.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-49-generic x86_64 Ubuntu 12.04.2 LTS
base_dir = /var/run/dovecot/
disable_plaintext_auth = no
hostname = mail.local
mail_location = maildir:~/Maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave

namespace {
  list = yes
  location = 
maildir:/home/shared:CONTROL=~/.Maildir/control/Shared:INDEX=~/.Maildir/index/Shared

  prefix = shared/
  separator = /
  subscriptions = yes
  type = public
}
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
  type = private
}
passdb {
  driver = pam
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  zlib_save = gz
  zlib_save_level = 6
}
postmaster_address = postmaster@localhost
protocols = imap sieve
service auth {
  unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0660
user = postfix
  }
}
service imap-login {
  inet_listener imaps {
port = 993
ssl = yes
  }
}
ssl_ca =  was automatically rejected:%n%r
}
protocol imap {
  mail_plugins = zlib, acl, imap_zlib
}



Mit freundlichen Gruessen / best regards
papaya Software GmbH

i.A. Alexander Weber

--
papaya Software GmbH | Im MediaPark 5 | 50670 Koeln | Germany
Tel./Ph.: +49-221-5743-8070 | Fax: +49-221-5743-8099
mailto:we...@papaya-cms.com | http://www.papaya-cms.com/
--
Geschaeftsfuehrer: Andreas Jacobi, André Schnitzler, Daniel Schäfer
Sitz&  Registergericht: Koeln | HRB 60030 | USt.-Id.-Nr.: DE 255642963
--

[Dovecot] Zlib plugin: changing compression save level

[Urban Loesch]

Hi,

I have running dovecot 2.1.15 with zlib plugin enabled and "zlib_save_level = 
6" since about 2 years now
without any problems.

What happens if I now change the zlib_save_level to 9?
Should that work without any problems, or become the current saved "*.m" files
incompatible or unreadable?

Thanks
Urban

Re: [Dovecot] Zlib plugin: Compress emails for one folder only

[André Rodier]

Thanks Timo,

I will see. If I have some time, I may have a look at the source code.

Kind regards,
André.

On 20 March 2013 17:58, Timo Sirainen  wrote:
> On 16.3.2013, at 14.43, Andre Rodier  wrote:
>
>> Hello everyone,
>>
>> Is it possible to configure the zlib plugin to compress the emails in one 
>> folder only?
>>
>> I would like to compress the emails when they are stored in my 'Archives' 
>> folders and sub folders.
>
> Nope. With hardcoding it would be a simple one line of code addition.
>
> Some day in future there will be namespace-specific settings. Then you could 
> create Archive namespace and enable zlib compression only for that namespace.
>



-- 
André Rodier

Re: [Dovecot] Zlib plugin: Compress emails for one folder only

[Timo Sirainen]

On 16.3.2013, at 14.43, Andre Rodier  wrote:

> Hello everyone,
> 
> Is it possible to configure the zlib plugin to compress the emails in one 
> folder only?
> 
> I would like to compress the emails when they are stored in my 'Archives' 
> folders and sub folders.

Nope. With hardcoding it would be a simple one line of code addition.

Some day in future there will be namespace-specific settings. Then you could 
create Archive namespace and enable zlib compression only for that namespace.

[Dovecot] Zlib plugin: Compress emails for one folder only

[Andre Rodier]

Hello everyone,

Is it possible to configure the zlib plugin to compress the emails in one 
folder only?

I would like to compress the emails when they are stored in my 'Archives' 
folders and sub folders.

Thanks,
André Rodier.

Re: [Dovecot] zlib plugin bug?

[Jan Phillip Greimann]

Am 11.03.2013 08:05, schrieb Angel L. Mateo:

zlib plugin doesn't put any flag in messages. The Z flag mentioned
in the wiki is one you could put if your are compressing an existing
mailbox in order to know which files you have previously compressed.


Thank you. I noticed later that there is no Z-flag by default for 
compressed mails. I think it should be added to the plugin, but maybe it 
isn't possible.


Greetings, Jan

Re: [Dovecot] zlib plugin bug?

[Angel L. Mateo]

El 08/03/13 14:13, Jan Phillip Greimann escribió:

Hi there,
got a problem with the zlib plugin, just wanted to test it on a
test-mailserver, configured like in the wiki
(http://wiki2.dovecot.org/Plugins/Zlib) and restarted the server.

I've send an email to a blank maildir++ mailbox, the file is saved as
gzip compressed file, but the "Z" flag is missing in the filename.

	zlib plugin doesn't put any flag in messages. The Z flag mentioned in 
the wiki is one you could put if your are compressing an existing 
mailbox in order to know which files you have previously compressed.


--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información
y las Comunicaciones Aplicadas (ATICA)
http://www.um.es/atica
Tfo: 868889150
Fax: 86337

[Dovecot] zlib plugin bug?

[Jan Phillip Greimann]

Hi there,
got a problem with the zlib plugin, just wanted to test it on a 
test-mailserver, configured like in the wiki 
(http://wiki2.dovecot.org/Plugins/Zlib) and restarted the server.


I've send an email to a blank maildir++ mailbox, the file is saved as 
gzip compressed file, but the "Z" flag is missing in the filename.


Is that a bug in the plugin, or in dovecot?

Greetings, Jan

-
--- additional data:

>>> find .
./dovecot-uidlist
./dovecot.mailbox.log
./tmp
./cur
./cur/1362746783.M97516P28534.s16,S=1321,W=1352:2,Sa
./maildirsize
./new
./dovecot.index.cache
./dovecot-keywords
./dovecot-uidvalidity.5139a483
./subscriptions
./dovecot.index.log
./dovecot-uidvalidity
./dovecot-acl-list

>>> file ./cur/1362746783.M97516P28534*
./cur/1362746783.M97516P28534.s16,S=1321,W=1352:2,Sa: gzip compressed 
data, from Unix



>>> dpkg -l |grep dove
ii  debian-dovecot-auto-keyring 2010.01.30 
GnuPG archive keys of the Automatic Dovecot Debian repository
ii  dovecot-common  2:2.2.0~rc2-0~auto+23 
Transitional package for dovecot
ii  dovecot-core2:2.2.0~rc2-0~auto+23 
secure mail server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-imapd   2:2.2.0~rc2-0~auto+23 
secure IMAP server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-lmtpd   2:2.2.0~rc2-0~auto+23 
secure LMTP server for Dovecot
ii  dovecot-managesieved2:2.2.0~rc2-0~auto+23 
secure ManageSieve server for Dovecot
ii  dovecot-mysql   2:2.2.0~rc2-0~auto+23 
MySQL support for Dovecot
ii  dovecot-pop3d   2:2.2.0~rc2-0~auto+23 
secure POP3 server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-sieve   2:2.2.0~rc2-0~auto+23 
sieve filters support for Dovecot


>>> doveconf -n
# 2.2.rc2 (52e5d4186006): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.7 ext3
auth_debug = yes
auth_verbose = yes
listen = *
mail_debug = yes
mail_gid = vmail
mail_location = maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs
mail_plugins = quota acl zlib
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  acl = vfile
  acl_anyone = allow
  autocreate = Trash
  autocreate2 = Drafts
  autocreate3 = Sent
  autocreate5 = Archives
  autosubscribe = Trash
  autosubscribe2 = Drafts
  autosubscribe3 = Sent
  quota = maildir:User quota
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  zlib_save = gz
  zlib_save_level = 6
}
protocols = " imap lmtp sieve pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
  }
}
service dict {
  idle_kill = 1 hours
  unix_listener dict {
group = vmail
mode = 0660
  }
}
service imap-login {
  inet_listener imap {
port = 0
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  inet_listener sieve_deprecated {
port = 2000
  }
}
service pop3-login {
  inet_listener pop3 {
port = 0
  }
}
ssl_cert = 

Re: [Dovecot] Zlib maildir reindex bug?

[Robert Schetterer]

Am 14.01.2013 20:12, schrieb Richard Platel:
> Maildir filename has wrong S value

may not related to index only , it sometimes comes up by upgrading
from 2.0x to 2.1x versions

look

http://www.dovecot.org/list/dovecot/2012-March/064211.html


some soltuions may

try set

maildir_broken_filename_sizes=yes

upgrade dovecot latest version 2.1x ( patches were released )

decompress all compressed mails by script etc



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

[Dovecot] Zlib maildir reindex bug?

[Richard Platel]

Hi

Running dovecot 2.1.7 (from debian repo).

We have some old compressed maildir messages with the wrong S= size in the 
filename (our fault).

If I delete index files, log in to the mailbox and try to FETCH the bad 
message, dovecot complains about the incorrect message size, attempts to fix 
the filename and disconnects the client.  However, it changes the filesize to 
the size of the message on disk, not the message's uncompressed size.

For subsequent clients, the FETCH works, but dovecot logs an error (and maybe 
re-indexes the mailbox?) and the message UID changes.


Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Cached 
message size larger than expected (1612 > 1556)
Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Maildir 
filename has wrong S value, renamed the file from 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=1612:2,Sab to 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab
Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted 
index cache file /mail/index01/505/236/b...@confidence.com/.Sent 
Items/dovecot.index.cache: Broken physical size for mail UID 1
Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: 
read(/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=1612:2,Sab) failed: 
Input/output error (uid=1)


Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Cached 
message size smaller than expected (931 < 1556)
Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Maildir 
filename has wrong S value, renamed the file from 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab
Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted 
index cache file /mail/index01/505/236/b...@confidence.com/.Sent 
Items/dovecot.index.cache: Broken physical size for mail UID 2
Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Cached 
message size smaller than expected (931 < 1556)
Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Maildir 
filename has wrong S value, renamed the file from 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab
Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted 
index cache file /mail/index01/505/236/b...@confidence.com/.Sent 
Items/dovecot.index.cache: Broken physical size for mail UID 2


Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Cached 
message size smaller than expected (931 < 1556)
Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Maildir 
filename has wrong S value, renamed the file from 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab
Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted 
index cache file /mail/index01/505/236/b...@confidence.com/.Sent 
Items/dovecot.index.cache: Broken physical size for mail UID 2
Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Cached 
message size smaller than expected (931 < 1556)
Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Maildir 
filename has wrong S value, renamed the file from 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to 
/mail/mailstore01/505/236/b...@confidence.com/.Sent 
Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab
Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted 
index cache file /mail/index01/505/236/b...@confidence.com/.Sent 
Items/dovecot.index.cache: Broken physical size for mail UID 2

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?

[Robert Schetterer]

Am 03.08.2012 15:56, schrieb Timo Sirainen:
> On 2.8.2012, at 10.09, Robert Schetterer wrote:
> 
>> Hi Timo, most of the Problems are fixed now
>> many thx for your script
>>
>> , but how to avoid that in future ( Maildir filename has wrong S value )
> 
> You'd need to find out why it started happening in the first place. 

the problem is/was getting maildirs fixed, no time to debug on
production system

in deed i have no idea why this happend, i found out some broken
were old maildirs, so for them i bet coruption by migration 2.0 -2.1
other cuted out attachments imap online with serveral clients on = user
fault in my meaning
left the cases which i reproduced myself, or some brand new mailboxes
it was about compressed mail and nocompressed with wrong message size in
one folder

What software wrote those filenames? Are new filenames still creating
with wrong S value?
Does your dovecot.conf use zlib plugin globally (instead of only in some
protocol sections, which might cause this kind of trouble)?


10-mail.conf:mail_plugins = $mail_plugins zlib expire virtual fts
fts_lucene mail_log autocreate notify acl listescape
20-imap.conf:  mail_plugins = imap_acl acl fts fts_lucene mail_log
autocreate notify expire virtual quota imap_quota zlib imap_zlib listescape
90-plugin.conf:zlib_save_level = 6
90-plugin.conf:zlib_save = gz

should be ok ?

> 
>> you released a patch
>>
>> diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c
>> --- a/src/lib-storage/index/istream-mail.c   Wed Mar 21 14:25:12 2012 +0200
>> +++ b/src/lib-storage/index/istream-mail.c   Wed Mar 21 17:58:00 2012 +0200
>> @@ -20,6 +20,7 @@
>>  struct mail *mail = mstream->mail;
>>  enum mail_lookup_abort orig_lookup_abort;
>>
>> +return FALSE;
>>  if (mstream->expected_size != (uoff_t)-1)
>>  return TRUE;
>>
>> would it be a good idea to include it in 2.1.9 for me ?
>> as it looks like its not in the public code ?
> 
> It's an ugly workaround for the wrong S=size problem when zlib is used. 
> Preferably you wouldn't ever need to use it.

ok , i fixed all mailboxes with your script and one from ralf with
compressed maildirs, updated to 2.1.9, using lucene now, disabled stats
( for kernel bugs ), and for now all problems are gone

if i cant find any better reason, i go for shit happens, and learned how
to fix it, however it were only 6 maildirs from 200 which got broken
going trough lot of updates in years , so this is ok in my mind

good time for users learn to build some mail archive from time to time
however no mail got lost

> 


-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?

[Timo Sirainen]

On 2.8.2012, at 10.09, Robert Schetterer wrote:

> Hi Timo, most of the Problems are fixed now
> many thx for your script
> 
> , but how to avoid that in future ( Maildir filename has wrong S value )

You'd need to find out why it started happening in the first place. What 
software wrote those filenames? Are new filenames still creating with wrong S 
value? Does your dovecot.conf use zlib plugin globally (instead of only in some 
protocol sections, which might cause this kind of trouble)?

> you released a patch
> 
> diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c
> --- a/src/lib-storage/index/istream-mail.cWed Mar 21 14:25:12 2012 +0200
> +++ b/src/lib-storage/index/istream-mail.cWed Mar 21 17:58:00 2012 +0200
> @@ -20,6 +20,7 @@
>   struct mail *mail = mstream->mail;
>   enum mail_lookup_abort orig_lookup_abort;
> 
> + return FALSE;
>   if (mstream->expected_size != (uoff_t)-1)
>   return TRUE;
> 
> would it be a good idea to include it in 2.1.9 for me ?
> as it looks like its not in the public code ?

It's an ugly workaround for the wrong S=size problem when zlib is used. 
Preferably you wouldn't ever need to use it.

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected / with 2.1.9 patch ?

[Robert Schetterer]

Am 01.08.2012 14:58, schrieb Timo Sirainen:
> On 1.8.2012, at 15.43, Robert Schetterer wrote:
> 
 If you don't mind the messages' IMAP UIDs changing, you could give the -f 
 parameter to fix those.
>>>
>>> i will try things couldn not get more worst
>>
>> ./maildir-size-check.sh -f maildir is an unknown option...?
> 
> Oh, looks like I forgot to re-upload it after adding that option. It's there 
> now.
> 

Hi Timo, most of the Problems are fixed now
many thx for your script

, but how to avoid that in future ( Maildir filename has wrong S value )

with Ralf

you released a patch

diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c
--- a/src/lib-storage/index/istream-mail.c  Wed Mar 21 14:25:12 2012 +0200
+++ b/src/lib-storage/index/istream-mail.c  Wed Mar 21 17:58:00 2012 +0200
@@ -20,6 +20,7 @@
struct mail *mail = mstream->mail;
enum mail_lookup_abort orig_lookup_abort;

+   return FALSE;
if (mstream->expected_size != (uoff_t)-1)
return TRUE;

would it be a good idea to include it in 2.1.9 for me ?
as it looks like its not in the public code ?

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 01.08.2012 14:58, schrieb Timo Sirainen:
> On 1.8.2012, at 15.43, Robert Schetterer wrote:
> 
 If you don't mind the messages' IMAP UIDs changing, you could give the -f 
 parameter to fix those.
>>>
>>> i will try things couldn not get more worst
>>
>> ./maildir-size-check.sh -f maildir is an unknown option...?
> 
> Oh, looks like I forgot to re-upload it after adding that option. It's there 
> now.
> 

looks like working fine, but it has some problems with pathnames with
special char and spaces

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Timo Sirainen]

On 1.8.2012, at 15.43, Robert Schetterer wrote:

>>> If you don't mind the messages' IMAP UIDs changing, you could give the -f 
>>> parameter to fix those.
>> 
>> i will try things couldn not get more worst
> 
> ./maildir-size-check.sh -f maildir is an unknown option...?

Oh, looks like I forgot to re-upload it after adding that option. It's there 
now.

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 01.08.2012 14:25, schrieb Robert Schetterer:
> Am 01.08.2012 14:17, schrieb Timo Sirainen:
>> On 1.8.2012, at 15.12, Robert Schetterer wrote:
>>
>>> Am 01.08.2012 14:06, schrieb Timo Sirainen:
 See what you get by running the maildir through:
 http://dovecot.org/tools/maildir-size-check.sh


>>>
>>> tons of i.e
>>>
>>> Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong
>>> S=2647728 value, should be S=3838835
>>
>> If you don't mind the messages' IMAP UIDs changing, you could give the -f 
>> parameter to fix those.
> 
> i will try things couldn not get more worst
>>
>> (There is also http://dovecot.org/tools/maildir-size-fix.pl which I of 
>> course noticed only after writing the whole script. That perl script should 
>> preferably be modified to support compressed mails 
> 
> jep compressed is a must have
> 
> and do those other checks, since it updates the uidlists and doesn't
> cause UIDs to change.)
>>
> 
> i will report next using script
> 

 ./maildir-size-check.sh -f maildir is an unknown option...?

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 01.08.2012 14:17, schrieb Timo Sirainen:
> On 1.8.2012, at 15.12, Robert Schetterer wrote:
> 
>> Am 01.08.2012 14:06, schrieb Timo Sirainen:
>>> See what you get by running the maildir through:
>>> http://dovecot.org/tools/maildir-size-check.sh
>>>
>>>
>>
>> tons of i.e
>>
>> Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong
>> S=2647728 value, should be S=3838835
> 
> If you don't mind the messages' IMAP UIDs changing, you could give the -f 
> parameter to fix those.

i will try things couldn not get more worst
> 
> (There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course 
> noticed only after writing the whole script. That perl script should 
> preferably be modified to support compressed mails 

jep compressed is a must have

and do those other checks, since it updates the uidlists and doesn't
cause UIDs to change.)
> 

i will report next using script

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Timo Sirainen]

On 1.8.2012, at 15.12, Robert Schetterer wrote:

> Am 01.08.2012 14:06, schrieb Timo Sirainen:
>> See what you get by running the maildir through:
>> http://dovecot.org/tools/maildir-size-check.sh
>> 
>> 
> 
> tons of i.e
> 
> Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong
> S=2647728 value, should be S=3838835

If you don't mind the messages' IMAP UIDs changing, you could give the -f 
parameter to fix those.

(There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course 
noticed only after writing the whole script. That perl script should preferably 
be modified to support compressed mails and do those other checks, since it 
updates the uidlists and doesn't cause UIDs to change.)

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 01.08.2012 14:06, schrieb Timo Sirainen:
> See what you get by running the maildir through:
> http://dovecot.org/tools/maildir-size-check.sh
> 
> 

tons of i.e

Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong
S=2647728 value, should be S=3838835

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Timo Sirainen]

See what you get by running the maildir through:
http://dovecot.org/tools/maildir-size-check.sh

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 01.08.2012 07:36, schrieb Robert Schetterer:
> Am 31.07.2012 23:44, schrieb Timo Sirainen:
>> On 1.8.2012, at 0.29, Robert Schetterer wrote:
>>
>>> Cached message size smaller than expected
>>> and
>>> Maildir filename has wrong S value
>>> and
>>> Corrupted index cache file
>>>
>>> in dove 2.1.8
>>>
>>> i can reproduce the problem with tb 14
>>> with 2 nat machines connected to the same server
>>>
>>> the problem relates with create new folder then copy
>>> some uncompressed and compressed mail to it, the i.e
>>> open it by other client
>>>
>>> sometime this happens at once , sometime after a few copy actions
>>
>> I can't reproduce this. Are you sure the problem isn't that the old mails 
>> already have a wrong S value and the errors happen when Dovecot tries to 
>> read them? 
> 
> at my recent onwledge
> mixing  compressed new mails from my sent folder, and older uncompressed
> ones ( which are shown nice both in tb ), with some copy action in a new
> folder from another client leads to wrong S
> and tb does not display them any longer sometime, perhaps a double zip
> happens?
> 
> It would help to know how to reproduce this on an entirely empty mail
> account.
>>
> 
> i will try to test, but first i have to deal with now broken maildirs
> i try disable zlip plugin now ,as workaround hopefully this does not
> lead to more problems on old maildirs , and fix the problem with new mail
> 

hi Timo, rolled back all stuff you recommended offlist
so situation is a bit better now, however why, i had to handle
cause more mailldirs get broken

i guess iam dealing with equal
Problems described here

http://www.mail-archive.com/dovecot@dovecot.org/msg43986.html

conversation with ralf, looks like your patch wasnt enough

i have a few mailboxes and folders now i cant fix
compressed files stay unreadable after move copy

by Maildir filename has wrong S value

i need some kind of mass repair actions etc , or some new patch
-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Am 31.07.2012 23:44, schrieb Timo Sirainen:
> On 1.8.2012, at 0.29, Robert Schetterer wrote:
> 
>> Cached message size smaller than expected
>> and
>> Maildir filename has wrong S value
>> and
>> Corrupted index cache file
>>
>> in dove 2.1.8
>>
>> i can reproduce the problem with tb 14
>> with 2 nat machines connected to the same server
>>
>> the problem relates with create new folder then copy
>> some uncompressed and compressed mail to it, the i.e
>> open it by other client
>>
>> sometime this happens at once , sometime after a few copy actions
> 
> I can't reproduce this. Are you sure the problem isn't that the old mails 
> already have a wrong S value and the errors happen when Dovecot tries to read 
> them? 

at my recent onwledge
mixing  compressed new mails from my sent folder, and older uncompressed
ones ( which are shown nice both in tb ), with some copy action in a new
folder from another client leads to wrong S
and tb does not display them any longer sometime, perhaps a double zip
happens?

It would help to know how to reproduce this on an entirely empty mail
account.
> 

i will try to test, but first i have to deal with now broken maildirs
i try disable zlip plugin now ,as workaround hopefully this does not
lead to more problems on old maildirs , and fix the problem with new mail

-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Timo Sirainen]

On 1.8.2012, at 0.29, Robert Schetterer wrote:

> Cached message size smaller than expected
> and
> Maildir filename has wrong S value
> and
> Corrupted index cache file
> 
> in dove 2.1.8
> 
> i can reproduce the problem with tb 14
> with 2 nat machines connected to the same server
> 
> the problem relates with create new folder then copy
> some uncompressed and compressed mail to it, the i.e
> open it by other client
> 
> sometime this happens at once , sometime after a few copy actions

I can't reproduce this. Are you sure the problem isn't that the old mails 
already have a wrong S value and the errors happen when Dovecot tries to read 
them? It would help to know how to reproduce this on an entirely empty mail 
account.

[Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected

[Robert Schetterer]

Hi Timo,
by having errors like

Cached message size smaller than expected
and
Maildir filename has wrong S value
and
Corrupted index cache file

in dove 2.1.8

i can reproduce the problem with tb 14
with 2 nat machines connected to the same server

the problem relates with create new folder then copy
some uncompressed and compressed mail to it, the i.e
open it by other client

sometime this happens at once , sometime after a few copy actions

however thunderbird ,then errors above apear, sometimes no mail is
shown, sometimes only uncompressed or only compressed

so funktion read both formats gets lost somehow
any idea, you need a better debug ?
-- 
Best Regards
MfG Robert Schetterer

Re: [Dovecot] zlib plugin doesn't read concatenated .gz mailbox

[Timo Sirainen]

On Fri, 2011-07-29 at 20:00 +0300, Dmitry Nezhevenko wrote:

> I've observed that dovecot doesn't displays all mails from compressed via
> gzip mailbox (mbox). At the same time "mutt -f mailbox.gz" displays it
> correctly with all messages.

Fixed: http://hg.dovecot.org/dovecot-2.0/rev/35e4a547231c

[Dovecot] zlib plugin doesn't read concatenated .gz mailbox

[Dmitry Nezhevenko]

Hi,

I've observed that dovecot doesn't displays all mails from compressed via
gzip mailbox (mbox). At the same time "mutt -f mailbox.gz" displays it
correctly with all messages.

I'm using archivemail to archive old mails. Internally it creates new
"gzip" archive with "expired" messages and then concatenates it with
archive.

The easiest way to reproduce it without archivemail is to create two
mailboxes, gzip them independently and then concatenate to one gzip
archive. mutt, gunzip and other tools understands such archives.

Also I've shared such mbox:
http://dion.org.ua/uploads/2011/07/dovecot_test_gzip.gz

It should contains ~20 messages. dovecot will "see" only two first.

PS. 
% sudo dovecot --version
2.0.13

-- 
WBR, Dmitry


signature.asc
Description: Digital signature

Re: [Dovecot] zlib

[Timo Sirainen]

On Sun, 2011-05-22 at 14:05 -0700, Daniel L. Miller wrote:
> How can I enable zlib for reading compressed messages but not writing?

That's the default. Writing is enabled only with zlib_save plugin
setting.

Re: [Dovecot] zlib

[Patrick Domack]

only enable the plugin for pop and imap, but not lda or lmtp modes.


Quoting "Daniel L. Miller" :


How can I enable zlib for reading compressed messages but not writing?

--
Daniel

[Dovecot] zlib

[Daniel L. Miller]

How can I enable zlib for reading compressed messages but not writing?

--
Daniel

Re: [Dovecot] zlib + mdbox

[interfaSys sàrl]

So I guess it has nothing to do with zlib then and that combination of
mdbox + zlib is safe and working as expected.

The only error messages I see in the logs are messages about time
jumping forward.

I'm using these plugins btw:
# Mail processes
mail_plugins = quota trash zlib fts fts_squat acl

# 20-imap.conf
mail_plugins = $mail_plugins imap_quota imap_zlib antispam imap_acl
autocreate


On 24/11/2010 17:00, Timo Sirainen wrote:
> Are you talking about imap-zlib plugin = COMPRESS extension? Just using zlib 
> plugin to compress mails shouldn't be visible to clients in any way.
> 
> On 24.11.2010, at 13.30, interfaSys sàrl wrote:
> 
>> I am.
>> The only issue I'm seeing is Blackberry devices that can't idle after a
>> while.
>> It was the same with Maildir, so maybe it's a zlib issue or something
>> else all together.
>>
>>
>>>
>>> Message: 2
>>> Date: Tue, 23 Nov 2010 18:19:18 -0800
>>> From: "Daniel L. Miller" 
>>> Subject: [Dovecot] zlib + mdbox
>>> To: Dovecot Mailing List 
>>> Message-ID: <4cec7626.4010...@amfes.com>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> Is anyone currently using zlib with mdbox?  Any issues?
>>>
>>
> 

Re: [Dovecot] zlib + mdbox

[Timo Sirainen]

Are you talking about imap-zlib plugin = COMPRESS extension? Just using zlib 
plugin to compress mails shouldn't be visible to clients in any way.

On 24.11.2010, at 13.30, interfaSys sàrl wrote:

> I am.
> The only issue I'm seeing is Blackberry devices that can't idle after a
> while.
> It was the same with Maildir, so maybe it's a zlib issue or something
> else all together.
> 
> 
>> 
>> Message: 2
>> Date: Tue, 23 Nov 2010 18:19:18 -0800
>> From: "Daniel L. Miller" 
>> Subject: [Dovecot] zlib + mdbox
>> To: Dovecot Mailing List 
>> Message-ID: <4cec7626.4010...@amfes.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>> 
>> Is anyone currently using zlib with mdbox?  Any issues?
>> 
> 

Re: [Dovecot] zlib + mdbox

[interfaSys sàrl]

I am.
The only issue I'm seeing is Blackberry devices that can't idle after a
while.
It was the same with Maildir, so maybe it's a zlib issue or something
else all together.


> 
> Message: 2
> Date: Tue, 23 Nov 2010 18:19:18 -0800
> From: "Daniel L. Miller" 
> Subject: [Dovecot] zlib + mdbox
> To: Dovecot Mailing List 
> Message-ID: <4cec7626.4010...@amfes.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> Is anyone currently using zlib with mdbox?  Any issues?
> 

[Dovecot] zlib + mdbox

[Daniel L. Miller]

Is anyone currently using zlib with mdbox?  Any issues?

--
Daniel

Re: [Dovecot] zlib plugin weirdness

[Kyle Wheeler]

On Friday, October  1 at 11:51 AM, quoth Kyle Wheeler:

That does it for gz-compressed files; it works!

However, for bz2-compressed files, I get:

Error: Next message unexpectedly corrupted in mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 
at 1907
Error: Unexpectedly lost From-line from mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 
at 1907
Error: i_stream_get_size() failed with mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: 
No such file or directory


I made the change suggested by 
http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 to istream-bzlib.c, 
and things now work perfectly for bz2-compressed files as well.


~Kyle
--
He who joyfully marches in rank and file has already earned my 
contempt. He has been given a large brain by mistake, since for him 
the spinal cord would suffice.

-- Albert Einstein


signature.asc
Description: Digital signature

Re: [Dovecot] zlib plugin weirdness

[Kyle Wheeler]

On Friday, October  1 at 05:46 PM, quoth Timo Sirainen:

On Fri, 2010-10-01 at 11:21 -0500, Kyle Wheeler wrote:

On Friday, October  1 at 05:06 PM, quoth Timo Sirainen:
>On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote:
>
>> I'm trying to get the zlib plugin working on my 2.0 server. I started 
>> with an mbox that Dovecot can read just fine. Then I gzipped it, and 
>> now Dovecot complains that it's corrupted:

>
>Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527

No, that causes a rather ugly panic:


What about the above +
http://hg.dovecot.org/dovecot-2.0/rev/e7768ec9d3de


That does it for gz-compressed files; it works!

However, for bz2-compressed files, I get:

Error: Next message unexpectedly corrupted in mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 
at 1907
Error: Unexpectedly lost From-line from mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 
at 1907
Error: i_stream_get_size() failed with mbox file 
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: 
No such file or directory


~Kyle
--
Time is an illusion. Lunchtime doubly so.
  -- Douglas Adams


signature.asc
Description: Digital signature

Re: [Dovecot] zlib plugin weirdness

[Timo Sirainen]

On Fri, 2010-10-01 at 11:21 -0500, Kyle Wheeler wrote:
> On Friday, October  1 at 05:06 PM, quoth Timo Sirainen:
> >On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote:
> >
> >> I'm trying to get the zlib plugin working on my 2.0 server. I started 
> >> with an mbox that Dovecot can read just fine. Then I gzipped it, and 
> >> now Dovecot complains that it's corrupted:
> >
> >Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527
> 
> No, that causes a rather ugly panic:

What about the above +
http://hg.dovecot.org/dovecot-2.0/rev/e7768ec9d3de

Re: [Dovecot] zlib plugin weirdness

[Kyle Wheeler]

On Friday, October  1 at 05:06 PM, quoth Timo Sirainen:

On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote:

I'm trying to get the zlib plugin working on my 2.0 server. I started 
with an mbox that Dovecot can read just fine. Then I gzipped it, and 
now Dovecot complains that it's corrupted:


Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527


No, that causes a rather ugly panic:

Error: 
zlib.read(/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz): 
unexpected EOF at 1764046
Panic: file istream-zlib.c: line 204 (i_stream_zlib_read): assertion 
failed: (zstream->eof_offset == high_offset)
Error: Raw backtrace: /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 
[0xb76769e1] -> /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 
[0xb7676a5f] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_error+0) 
[0xb7676d18] -> /tmp/dovecottesting/lib/dovecot/lib20_zlib_plugin.so 
[0xb74aaf7a] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_stream_read+0x7c) 
[0xb767d50c] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb7730cc1] 
-> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(istream_raw_mbox_get_header_offset+0x78) 
[0xb77316a8] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb773bb99] 
-> /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 
[0xb773e154] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_sync+0x4b) 
[0xb773f35b] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_storage_sync_init+0x6d) 
[0xb773f4cd] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x39) 
[0xb76d0949] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x44) 
[0xb76d11f4] -> dovecot/imap [k...@memoryhole.net 64.253.106.173 
SELECT](cmd_select_full+0x1d2) [0x8053962] -> dovecot/imap 
[k...@memoryhole.net 64.253.106.173 SELECT](cmd_select+0x19) 
[0x80543d9] -> dovecot/imap [k...@memoryhole.net 64.253.106.173 
SELECT] [0x805673c] -> dovecot/imap [k...@memoryhole.net 
64.253.106.173 SELECT] [0x80567d9] -> dovecot/imap 
[k...@memoryhole.net 64.253.106.173 SELECT](client_handle_input+0x2d) 
[0x805694d] -> dovecot/imap [k...@memoryhole.net 64.253.106.173 
SELECT](client_input+0x5f) [0x805733f] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xf5) 
[0xb7683275] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) 
[0xb7682250] -> 
/tmp/dovecottesting/lib/dovecot/libdovecot.so.0(master_service_run+0x2a) 
[0xb766f72a] -> dovecot/imap [k...@memoryhole.net 64.253.106.173 
SELECT](main+0x2b5) [0x805ffb5] -> 
/lib/i686/cmov/libc.so.6(__libc_start_main+0xe5) [0xb74e3455] -> 
dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x804eac1]
master: Error: service(imap): child 15613 killed with signal 6 (core 
dumped)


~Kyle
--
Why, this Satan's drink is so delicious that it would be a pity to let 
the infidels have exclusive use of it. We shall fool Satan by 
baptizing it, and making it a truly Christian beverage.

  -- Pope Clement VII, referring to coffee


signature.asc
Description: Digital signature

Re: [Dovecot] zlib plugin weirdness

[Timo Sirainen]

On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote:

> I'm trying to get the zlib plugin working on my 2.0 server. I started 
> with an mbox that Dovecot can read just fine. Then I gzipped it, and 
> now Dovecot complains that it's corrupted:

Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527

[Dovecot] zlib plugin weirdness

[Kyle Wheeler]

Hello,

I'm trying to get the zlib plugin working on my 2.0 server. I started 
with an mbox that Dovecot can read just fine. Then I gzipped it, and 
now Dovecot complains that it's corrupted:


Error: Next message unexpectedly corrupted in mbox file
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz
at 1907
Error: Unexpectedly lost From-line from mbox file
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz
at 1907
Error:

zlib.read(/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz):
unexpected EOF at 1765954
Error: i_stream_get_size() failed with mbox file
/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz:
No such file or directory

A similar thing happens when I bzip it. I was worried that it wasn't 
even using the zlib plugin, but since it makes reference to zlib.read, 
I assume that it is. Does anyone know what might be going wrong?


~Kyle
--
The only way to oppose a bad idea is to replace it with a good idea.
  -- Jack Kemp


signature.asc
Description: Digital signature

Re: [Dovecot] zlib delivery in 2.0beta6

[Timo Sirainen]

On Sun, 2010-06-13 at 10:59 +, Bernhard Schmidt wrote:
> Jun 13 12:53:55 mail dovecot: imap(bernilrz): Error: FETCH [] for
> mailbox unsorted UID 33930 got too little data: 1018 vs 2850
> Jun 13 12:53:55 mail dovecot: imap(bernilrz): Error: Maildir filename
> has wrong W value:
> /var/mail/vmail/b/bernilrz/.unsorted/cur/1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850

Looks like the plugin API change still had a bug in it: 
http://hg.dovecot.org/dovecot-2.0/rev/07a7d352af66

[Dovecot] zlib delivery in 2.0beta6

[Bernhard Schmidt]

Hi,

I just wanted to test the zlib plugin in Dovecot 2.0 and have a problem.
Delivery via LMTP works fine and the mail is saved compressed.

-rw--- 1 vmail vmail 1012 13. Jun 12:51
1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850:2,
# file 1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850:2,
1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850:2,: gzip
compressed data, from Unix
# zcat 1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850:2, |
wc
 51 2772799

But a client (both Thunderbird and Mutt) cannot access the mail.

Jun 13 12:53:55 mail dovecot: imap-login: Login: user=,
method=PLAIN, rip=2001:a60:f001:1:219:66ff:fe8b:a6e,
lip=2001:1b10:1000::110:1:1, mpid=15416, TLS
Jun 13 12:53:55 mail dovecot: imap(bernilrz): Error: FETCH [] for
mailbox unsorted UID 33930 got too little data: 1018 vs 2850
Jun 13 12:53:55 mail dovecot: imap(bernilrz): Error: Maildir filename
has wrong W value:
/var/mail/vmail/b/bernilrz/.unsorted/cur/1276426292.M93539P15363.mail.svr02.mucip.net,S=2799,W=2850
Jun 13 12:53:55 mail dovecot: imap(bernilrz): Error: Corrupted index
cache file /var/cache/dovecot/b/bernilrz/.unsorted/dovecot.index.cache:
Broken virtual size for mail UID 33930
Jun 13 12:53:55 mail dovecot: imap(bernilrz): Disconnected: FETCH failed
bytes=124/1952

I figured the cache might just be corrupt and I deleted all cachefiles,
but the issue keeps reappearing.

Has anyone done this before?

Config:
# 2.0.beta6 (7a02b84ad7f5): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.33.4 x86_64 Debian squeeze/sid 
default_login_user = dovecot
disable_plaintext_auth = no
listen = 83.170.6.69, 2001:1b10:1000::110:1:1
mail_gid = vmail
mail_location = maildir:%h/:INDEX=/var/cache/dovecot/%1n/%n
mail_uid = vmail
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
passdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
plugin {
  quota = dict:user::file:%h/dovecot-quota
  quota_rule = *:storage=500M
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  zlib_save = gz
  zlib_save_level = 6
}
postmaster_address = postmas...@birkenwald.de
protocols = imap pop3 lmtp managesieve
quota_full_tempfail = yes
service auth {
  unix_listener /var/spool/postfix-mailout/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener auth-userdb {
group = vmail
mode = 0660
  }
}
service lmtp {
  inet_listener {
port = 24
  }
}
service managesieve-login {
  inet_listener sieve_deprecated {
port = 2000
  }
  process_min_avail = 1
}
ssl_cert = 

Re: [Dovecot] Zlib plugin dovecot 2.0beta4

[Alex Baule]

Mostrar romanização
The extra line added to the header is to maintain compatibility with
SMTP(share library code), which already works well. Besides having other
benefits that are due to the hash in the header.

I tried accessing the same way that istream is done in the zlib (
zlib_permail_get_stream ), where the reference to mail-> uid == 0, (nonzero
is to read the file -> dovecot).


if (imail->data.stream != NULL || (_mail->uid == 0 && zuser->disable_uis ==
1)) { //disable_uis ir read like the zlib_save variable , if is not present
, we don't need to get stream, and works like no modifications.
  return zmail->super.get_stream(_mail, hdr_size, body_size, stream_r);
}else if (imail->data.stream != NULL || _mail->uid == 0 ) { // OK, its a
istream from received email.
  input = imail->data.stream;
  imail->data.stream = i_stream_create_emx(input);
}



But I do not know if it is correct.

If you can tell me where is exactly the way or the point that i must catch
the istream input, i do the rest.

Tks Timo !




2010/5/26 Timo Sirainen 

> On Tue, 2010-05-11 at 16:41 -0300, Alex Baule wrote:
>
> > But i need to get / change some values to create the "link" from header
> to
> > body.
> >
> > When a email is saved, have the S an W flags, that is the size of email.
> > There is a way to increase this Flags ?  I need to put one more line in
> the
> > header, to have some link to body.
>
> So you want to add one header and have the S/W increase the number? The
> problem isn't only the S/W, it's also that Dovecot saves the message
> size to cache file. The only good way to solve this is to change your
> input stream to return the header, so that all of Dovecot's internal
> code sees it and internally adds them to the S/W/cache.
>
> But do you really even need a header for the link? Maybe you could just
> use IMAP UID number? Like if you have ~/Maildir/.foo/cur/abcd:2, file
> with UID=200, its body could be in e.g. ~/Maildir/.foo/body/200.
>
>
>

Re: [Dovecot] Zlib plugin dovecot 2.0beta4

[Timo Sirainen]

On Tue, 2010-05-11 at 16:41 -0300, Alex Baule wrote:

> But i need to get / change some values to create the "link" from header to
> body.
> 
> When a email is saved, have the S an W flags, that is the size of email.
> There is a way to increase this Flags ?  I need to put one more line in the
> header, to have some link to body.

So you want to add one header and have the S/W increase the number? The
problem isn't only the S/W, it's also that Dovecot saves the message
size to cache file. The only good way to solve this is to change your
input stream to return the header, so that all of Dovecot's internal
code sees it and internally adds them to the S/W/cache.

But do you really even need a header for the link? Maybe you could just
use IMAP UID number? Like if you have ~/Maildir/.foo/cur/abcd:2, file
with UID=200, its body could be in e.g. ~/Maildir/.foo/body/200.

[Dovecot] Zlib plugin dovecot 2.0beta4

[Alex Baule]

Hi Timo...

I made the plugin like you say.

The header part is saved by dovecot, with o_stream_send , and the body with
my function.

This implementation, works fine.

But i need to get / change some values to create the "link" from header to
body.

When a email is saved, have the S an W flags, that is the size of email.
There is a way to increase this Flags ?  I need to put one more line in the
header, to have some link to body.

To Increase the S Flag i increase the ostream.offset, but the W is based in
the offset from istream, that's right ?

There is a way to do this without rewrite the read function !? Exemple, when
the email is moved from tmp to cur.

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

Hi Timo...

I made the plugin like you say.

The header part is saved by dovecot, with o_stream_send , and the body with
my function.

This implementation, works fine.

But i need to get / change some values to create the "link" from header to
body.

When a email is saved, have the S an W flags, that is the size of email.
There is a way to increase this Flags ?  I need to put one more line in the
header, to have some link to body.

The second is, the name of file.

I use the o_stream_get_name, passing the output ostream, but the return is
empty. The name is avaliable when ?

Tks Timo.


2010/4/20 Timo Sirainen 

> You should write the message header to the ostream that you get, and
> message body to some other file that you create. Use the o_stream_send()
> function to write the message header to the ostream (which does
> basically the same as write(fd)). I don't see how getting the stream's
> fd would help with anything.
>
> On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote:
> > Because my plugin will separate the body of the message header.
> >
> > With the FD I have control over the message to separate it.
> >
> > Do you have any suggestions for this separation?
> >
> > I do not quite understand how the struct ostream works.
> >
> > Tks Timo.
> >
> >
> >
> > 2010/4/20 Timo Sirainen 
> >
> > > On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:
> > >
> > > > There is a way to get the file descriptor from output used in ostream
> > > zlib
> > > > plugin ?
> > >
> > > No. Why do you need it? You should be writing to ostream, not to its
> fd.
> > >
> > >
>
>

[Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

Hello Everyone.

There is possible to rename the email saved by a plugin, like zlib ?

i explain i need to add 1 line in email header, because of that, i want
to rewrite the filename to update the S and W flag.

Tks !

[Dovecot] zlib plugin dovecot 2.0 - write email

[Alex Baule]

Hello Everyone.

There is possible to rename the email writed by a plugin, like zlib ?

i explain i need to add 1 line in email header, because of that, i want
to rewrite the filename to update the S and W flag.

Tks !

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

Hi Timo


I'm doing the plugin based on what you said.

But I'll add a line in the message header,  so the S and W (size of email)
flags, on the file name has to be changed ... There is a way  to do that in
the plugin?


2010/4/20 Alex Baule 

> If I had the file descriptor, I will reuse my functions to do this,  made
> in a separate library, that i use in the MTA.
>
> So no need to rewrite again the part of split the message.
>
>
>
> 2010/4/20 Timo Sirainen 
>
>> You should write the message header to the ostream that you get, and
>> message body to some other file that you create. Use the o_stream_send()
>> function to write the message header to the ostream (which does
>> basically the same as write(fd)). I don't see how getting the stream's
>> fd would help with anything.
>>
>> On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote:
>> > Because my plugin will separate the body of the message header.
>> >
>> > With the FD I have control over the message to separate it.
>> >
>> > Do you have any suggestions for this separation?
>> >
>> > I do not quite understand how the struct ostream works.
>> >
>> > Tks Timo.
>> >
>> >
>> >
>> > 2010/4/20 Timo Sirainen 
>> >
>> > > On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:
>> > >
>> > > > There is a way to get the file descriptor from output used in
>> ostream
>> > > zlib
>> > > > plugin ?
>> > >
>> > > No. Why do you need it? You should be writing to ostream, not to its
>> fd.
>> > >
>> > >
>>
>>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

If I had the file descriptor, I will reuse my functions to do this,  made in
a separate library, that i use in the MTA.

So no need to rewrite again the part of split the message.


2010/4/20 Timo Sirainen 

> You should write the message header to the ostream that you get, and
> message body to some other file that you create. Use the o_stream_send()
> function to write the message header to the ostream (which does
> basically the same as write(fd)). I don't see how getting the stream's
> fd would help with anything.
>
> On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote:
> > Because my plugin will separate the body of the message header.
> >
> > With the FD I have control over the message to separate it.
> >
> > Do you have any suggestions for this separation?
> >
> > I do not quite understand how the struct ostream works.
> >
> > Tks Timo.
> >
> >
> >
> > 2010/4/20 Timo Sirainen 
> >
> > > On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:
> > >
> > > > There is a way to get the file descriptor from output used in ostream
> > > zlib
> > > > plugin ?
> > >
> > > No. Why do you need it? You should be writing to ostream, not to its
> fd.
> > >
> > >
>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Timo Sirainen]

You should write the message header to the ostream that you get, and
message body to some other file that you create. Use the o_stream_send()
function to write the message header to the ostream (which does
basically the same as write(fd)). I don't see how getting the stream's
fd would help with anything.

On Tue, 2010-04-20 at 11:54 -0300, Alex Baule wrote:
> Because my plugin will separate the body of the message header.
> 
> With the FD I have control over the message to separate it.
> 
> Do you have any suggestions for this separation?
> 
> I do not quite understand how the struct ostream works.
> 
> Tks Timo.
> 
> 
> 
> 2010/4/20 Timo Sirainen 
> 
> > On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:
> >
> > > There is a way to get the file descriptor from output used in ostream
> > zlib
> > > plugin ?
> >
> > No. Why do you need it? You should be writing to ostream, not to its fd.
> >
> >



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

Because my plugin will separate the body of the message header.

With the FD I have control over the message to separate it.

Do you have any suggestions for this separation?

I do not quite understand how the struct ostream works.

Tks Timo.



2010/4/20 Timo Sirainen 

> On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:
>
> > There is a way to get the file descriptor from output used in ostream
> zlib
> > plugin ?
>
> No. Why do you need it? You should be writing to ostream, not to its fd.
>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Timo Sirainen]

On Tue, 2010-04-20 at 11:40 -0300, Alex Baule wrote:

> There is a way to get the file descriptor from output used in ostream zlib
> plugin ?

No. Why do you need it? You should be writing to ostream, not to its fd.



signature.asc
Description: This is a digitally signed message part

[Dovecot] zlib Plugin Dovecot 2.0 - ostream

[Alex Baule]

Hello Everyone...

In zlib save mail have it:

struct ostream *output;

There is a way to get the file descriptor from output used in ostream zlib
plugin ?

to get in istream is i_stream_get_fd, but don't have o_stream_get_fd..

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Timo Sirainen]

On 16.4.2010, at 19.15, Alex Baule wrote:

> This iov is every line from the new emaill to save ?
> 
> Or this is a email block with X bytes ?
> 
> In my function to write email, i split the email by \n, to save it splited,
> and if the iov is splited by \n, is good for me.

You can't assume anything about iov. iov_count is typically 1, but can be much 
more. Each block's length can be from 1 byte to anything (typically less than 
8kB I guess).

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

Ok !

That's easy.

In the sendv rewrite, have this:

 for (i = 0; i < iov_count; i++) {
   if (o_stream_emexis_send_chunk(emexis_stream, iov[i].iov_base,
iov[i].iov_len) < 0)
   return -1;
   bytes += iov[i].iov_len;
   }

This iov is every line from the new emaill to save ?

Or this is a email block with X bytes ?

In my function to write email, i split the email by \n, to save it splited,
and if the iov is splited by \n, is good for me.


2010/4/16 Timo Sirainen 

> On Fri, 2010-04-16 at 18:34 +0300, Timo Sirainen wrote:
> > On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote:
> >
> > > Part of reading I already migrated, only that the piece of writing (the
> new
> > > part in the plugin) I need to understand some things about the
> operation in
> > > order to migrate and redo the same functions with the separation of
> body /
> > > header
> >
> > So you also want to add support for the split operating directly during
> > saving new mails?
> >
> > I guess you can copy&paste most of the code from zlib-plugin.c and just
> > implement a new ostream. ostream-bzlib.c should give a good example. You
> > just need to implement:
>
> Oh, and the output ostream that you get from create() is the one that
> ends up in the maildir. So I guess you should be writing headers there,
> and write the body to another file some other way.
>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Timo Sirainen]

On Fri, 2010-04-16 at 18:34 +0300, Timo Sirainen wrote:
> On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote:
> 
> > Part of reading I already migrated, only that the piece of writing (the new
> > part in the plugin) I need to understand some things about the operation in
> > order to migrate and redo the same functions with the separation of body /
> > header
> 
> So you also want to add support for the split operating directly during
> saving new mails?
> 
> I guess you can copy&paste most of the code from zlib-plugin.c and just
> implement a new ostream. ostream-bzlib.c should give a good example. You
> just need to implement:

Oh, and the output ostream that you get from create() is the one that
ends up in the maildir. So I guess you should be writing headers there,
and write the body to another file some other way.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Timo Sirainen]

On Fri, 2010-04-16 at 09:55 -0300, Alex Baule wrote:

> Part of reading I already migrated, only that the piece of writing (the new
> part in the plugin) I need to understand some things about the operation in
> order to migrate and redo the same functions with the separation of body /
> header

So you also want to add support for the split operating directly during
saving new mails?

I guess you can copy&paste most of the code from zlib-plugin.c and just
implement a new ostream. ostream-bzlib.c should give a good example. You
just need to implement:

 - sendv(): this works similar to writev() syscall. Here you need to
keep read the input bytes and track when the message header ends and
write to different files before/after that.

 - flush(): If you do any buffering, flush it here.

That's about it. Much simpler than istreams!



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

Thanks Timo,

I have another question about the plugin.

Part of reading I already migrated, only that the piece of writing (the new
part in the plugin) I need to understand some things about the operation in
order to migrate and redo the same functions with the separation of body /
header

Can you explain to me ?

Tks again !



2010/4/16 Timo Sirainen 

> On Fri, 2010-04-09 at 11:42 -0300, Alex Baule wrote:
>
> > I developed a plugin based on the zlib plugin, but want to port it to
> > dovecot 2.0.
> >
> > There are some differences in the 2.0 plugin, which I did not quite
> > understand how it works for me to readjust it in my plugin.
>
> I don't think you need to do (almost?) any changes. v2.0's zlib plugin
> is very different from v1.x, but that's because I'm using zlib library
> in a different way. Not because of some istream changes.
>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Timo Sirainen]

On Fri, 2010-04-09 at 11:42 -0300, Alex Baule wrote:

> I developed a plugin based on the zlib plugin, but want to port it to
> dovecot 2.0.
> 
> There are some differences in the 2.0 plugin, which I did not quite
> understand how it works for me to readjust it in my plugin.

I don't think you need to do (almost?) any changes. v2.0's zlib plugin
is very different from v1.x, but that's because I'm using zlib library
in a different way. Not because of some istream changes.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

Hi Paschal

You do not understand what I want.

I developed a plugin based on the zlib plugin, but want to port it to
dovecot 2.0.

There are some differences in the 2.0 plugin, which I did not quite
understand how it works for me to readjust it in my plugin.

As in the source code there are not many comments, I do not quite know what
each function does, if they are all necessary in my adaptation.

In developing the plugin in version 1.4, Timo helped me a lot.

What I need to know is how the hooks on write emails do this,  and what the
differences in the read function, since the code is different from version
1.4 to 2.0.


2010/4/9 Alex Baule 

> I Dont want the configuration.
>
> I Want know how is implemented.
>
> Explain, the functions, the hooks to get write a email
>
> stuffs like this.
>
> 2010/4/8 Alex Baule 
>
> Hi Everyone
>>
>> Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ??
>>
>> In zlib from 2.0 beta4, Which part of the plugin is responsible for write
>> the compressed message ?
>>
>>
>>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

Hi Paschal

You do not understand what I want.

I developed a plugin based on the zlib plugin, but want to port it to
dovecot 2.0.

There are some differences in the 2.0 plugin, which I did not quite
understand how it works for me to readjust it in my plugin.

As in the source code there are not many comments, I do not quite know what
each function does, if they are all necessary in my adaptation.

In developing the plugin in version 1.4, Timo helped me a lot.

What I need to know is how the hooks on write emails do this,  and what the
differences in the read function, since the code is different from version
1.4 to 2.0.

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Pascal Volk]

On 04/09/2010 02:45 PM Alex Baule wrote:
> I Dont want the configuration.
> 
> I Want know how is implemented.
> 
> Explain, the functions, the hooks to get write a email
> 
> stuffs like this.

How it is implemented:
http://hg.dovecot.org/dovecot-2.0/file/tip/src/plugins/zlib


Regards,
Pascal
-- 
The trapper recommends today: defaced.1009...@localdomain.org

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

I Dont want the configuration.

I Want know how is implemented.

Explain, the functions, the hooks to get write a email

stuffs like this.

2010/4/8 Alex Baule 

> Hi Everyone
>
> Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ??
>
> In zlib from 2.0 beta4, Which part of the plugin is responsible for write
> the compressed message ?
>
>
>

Re: [Dovecot] zlib Plugin Dovecot 2.0

[Pascal Volk]

On 04/08/2010 04:56 PM Alex Baule wrote:
> Hi Everyone
> 
> Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ??
> 
> In zlib from 2.0 beta4, Which part of the plugin is responsible for write
> the compressed message ?
> 

For reading compressed files you will need to enable zlib plugin for the
used protocols. When using imap and pop3, configure something like:

protocol imap {
  mail_plugins = zlib
}
protocol pop3 {
  mail_plugins = zlib
}

If you want to store compressed messages (only for v2.0+) you have to
configure the zlib plugin in the plugin section:

plugin {
  zlib_save_level = 6 # 1 - 9 (6 = default)
  zlib_save = gz # or bz2
}

When you are using the dovecot-lda binary, add the zlib plugin:

protocol lda {
  mail_plugins = zlib
}

And when using the lmtp, also activate the zlib for the lmtp:

protocol lmtp {
  mail_plugins = zlib
}

And finally, if your IMAP client is able to read compressed mails,
change your protocol imap configuration to something like:

protocol imap {
  mail_plugins = zlib imap_zlib
}


Regards {
  Pascal
}
-- 
The trapper recommends today: cafebabe.1009...@localdomain.org

[Dovecot] zlib Plugin Dovecot 2.0

[Alex Baule]

Hi Everyone

Someone can explain to me the difference from zlib 1.2.X and 2.0 beta4 ??

In zlib from 2.0 beta4, Which part of the plugin is responsible for write
the compressed message ?

Re: [Dovecot] zlib plugin trouble

[Stéphane Cottin]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


Le 16 févr. 2010 à 16:55, Timo Sirainen a écrit :

> On 16.2.2010, at 17.39, Stéphane Cottin wrote:
> 
 mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() 
 failed: DATA_ERROR_MAGIC
> 
>> I made a simple script to gzip uncompressed messages and convert bzip2 to 
>> gzip (I can share it if anyone cares), There is no bzip2 compressed files 
>> left.
> 
> The above error message string is coming from bzip2 library.
> 
> % grep DATA_ERROR_MAGIC /usr/lib/libz.so  
> % grep DATA_ERROR_MAGIC /usr/lib/libbz2.so
> Binary file /usr/lib/libbz2.so matches
> 
> So I think you do have some bzip2 compressed files left.

Argh, you're right !

Sorry for the noise and many thanks for your time.
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAkt6wmIACgkQUeIu/wInUezxIQCfZ3rhPwuro1LUCVasi0dI5ZjE
dvcAnA007JVvBjXs9rav1cnLsjfL9jpi
=+VaW
-END PGP SIGNATURE-

Re: [Dovecot] zlib plugin trouble

[Timo Sirainen]

On 16.2.2010, at 17.39, Stéphane Cottin wrote:

>>> mail.err: Feb 16 14:22:26 dovecot: IMAP(hid...@domain.com): gzread() 
>>> failed: DATA_ERROR_MAGIC

> I made a simple script to gzip uncompressed messages and convert bzip2 to 
> gzip (I can share it if anyone cares), There is no bzip2 compressed files 
> left.

The above error message string is coming from bzip2 library.

% grep DATA_ERROR_MAGIC /usr/lib/libz.so  
% grep DATA_ERROR_MAGIC /usr/lib/libbz2.so
Binary file /usr/lib/libbz2.so matches

So I think you do have some bzip2 compressed files left.