Re: [Dovecot] Dovecot never release preallocated space in mdbox
On 23.8.2013, at 15.32, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-02 8:30 AM, Timo Sirainen t...@iki.fi wrote: The problem is that there is no rotation time. In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things. I'm considering migrating my users to mdbox, and thought that I had sent a follow-up to this, but found it in my Drafts folder... This does give me a little pause for concern... Shouldn't there be an internal, automatic mechanism in place for compacting (this I think is the more appropriate term, since it is what Thunderbird and Outlook both use, although it is used for local client files, as opposed to server based files) these files? Personally I'd like it to happen immediately, when a user deletes a(some) message(es). Maybe add a new setting for 'mdbox_compact_trigger' or something like that, where it only does it if it will reclaim at least ## amount of space - although checking for that in and of itself might be as much or more work as just doing it immediately when the email(s) is(are) deleted. There used to be some automatic purging done if enough mails were deleted, but that implementation made the performance much worse so I removed it. Some other better working automation could be added I guess. But, even if it wasn't fully automatic, maybe simplest would be just another option to pass to doveadm purge (ie, -C for 'compact)? You mean for deleting the preallocated space? I guess it could look at the mdbox_rotate_interval and mdbox_preallocate_space settings and based on that remove the preallocated space from the old files.. But I'm still not convinced it's useful to combine those settings in any case, so doesn't seem like worth the trouble to implement it. Also - what happens currently when *all* messages in any given mdbox file are deleted? Is the mdbox file deleted? When any messages are deleted from mdbox, the non-deleted messages are moved to another mdbox file and the old mdbox file is deleted.
Re: [Dovecot] Dovecot never release preallocated space in mdbox
On 2013-08-02 8:30 AM, Timo Sirainen t...@iki.fi wrote: The problem is that there is no rotation time. In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things. I'm considering migrating my users to mdbox, and thought that I had sent a follow-up to this, but found it in my Drafts folder... This does give me a little pause for concern... Shouldn't there be an internal, automatic mechanism in place for compacting (this I think is the more appropriate term, since it is what Thunderbird and Outlook both use, although it is used for local client files, as opposed to server based files) these files? Personally I'd like it to happen immediately, when a user deletes a(some) message(es). Maybe add a new setting for 'mdbox_compact_trigger' or something like that, where it only does it if it will reclaim at least ## amount of space - although checking for that in and of itself might be as much or more work as just doing it immediately when the email(s) is(are) deleted. But, even if it wasn't fully automatic, maybe simplest would be just another option to pass to doveadm purge (ie, -C for 'compact)? Also - what happens currently when *all* messages in any given mdbox file are deleted? Is the mdbox file deleted? Thanks, -- Best regards, */Charles/*
Re: [Dovecot] Dovecot never release preallocated space in mdbox
On Mon, 2013-07-29 at 11:48 +0200, Stéphane BERTHELOT wrote: mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox I don't think the remaining config is relevant but ask me if you need some other parts. Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched (to use a similar term than man fallocate on Linux), even when rotating m.* files. Yeah, those settings weren't really intended to be used together. There would certainly be smart to use something similar to FALLOC_FL_PUNCH_HOLE on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space. The problem is that there is no rotation time. In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things..
Re: [Dovecot] Dovecot never release preallocated space in mdbox
Le 02/08/2013 14:30, Timo Sirainen a écrit : On Mon, 2013-07-29 at 11:48 +0200, Stéphane BERTHELOT wrote: mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox I don't think the remaining config is relevant but ask me if you need some other parts. Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched (to use a similar term than man fallocate on Linux), even when rotating m.* files. Yeah, those settings weren't really intended to be used together. I am not sure to understand. Preallocation is based on rotate size actually, so you mean I shouldn't use a rotate interval with this kind of settings ? It would effectively lose less space since all files would be around the preallocated size except the last one (current). There would certainly be smart to use something similar to FALLOC_FL_PUNCH_HOLE on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space. The problem is that there is no rotation time. In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things.. Ok, I'm sorry I forgot to add that I'm using a cronjob to doveadm purge weekly. That effectively may be a good time to remove unneeded preallocated sectors. I didn't have a look at the internals of the devecot process but I meant that this could happen when it chooses to switch to another file (because of rotation time, not purge). Effectively it may be at a time that we would want to avoid adding more IO... When reading linux docs lately the more appropriate term may be truncation rather than hole punching. That's what I did on command line (truncate with proper args) and it resetted the allocated size to real size. I am pretty sure my understanding on those options was wrong so I already changed to 2M/no preallocation (since I didn't find any evidence that preallocation is making dovecot perform so much better ; there seemed to be a thread once on this list but noone concluded on that I think). My objective was to get a good compromise on performance (since I'm using ext4 anyways) and backup time (otherwise I wouldn't bother with rotation to compensate the potentially full rebuild of doveadm purge) Maybe a documentation update should do the trick or a configuration warning stating that preallocation+rotate time would leave wasted space on disk.
Re: [Dovecot] Dovecot never release preallocated space in mdbox
On Mon, Jul 29, 2013 at 11:48:00AM +0200, Stéphane BERTHELOT wrote: mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes On mailboxes patterns with low incoming mail ( 100kb / day) this would waste much space. Of course I can decrease rotate size a lot but it would then produce a lot of files and would certainly become similar performance-wise to sdbox/maildir/... 128MB is quite a large rotate size if you care about disk space.. We use the default 2 MB, which still packs quite a lot of messages per file compared to maildir. Single maildir-files seems to be around 5-30KB (compressed), which should amount to 50-400 messages per m-file. I don't think that should be similar to maildir/sdbox performance-wise. -jf
