Re: Quotas and disk usage
On Thu, January 14, 2010 8:26 pm, Wil Cooley wrote:
Adam Tauno Williams wrote:
Some earlier versions of Cyrus had quota calculation issues with big
quotas (2GB? 4GB?) or big mailboxes. I think there is a blurb in WMOGAG
about that, and what version fixed it.
It happens with 2.2 crossing the 4GiB range, but not related to what he's
seeing.
bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user1 Quota % Used Used
Root
5242880 63 3303494 user.user1
bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user2 Quota % Used Used
Root
41943040 93 39410366 user.user2
bash-3.00$ du -sh user1 7.6G user1
bash-3.00$ du -sh user2 6.4G user2
Do you have delayed expunge enabled? If so actual disk size may very
well exceed quota values - because the expunged-but-not-yet-expired messages
are still on the disk.
Other possible contributors:
o Squatter indexes (my estimation is that they add 50%) o Directory sizes
themselves o Other cyrus.* metadata
You can use something like this to calculate just the total (sorry, you need
GNU
du, find and xargs):
find `mbpath user.user1` -type f -name \*. -print0 \ |xargs -0 du -cm|awk
'/total/ { tot=tot+$1 } END { print tot MiB }'
You need the awk if there are more files than possible for a single
invocation of du. And you can calculate the metadata overhead by negating
the -name parameter:
find `mbpath user.user1` -type f \! -name \*. -print0 \ |xargs -0 du -cm|awk
'/total/ { tot=tot+$1 } END { print tot MiB }'
On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several
multiply-linked message files (sitting in different folders/directories).
Those will be counted twice/thrice/... towards the Cyrus quota but not 'du'.
We were unable to find out how some IMAP clients were capable of having the
server create them.
(This being said, we never invested much effort into it :-)
% find `mbpath user/NN` -type f \! -links 1
On our new server (2.3 on Solaris 10 with ZFS and basic filesystem compression
enabled) there is a (zfs get compressratio) difference of 27 to 28 %
Eric Luyten, Computing Centre VUB/ULB, postmaster.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Quotas and disk usage
Quoting Eric Luyten eric.luy...@vub.ac.be:
On Thu, January 14, 2010 8:26 pm, Wil Cooley wrote:
Adam Tauno Williams wrote:
Some earlier versions of Cyrus had quota calculation issues with big
quotas (2GB? 4GB?) or big mailboxes. I think there is a blurb in WMOGAG
about that, and what version fixed it.
It happens with 2.2 crossing the 4GiB range, but not related to what he's
seeing.
bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user1 Quota % Used Used
Root
5242880 63 3303494 user.user1
bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user2 Quota % Used Used
Root
41943040 93 39410366 user.user2
bash-3.00$ du -sh user1 7.6G user1
bash-3.00$ du -sh user2 6.4G user2
Do you have delayed expunge enabled? If so actual disk size may very
well exceed quota values - because the
expunged-but-not-yet-expired messages
are still on the disk.
Other possible contributors:
o Squatter indexes (my estimation is that they add 50%) o Directory sizes
themselves o Other cyrus.* metadata
You can use something like this to calculate just the total (sorry, you need
GNU
du, find and xargs):
find `mbpath user.user1` -type f -name \*. -print0 \ |xargs -0 du -cm|awk
'/total/ { tot=tot+$1 } END { print tot MiB }'
You need the awk if there are more files than possible for a single
invocation of du. And you can calculate the metadata overhead by negating
the -name parameter:
find `mbpath user.user1` -type f \! -name \*. -print0 \ |xargs -0 du -cm|awk
'/total/ { tot=tot+$1 } END { print tot MiB }'
On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several
multiply-linked message files (sitting in different folders/directories).
Those will be counted twice/thrice/... towards the Cyrus quota but not 'du'.
We were unable to find out how some IMAP clients were capable of having the
server create them.
(This being said, we never invested much effort into it :-)
% find `mbpath user/NN` -type f \! -links 1
It's not the client but the Server doing this. See man imapd.conf
singleinstancestore: 1
If enabled, imapd, lmtpd and nntpd attempt to only write one copy
of a message per partition and create hard links, resulting in a
potentially large disk savings.
On our new server (2.3 on Solaris 10 with ZFS and basic filesystem
compression
enabled) there is a (zfs get compressratio) difference of 27 to 28 %
Eric Luyten, Computing Centre VUB/ULB, postmaster.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
M.MengeTel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen
smime.p7s
Description: S/MIME Signatur
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
cyrus-imapd-2.2.12-10 -- impurge and -f recursion --
I run cyrus-imapd under CentOS-4.8 (a RedHat el4 clone). I am trying to discover if there is any way to purge a user's INBOX of old and over-sized messages without affecting the contents of any subfolders thereof. I have looked at impurge as the mechanism. However,my reading of the man page, and of online discussions relating to this matter, indicate that in order for impurge to remove any mail from the INBOX then the -f switch has to be passed. But, I also gather that whenever the -f switch is used then ALL subfolders of the target directory are purged recursively as well. This strikes me as a rather odd combination of attributes but I cannot see any way to prevent recursion if the -f switch is used. What is the recommended way to purge mail from a user's INBOX only using criteria such as age and size? Sincerely, -- *** E-Mail is NOT a SECURE channel *** James B. Byrnemailto:byrn...@harte-lyne.ca Harte Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Quotas and disk usage
On Fri, January 15, 2010 9:50 am, Michael Menge wrote: Quoting Eric Luyten eric.luy...@vub.ac.be: On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several multiply-linked message files (sitting in different folders/directories). Those will be counted twice/thrice/... towards the Cyrus quota but not 'du'. We were unable to find out how some IMAP clients were capable of having the server create them. (This being said, we never invested much effort into it :-) % find `mbpath user/NN` -type f \! -links 1 It's not the client but the Server doing this. See man imapd.conf singleinstancestore: 1 If enabled, imapd, lmtpd and nntpd attempt to only write one copy of a message per partition and create hard links, resulting in a potentially large disk savings. Michael, Due to the way our Postfix interacts with Cyrus I'm pretty sure these doubly-linked message files do not start existing at delivery (lmtp) time, but during IMAP sessions. Some of them (I looked into it earlier today) appear to be the fruits of incorrectly terminated (Mac Mail) user sessions, where the expunge on-exit is not performed. Eric. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
