Re: Quotas and disk usage
On Fri, January 15, 2010 9:50 am, Michael Menge wrote: > Quoting Eric Luyten : >> On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several >> multiply-linked message files (sitting in different folders/directories). >> >> Those will be counted twice/thrice/... towards the Cyrus quota but not >> 'du'. >> We were unable to find out how some IMAP clients were capable of having the >> server create them. (This being said, we never invested much effort into it >> :-) >> >> >> % find `mbpath user/NN` -type f \! -links 1 >> >> > > It's not the client but the Server doing this. See man imapd.conf > > > singleinstancestore: 1 > If enabled, imapd, lmtpd and nntpd attempt to only write one copy > of a message per partition and create hard links, resulting in a potentially > large disk savings. > Michael, Due to the way our Postfix interacts with Cyrus I'm pretty sure these doubly-linked message files do not start existing at delivery (lmtp) time, but during IMAP sessions. Some of them (I looked into it earlier today) appear to be the fruits of incorrectly terminated (Mac Mail) user sessions, where the "expunge on-exit" is not performed. Eric. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
cyrus-imapd-2.2.12-10 -- impurge and -f recursion --
I run cyrus-imapd under CentOS-4.8 (a RedHat el4 clone). I am trying to discover if there is any way to purge a user's INBOX of old and over-sized messages without affecting the contents of any subfolders thereof. I have looked at impurge as the mechanism. However,my reading of the man page, and of online discussions relating to this matter, indicate that in order for impurge to remove any mail from the INBOX then the -f switch has to be passed. But, I also gather that whenever the -f switch is used then ALL subfolders of the target directory are purged recursively as well. This strikes me as a rather odd combination of attributes but I cannot see any way to prevent recursion if the -f switch is used. What is the recommended way to purge mail from a user's INBOX only using criteria such as age and size? Sincerely, -- *** E-Mail is NOT a SECURE channel *** James B. Byrnemailto:byrn...@harte-lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Quotas and disk usage
Quoting Eric Luyten : On Thu, January 14, 2010 8:26 pm, Wil Cooley wrote: Adam Tauno Williams wrote: Some earlier versions of Cyrus had quota calculation issues with big quotas (2GB? 4GB?) or big mailboxes. I think there is a blurb in WMOGAG about that, and what version fixed it. It happens with 2.2 crossing the 4GiB range, but not related to what he's seeing. bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user1 Quota % Used Used Root 5242880 63 3303494 user.user1 bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user2 Quota % Used Used Root 41943040 93 39410366 user.user2 bash-3.00$ du -sh user1 7.6G user1 bash-3.00$ du -sh user2 6.4G user2 Do you have delayed expunge enabled? If so actual disk size may very well exceed quota values - because the expunged-but-not-yet-expired messages are still on the disk. Other possible contributors: o Squatter indexes (my estimation is that they add 50%) o Directory sizes themselves o Other cyrus.* metadata You can use something like this to calculate just the total (sorry, you need GNU du, find and xargs): find `mbpath user.user1` -type f -name \*. -print0 \ |xargs -0 du -cm|awk '/total/ { tot=tot+$1 } END { print tot "MiB" }' You need the awk if there are more files than possible for a single invocation of "du". And you can calculate the metadata overhead by negating the "-name" parameter: find `mbpath user.user1` -type f \! -name \*. -print0 \ |xargs -0 du -cm|awk '/total/ { tot=tot+$1 } END { print tot "MiB" }' On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several multiply-linked message files (sitting in different folders/directories). Those will be counted twice/thrice/... towards the Cyrus quota but not 'du'. We were unable to find out how some IMAP clients were capable of having the server create them. (This being said, we never invested much effort into it :-) % find `mbpath user/NN` -type f \! -links 1 It's not the client but the Server doing this. See man imapd.conf singleinstancestore: 1 If enabled, imapd, lmtpd and nntpd attempt to only write one copy of a message per partition and create hard links, resulting in a potentially large disk savings. On our new server (2.3 on Solaris 10 with ZFS and basic filesystem compression enabled) there is a ("zfs get compressratio") difference of 27 to 28 % Eric Luyten, Computing Centre VUB/ULB, postmaster. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Quotas and disk usage
On Thu, January 14, 2010 8:26 pm, Wil Cooley wrote: > Adam Tauno Williams wrote: > > >> Some earlier versions of Cyrus had quota calculation issues with big >> quotas (2GB? 4GB?) or big mailboxes. I think there is a blurb in WMOGAG >> about that, and what version fixed it. > > It happens with 2.2 crossing the 4GiB range, but not related to what he's > seeing. > >>> bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user1 Quota % Used Used >>> Root >>> 5242880 63 3303494 user.user1 >>> bash-3.00$ /opt/cyrus-imap/bin/quota -f user.user2 Quota % Used Used >>> Root >>> 41943040 93 39410366 user.user2 >>> bash-3.00$ du -sh user1 7.6G user1 >>> bash-3.00$ du -sh user2 6.4G user2 >>> >> >> Do you have delayed expunge enabled? If so actual disk size may very >> well exceed quota values - because the expunged-but-not-yet-expired messages >> are still on the disk. > > Other possible contributors: > o Squatter indexes (my estimation is that they add 50%) o Directory sizes > themselves o Other cyrus.* metadata > > You can use something like this to calculate just the total (sorry, you need > GNU > du, find and xargs): > > find `mbpath user.user1` -type f -name \*. -print0 \ |xargs -0 du -cm|awk > '/total/ { tot=tot+$1 } END { print tot "MiB" }' > > > You need the awk if there are more files than possible for a single > invocation of "du". And you can calculate the metadata overhead by negating > the "-name" parameter: > > > find `mbpath user.user1` -type f \! -name \*. -print0 \ |xargs -0 du -cm|awk > '/total/ { tot=tot+$1 } END { print tot "MiB" }' On our previous Cyrus server (2.2 on Solaris 9 with UFS) I detected several multiply-linked message files (sitting in different folders/directories). Those will be counted twice/thrice/... towards the Cyrus quota but not 'du'. We were unable to find out how some IMAP clients were capable of having the server create them. (This being said, we never invested much effort into it :-) % find `mbpath user/NN` -type f \! -links 1 On our new server (2.3 on Solaris 10 with ZFS and basic filesystem compression enabled) there is a ("zfs get compressratio") difference of 27 to 28 % Eric Luyten, Computing Centre VUB/ULB, postmaster. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html