Re: using skiplist for quotas.db
so I wonder, will skiplist be a better choice? obviously running quota(8) will be a very cheap operation, but I'm worried about contention on the quota database during delivery etc. (these users are for the most part not actively using the system -- they get less than one message per day each(!) -- so I probably worry too much.) The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. What filesystem are you using? Have you considered trying reiserfs with tails enabled. It's quite impressive how little disk space it actually uses (don't believe du output). We used to use NVRAM cards, and even with 100,000 users, it was only about 30 megabytes for the quota files if memory serves me right (300 bytes each for directory entry, stat data, and file data), and I imagine the entire thing ends up in the OS buffer cache in pretty short order... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
Rob Mueller wrote: so I wonder, will skiplist be a better choice? obviously running quota(8) will be a very cheap operation, but I'm worried about contention on the quota database during delivery etc. (these users are for the most part not actively using the system -- they get less than one message per day each(!) -- so I probably worry too much.) The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
On Fri, 2007-01-19 at 21:58 +1100, Rob Mueller wrote: What filesystem are you using? Have you considered trying reiserfs with tails enabled. It's quite impressive how little disk space it actually uses (don't believe du output). We used to use NVRAM cards, and even with 100,000 users, it was only about 30 megabytes for the quota files if memory serves me right (300 bytes each for directory entry, stat data, and file data), and I imagine the entire thing ends up in the OS buffer cache in pretty short order... we're using ext3. reiserfs is not an option, thank-you-very-much. it has hosed our spool way too many times. an FS which can't handle hardware failure (e.g. a flaky fiber cable) should be taken out and shot. I appreciate your taking the time to make suggestions, though :-) -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
On Fri, 2007-01-19 at 07:49 -0500, Ken Murchison wrote: Rob Mueller wrote: The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. hmmm. *reads cyrusdb_skiplist.c* there's an interesting comment in mycommit about moving the unlock to before the fsync. that would be a very nice performance hack, but should preferably be settable per database, e.g. adding a skiplist_nosync to mimic berkeley_nosync. after all, the quota database isn't critical information, it's updated from an authoritative source, anyway. I'll see if I can muster the courage to try this out. the worst that can happen is that users don't have any quota limits at all, right? :) thanks for the input, Rob and Ken. -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: using skiplist for quotas.db
Or things get mangled and people end up over quota. (from my Treo) -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University -Original Message- From: Kjetil Torgrim Homme [EMAIL PROTECTED] To: Ken Murchison [EMAIL PROTECTED] Cc: Rob Mueller [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: 1/19/07 9:42 AM Subject: Re: using skiplist for quotas.db On Fri, 2007-01-19 at 07:49 -0500, Ken Murchison wrote: Rob Mueller wrote: The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. hmmm. *reads cyrusdb_skiplist.c* there's an interesting comment in mycommit about moving the unlock to before the fsync. that would be a very nice performance hack, but should preferably be settable per database, e.g. adding a skiplist_nosync to mimic berkeley_nosync. after all, the quota database isn't critical information, it's updated from an authoritative source, anyway. I'll see if I can muster the courage to try this out. the worst that can happen is that users don't have any quota limits at all, right? :) thanks for the input, Rob and Ken. -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
using skiplist for quotas.db
we're running Cyrus 2.2.12 on a server with many virtual domains and users. currently, we're using quotalegacy, but running quota(8) requires reading many directories (1736 to be exact) to enumerate the quota roots (keys) and opening the quota file for each of more than 100k users. usually, this completes in 1-2 minutes, but during high I/O load on the system (backup), it can take 15 minutes. so I wonder, will skiplist be a better choice? obviously running quota(8) will be a very cheap operation, but I'm worried about contention on the quota database during delivery etc. (these users are for the most part not actively using the system -- they get less than one message per day each(!) -- so I probably worry too much.) also, will the skiplist file get fragmented? if so, is there a tool to defragment it? -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
