Re: [vchkpw] Opinions needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 DAve wrote: In the last two years we and other ISPs we know have dropped any hope of quotas. Disk space is cheap, and clients want unlimited space because, All the online email services have no limit!. While we still have quotas, I am pretty much asked by the account manager to increase them every time a client exceeds the limit. I see the quotas going away soon. While there may be some select providers to whom disk space is meaningless, I'd venture a bet that at least 50% of providers are going to want to limit their users and domains to a certain, reasonable, amount of storage. Either way, this usage server is coming along nicely, but is also keeping me a bit busy. I will be leaving for a short vacation but when I return, I will be doing the CVS to Subversion conversion, as well as trying to release a beta usage server for developers to play around with if they see fit. Thanks for your input though! - -- /* Matt Brookings m...@inter7.com GnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJf4TI6QgvSNlBT3ARAqzYAKChixDUEcAtsv8AfShxPzNEijfuKACggDgG V1HdSwmZ8oMreIrcxpy2PaY= =HguO -END PGP SIGNATURE-
Re: [vchkpw] Opinions needed
On Jan 22, 2009, at 1:43 AM, Matt Brookings wrote: Where do you come up with 128bit from? On most systems a 'long long' is going to be a 64bit integer. That's what I'm currently using. Sorry, you're right on that. Why isn't it enough to keep the current quota system, and just update all related code to use a long long (64 bits) for the byte counter? Will quotas ever grow that huge? In the next 10 years? You could even keep the 32 bit numbers, and just update all quota- checking code to round sizes up to the nearest KB -- now you're able to have quotas up to 2000 GB (2TB). Since a single email isn't going to be 2GB, each entry in maildirsize can still be a 32-bit int and you may not have to modify qmail or the POP/IMAP servers. You only need to update the code that rebuilds the file and checks for an overquota condition. -Tom !DSPAM:497a2d9e32683946919024!
Re: [vchkpw] Opinions needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tom Collins wrote: On Jan 22, 2009, at 1:43 AM, Matt Brookings wrote: Where do you come up with 128bit from? On most systems a 'long long' is going to be a 64bit integer. That's what I'm currently using. Sorry, you're right on that. Why isn't it enough to keep the current quota system, and just update all related code to use a long long (64 bits) for the byte counter? Will quotas ever grow that huge? In the next 10 years? I'm not replacing the current quota system, and I agree, we should update all the quota code in vpopmail to be 64bits. You could even keep the 32 bit numbers, and just update all quota-checking code to round sizes up to the nearest KB -- now you're able to have quotas up to 2000 GB (2TB). Since a single email isn't going to be 2GB, each entry in maildirsize can still be a 32-bit int and you may not have to modify qmail or the POP/IMAP servers. You only need to update the code that rebuilds the file and checks for an overquota condition. Here's what I'm doing. I have already created a daemon which answers requests about user and domain storage. It has a set of rules by which it operates. 1) Always respond quickly, even if the information is not up to date This is to be sure that processes requesting information on user usage, like vdelivermail, or an SMTP daemon, never bottleneck on the daemon if the system is under heavy load, or fail even if the daemon is not running or fails. This is supported by both the server and client code. Neither are allowed to cause a bottleneck. 2) If a request will take too long to fill and there is no information available, it must return a zero usage response so that delivery will not fail. It's more important to never fail erroneously than to keep quotas enforced in real-time. Again, supported by both the server and client. 3) All updates can be deferred into a queue to be handled at reasonable I/O levels. If a request will take a very long period of time, or is known to use large amounts of disk I/O, for instance, opening a directory with 150k messages in it, the last known information on this directory will be used and the directory can be tagged as an extreme I/O hog to further defer future updates. This will have it's own set of rules for dealing with a growing queue of deferred updates, etc. 4) Do not enforce, or care, about quotas The goal of the daemon is not to provide quota enforcement, it merely provides information on user usage. Comparing usage to quotas, and enforcing quotas is entirely up to the program using the client API. The client API is very easy to use. Here is an example: // storage_t is a typedef of a 64bit unsigned integer defined by the server code storage_t user_usage = 0, domain_usage = 0; cHandle = client_connect(); // Error checking here client_query(cHandle, u...@domain, strlen(u...@domain), user_usage, domain_usage); if (user_usage = user_quota) fail(); if (domain_usage = domain_quota) fail(); I've been running this code on a few production servers where vdelivermail and qmail-smtpd were patched not to enforce quotas, but simply to run queries on every user it sees. Most queries are responded to in 0.010 seconds or less with accuracy to within 15 seconds of actual disk usage. There's still lots of work to be done on it, but I hope to release it in a beta form after I return from vacation Feb 2nd. My current TODO: * Disk updates must be deferrable; maybe with threading * Statistics gathering like average disk poll time, biggest poll time, shortest poll time, etc As you can see, the deferring updates is not yet implemented. The performance I'm currently seeing is acceptable, but it would probably not cut it on a *very* large system with many absent users and catchalls collecting gigs of email in a single directory. I hope to release a beta to developers once the deferring code is ready. Opinions on this? :) - -- /* Matt Brookings m...@inter7.com GnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJejRK6QgvSNlBT3ARAnXvAJ4xXnJp4WpSoTR1pHcmXrY1MuRY6ACgqyjM 6fQJQoatA7+31qxgJqB7ViY= =b74b -END PGP SIGNATURE-
Re: [vchkpw] Opinions needed
Matt Brookings wrote: My current TODO: * Disk updates must be deferrable; maybe with threading * Statistics gathering like average disk poll time, biggest poll time, shortest poll time, etc As you can see, the deferring updates is not yet implemented. The performance I'm currently seeing is acceptable, but it would probably not cut it on a *very* large system with many absent users and catchalls collecting gigs of email in a single directory. I hope to release a beta to developers once the deferring code is ready. Opinions on this? :) Not sure what opinions you want, programmatic or project worthiness 8^) In the last two years we and other ISPs we know have dropped any hope of quotas. Disk space is cheap, and clients want unlimited space because, All the online email services have no limit!. While we still have quotas, I am pretty much asked by the account manager to increase them every time a client exceeds the limit. I see the quotas going away soon. DAve -- The whole internet thing is sucking the life out of me, there ain't no pony in there. !DSPAM:497a394932681764082979!
Re: [vchkpw] Opinions needed - Bignum Arithmetic
On Wed, 2009-01-21 at 21:13 -0800, Tom Collins wrote: Two thoughts on quotas. You're going to have to deal in sub-megabyte numbers, since the size of most messages are measured in KB. Maybe you could track the quota in kbytes, rounding up/down as necessary? All programs that deal with the quota (maildirsize file) will have to use 128-bit numbers (long long?) or whatever new method you come up with. This includes not just vpopmail, but your IMAP server and potentially maildrop and qmail (if you have any .qmail files that are handled by qmail and not vdelivermail). Maybe they can use a dynamically linked library? In computer science, arbitrary-precision arithmetic, also called bignum arithmetic, is a technique whereby programs perform calculations on integers or rational numbers with an arbitrary number of digits of precision. I am planning to use GMP GNU Multiple Precision Arithmentic Library. It claims to be the fastest bignum library on planet. GMP is a free library/ There is no practical limit to the precision except the ones implied by the available memory in the machine. GMP has a rich set of functions to help you deal with these large numbers. http://gmplib.org At this point, I am not sure if bignum is an overkill, but I will be trying this soon and will keep you folks posted Regards Manvendra (http://www.indimail.org) !DSPAM:4978452a32681009520127!
[vchkpw] Opinions needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Some of you may be aware I'm working on changes and additions to the quota system in vpopmail. Part of the aim of the updated system is to provide future-proofing against the problems we're having now, where quota sizes and usage counts are overflowing in various circumstances, either in vpopmail, or in applications utilizing vpopmail. My question is this; would anyone ever require a quota below a megabyte, or, would any application ever really need to know about specific usage counts below a megabyte? When calculating usage, I'm thinking about making the smallest measure of unit a megabyte storing the result in a 64bit unsigned integer. The 64bit value is almost a requirement, but making the smallest unit of measure a megabyte, should future-proof for quite a bit longer. - -- /* Matt Brookings m...@inter7.com GnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJdzVU6QgvSNlBT3ARAiFnAKCeHYx2gywPskZk5S//QBAud2VjrACfTynz 7ASJaP8uqjghUBvwbWi/SxQ= =iOoj -END PGP SIGNATURE-
Re: [vchkpw] Opinions needed
I personally don't require it. It also seems to me with disk space costing what it does, and compression being readily available (via plugins, ZFS, or what have you) it shouldn't really be an issue. Are there any applications that display the actual quota amount that do NOT round to the megabyte? It seems to me that most show percentage used, and round to the megabyte if they do display the actual number - maybe that's where the problem will lie.. Applications that convert the actual quota from bytes to megabytes will break. Rick Matt Brookings wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Some of you may be aware I'm working on changes and additions to the quota system in vpopmail. Part of the aim of the updated system is to provide future-proofing against the problems we're having now, where quota sizes and usage counts are overflowing in various circumstances, either in vpopmail, or in applications utilizing vpopmail. My question is this; would anyone ever require a quota below a megabyte, or, would any application ever really need to know about specific usage counts below a megabyte? When calculating usage, I'm thinking about making the smallest measure of unit a megabyte storing the result in a 64bit unsigned integer. The 64bit value is almost a requirement, but making the smallest unit of measure a megabyte, should future-proof for quite a bit longer. - -- /* Matt Brookings m...@inter7.com GnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJdzVU6QgvSNlBT3ARAiFnAKCeHYx2gywPskZk5S//QBAud2VjrACfTynz 7ASJaP8uqjghUBvwbWi/SxQ= =iOoj -END PGP SIGNATURE- -- Rick Romero Need IT assistance? VF IT Services / VFEmail.net www.vfit.biz / www.vfemail.net !DSPAM:49773dc432683754956235!
Re: [vchkpw] Opinions needed
My question is this; would anyone ever require a quota below a megabyte, or, would any application ever really need to know about specific usage counts below a megabyte? Not us, however current applications assume the format is in bytes (for example the maildirsize file) so calucalations will be off in some frontends or custom made backends when calculating the usage. Sincerely, - Wouter van der Schagt !DSPAM:49773fc232689127091031!
Re: [vchkpw] Opinions needed
i don't really see 16 exabyte mailboxes in near future. But when the time comes for those, there should be 128bit processors and operating systems, so replacing the 64bit unsigned int with a 128bit one at that point should not be a too great deal. Maybe future proof by defining the thing in one place. just my 2 cents. ++jukka !DSPAM:4977794732681263011309!
Re: [vchkpw] Opinions needed
On Jan 21, 2009, at 6:46 AM, Matt Brookings wrote: My question is this; would anyone ever require a quota below a megabyte, or, would any application ever really need to know about specific usage counts below a megabyte? When calculating usage, I'm thinking about making the smallest measure of unit a megabyte storing the result in a 64bit unsigned integer. The 64bit value is almost a requirement, but making the smallest unit of measure a megabyte, should future-proof for quite a bit longer. Two thoughts on quotas. You're going to have to deal in sub-megabyte numbers, since the size of most messages are measured in KB. Maybe you could track the quota in kbytes, rounding up/down as necessary? All programs that deal with the quota (maildirsize file) will have to use 128-bit numbers (long long?) or whatever new method you come up with. This includes not just vpopmail, but your IMAP server and potentially maildrop and qmail (if you have any .qmail files that are handled by qmail and not vdelivermail). Maybe they can use a dynamically linked library? -Tom !DSPAM:4978006832684277763451!
