Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
Eric Shubert wrote: MagicWISP Sales wrote: I just wanted to thank everybody for their help. I had a qtp toaster running on VMWare on a Quad Pentium 2 machine ndash; let me tell you that will not work. It was an experiment that got pressed into service as an emergency. Scan times after killing blacklists and ClamAV on emails of 222k were at 300 seconds, wow. It caused duplicate emails and all kinds of craziness, even after adding Spamdyke. I used one of Jakes tips on his video page and cut the scan time down to an average of 110 on the same email. Quite an improvement but still too high to be usable. I moved the VM to a temp machine this week, an old E-Machines computer of all things with an AMD Athlon XP 2000 processor with a whole 1G of memory, and scan times on the same message fell to 17 seconds. Again ndash; still too high, but at least the duplicate emails stopped. I just used another tweak from Jakes video site and the scan time fell to 9 seconds, with the blacklists and ClamAV scans going again. That will work until the replacement server gets here next week. If you are using VMWare, you really need a higher end machine, I think that could be the moral of this story. The first server that was in use, had a sister sitting right next to it that ran the QTP, plus webhost, plus a radius server, and never ever broke a sweat. VMWare is a hog to say the least. Jakersquo;s video subscription may be the best thing I have found for some quick instruction on usage of real world tweaks. Jake and Eric are always willing to help, and have great experiences to provide. Also a shout out to Brent for fixing the Spamdyke script ndash; it works. Now if I can figure out how to make the smtp log change at midnight instead of whenever it wants, I will be happy. Great job all of you guys ndash; you really are lifesavers!!! Thanks for sharing. Before I comment on this, I want to check the facts. This was a Quad P-II, and not a Quad P-4, right? Just checking. Not that it would make all that much difference. I'll have some comments to make regarding your experience with VM guests soon. FWIW, I just migrated a QMT from one VM guest to another, on the same host. The former ran nicely (and still does on another host). The new QMT VM is a pig. I'm not sure what the problem is yet. Top takes 10% of the cpu on the pig host, while on another guest w/ same kernel, top runs less than 1%. So somewhere I'm seeing a ~10x performance difference between 2 guests on the same host. It'll be interesting (to say the least) to find the reason why. There are some differences between the 3 guests, but not too many. I hope to nail it soon. I've determined that as long as the ram on the guest is less than 656M, the guest runs speedily. If it's over that, then performance degrades by a factor of 10. This threshold is not hard and fixed, as on another host the limit appeared to be 680M. I'm afraid to ask how you came to that strange number... I did see you mention that before somewhere but it was related to running on 32 bit. I'm running x64 host and guest and haven't seen anything like that. In fact my QMT guest was running on 4G for a while and after watching it, I saw it never really used more than a couple hundred meg. The rest went to cache. I just knocked it back to 1G a couple weeks ago and it seems to be just as happy and perky. This appears to be an issue with memory management (duh). This VM guest has vmware-tools installed, which does have a guest memory management component. Could be a bug in that code. I'll be doing more testing to try identify the culprit, and post findings on the wiki. I think that's a note worthy of the wiki. Most people don't understand that the tools do more than video drivers. I'll add that... The difference in performance is rather astounding, and counter intuitive in that *less* ram runs faster. I'm sure there's a logical explanation, but I don't yet know what it is. Haven't seen it in x64. Phil -- -Eric 'shubes'
Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
On 2/27/2010 9:34 PM, MagicWISP Sales wrote: Jake, You are probably right, but we are a small ISP, and never will have a large mail server. If this server ever reaches 500 accounts I will be way more than shocked. So this works well for us. The really nice thing is restoral - it's a snap. If I have a few thousand email accounts, it would be a different story. ;) I do not consider 500 users large, but that does seem to be the break point for virtual environments that I've seen to date. 'Course it's hard to test anything, since I don't have 500 users laying around that are willing to let me mess with their email - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com
Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
On 02/28/2010 09:36 AM, Eric Shubert wrote: Jake Vickers wrote: On 02/27/2010 12:08 PM, MagicWISP Sales wrote: Yes the old server was a Quad PII - 4 Intel 400Mhz processors, it ran like a champ with QMT, Apache, Bind, and FreeRadius. As soon as the Apache, Bind, and QMT were rolled into VMWare -it became a dog. I used about every tweak I could find on the internet to make it work better. I think the problem is VMWare Server 2 - it's just slow. Unfortunately I could not use Xen or KVM because of the old processors. From what I could research, they would have worked much better. I am pretty anxious to see how this runs on another server with dual Intel 3.Ghz processors. Unfortunately they aren't dual core, but we should still see some marked improvement. I'm actually testing/using VirtualBox for things right now. I think in the long run that running a large mailserver on any virtual environment is a bad idea. Keyword: large. (Please be a little more specific) I'm not inclined to accept this premise yet. It's certain that any host needs to be adjusted in certain ways to run efficiently as a guest. Achieving high performance in a virtual guest will also likely require the latest innovations in virtualization hardware (paravirtualization). With a QMT guest, the need for paravirtualization hardware is appearing to me to be more of a certainty in order to reach high levels of efficiency, but I'm not positive about this yet - just a hunch. BL, I wouldn't throw in the towel on QMT virtualization in a large environment quite yet. (And to be clear, virtualized QMTs appears to be working nicely in small to medium environments at this point). You are correct that large needs to be quantified. In the scenarios that I've been brought into, it has been either ~200 users all using IMAP or ~500 users using mixed. Initially it was thought to be a NFS/RAID bottleneck and after moving to attached storage the performance increased by 200% or so. Still was not enough to correct *all* the issues, but it did solve 85% of them. Other tweaks were done, all from a guest perspective. I did not have any direct interaction with the host system (using ESX, VMware Server, and Xen), so I cannot say for sure what tuning needed to be done on those systems. I'd hate to think all 3 of the biggies had the same exact bottlenecks, but we cannot assume anything. I'm not a big virtualization guy - I just know systems with the same metrics work without an issue on dedicated machines. Of course any host is going to run faster on bare iron. The question is, how much of a performance hit are you willing to give up to gain the benefits of virtualization? This is not a one size fits all situation. Correct, but I think it's a safe blanket statement to say that at this time, large* systems should be dedicated. (*large still needing to be quantified). There is a whole other world of variables and tuning that vitualization adds to the soup mix. Until we can identify what the actual issues are, we need to tread carefully. On the 200 user IMAP system, even performing a simple du -sh * on the mailstore took *minutes* on some dirs, with no reasoning behind it. One dir took 6 minutes and was 6G, and another that was 24G took 1 min or so. There may be another guest hogging the cycles during that, which adds a whole other level of complexity to the mix. I heard a fellow say yesterday that with VirtualBox you take a 50% performance hit on the guest. I don't think I'd find that acceptable that in any case. I expect though that this is with no tuning, and that with some tuning that hit can be reduced. I haven't tried out VirtualBox yet, so I'm glad Jake's doing so. My impression though is that VirtualBox is stronger (grew out of) the desktop application arena, and is being retro-fitted for server applications. On the other hand, VMware started with servers and grew into desktops. This is largely why I expect that VMware Server will continue to be better suited to server virtualization than VB. That could change with time, but I'm not expecting it any time soon, especially with Oracle-Sun merger. Not sure about the 50% performance hit; I know the install base is *much* smaller than VMware server, and seems to run a little faster on my laptop. I only use virtualization for development and testing at this point, so cannot say what actual metrics would be either way. I personally think running a big mail environment on a VM (VMware, Xen, VirtualBox, et al) does not fit any of my applications and have no desire to try and prove myself wrong, so this would require outside testing from the community. I'll need to verify, but I think even the big official mail server packages are not supported if run in virtual environment and probably for reasons like this. I can think of other reasons that I think are more likely. ;) Other than poor performance and nightmare-ish
Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
On 02/28/2010 09:41 AM, Eric Shubert wrote: Jake Vickers wrote: On 2/27/2010 9:34 PM, MagicWISP Sales wrote: Jake, You are probably right, but we are a small ISP, and never will have a large mail server. If this server ever reaches 500 accounts I will be way more than shocked. So this works well for us. The really nice thing is restoral - it's a snap. If I have a few thousand email accounts, it would be a different story. ;) I do not consider 500 users large, but that does seem to be the break point for virtual environments that I've seen to date. I'd be curious about these installations. .) is pop vs imap a factor? .) courier or dovecot? .) which VM host/version? .) paravirtual HW? .) I/O scheduling methods? (host and guest) .) VM guests on separate HDs or partitions? Of course, it's also important to know what (if any) tuning has already been done. BL, where/what is the bottleneck?? 'Course it's hard to test anything, since I don't have 500 users laying around that are willing to let me mess with their email That's a stumbling block all right. ;) I'd have to pull notes on the other systems - ESX 4 is the only one I have committed to memory. 200 users or so, all running IMAP. Courier (since it is the official package of QMT at this time - Dovecot to follow in the future). Issue was mainly some users getting timeouts on IMAP connects and accesses. No definitive pattern to it. One mailbox had 4M of mail, another had 3G of mail in folders. Both experienced the same type issue, and not consistently. I do not have any details about the host environment. I was not even told it was a VM environment until I got to poking around trying to find out why the issue was happening. Originally were mapped to a NFS share that was a RAID5 volume, and later moved to a direct attached storage disk. Moving from NFS/RAID to DAS reduced the issues by 200% or so, but still had 2-3 issues at peak times. No idea on other guests on the system. - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com
RE: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
Yes the old server was a Quad PII - 4 Intel 400Mhz processors, it ran like a champ with QMT, Apache, Bind, and FreeRadius. As soon as the Apache, Bind, and QMT were rolled into VMWare -it became a dog. I used about every tweak I could find on the internet to make it work better. I think the problem is VMWare Server 2 - it's just slow. Unfortunately I could not use Xen or KVM because of the old processors. From what I could research, they would have worked much better. I am pretty anxious to see how this runs on another server with dual Intel 3.Ghz processors. Unfortunately they aren't dual core, but we should still see some marked improvement. Jack -Original Message- From: Eric Shubert [mailto:e...@shubes.net] Sent: Saturday, February 27, 2010 10:34 AM To: qmailtoaster-list@qmailtoaster.com Subject: [qmailtoaster] Re: How things can work and do sort themselves out with a little help MagicWISP Sales wrote: I just wanted to thank everybody for their help. I had a qtp toaster running on VMWare on a Quad Pentium 2 machine - let me tell you that will not work. It was an experiment that got pressed into service as an emergency. Scan times after killing blacklists and ClamAV on emails of 222k were at 300 seconds, wow. It caused duplicate emails and all kinds of craziness, even after adding Spamdyke. I used one of Jakes tips on his video page and cut the scan time down to an average of 110 on the same email. Quite an improvement but still too high to be usable. I moved the VM to a temp machine this week, an old E-Machines computer of all things with an AMD Athlon XP 2000 processor with a whole 1G of memory, and scan times on the same message fell to 17 seconds. Again - still too high, but at least the duplicate emails stopped. I just used another tweak from Jakes video site and the scan time fell to 9 seconds, with the blacklists and ClamAV scans going again. That will work until the replacement server gets here next week. If you are using VMWare, you really need a higher end machine, I think that could be the moral of this story. The first server that was in use, had a sister sitting right next to it that ran the QTP, plus webhost, plus a radius server, and never ever broke a sweat. VMWare is a hog to say the least. Jake's video subscription may be the best thing I have found for some quick instruction on usage of real world tweaks. Jake and Eric are always willing to help, and have great experiences to provide. Also a shout out to Brent for fixing the Spamdyke script - it works. Now if I can figure out how to make the smtp log change at midnight instead of whenever it wants, I will be happy. Great job all of you guys - you really are lifesavers!!! Thanks for sharing. Before I comment on this, I want to check the facts. This was a Quad P-II, and not a Quad P-4, right? Just checking. Not that it would make all that much difference. I'll have some comments to make regarding your experience with VM guests soon. FWIW, I just migrated a QMT from one VM guest to another, on the same host. The former ran nicely (and still does on another host). The new QMT VM is a pig. I'm not sure what the problem is yet. Top takes 10% of the cpu on the pig host, while on another guest w/ same kernel, top runs less than 1%. So somewhere I'm seeing a ~10x performance difference between 2 guests on the same host. It'll be interesting (to say the least) to find the reason why. There are some differences between the 3 guests, but not too many. I hope to nail it soon. -- -Eric 'shubes' - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com
Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
On 02/27/2010 12:08 PM, MagicWISP Sales wrote: Yes the old server was a Quad PII - 4 Intel 400Mhz processors, it ran like a champ with QMT, Apache, Bind, and FreeRadius. As soon as the Apache, Bind, and QMT were rolled into VMWare -it became a dog. I used about every tweak I could find on the internet to make it work better. I think the problem is VMWare Server 2 - it's just slow. Unfortunately I could not use Xen or KVM because of the old processors. From what I could research, they would have worked much better. I am pretty anxious to see how this runs on another server with dual Intel 3.Ghz processors. Unfortunately they aren't dual core, but we should still see some marked improvement. I'm actually testing/using VirtualBox for things right now. I think in the long run that running a large mailserver on any virtual environment is a bad idea. I'll need to verify, but I think even the big official mail server packages are not supported if run in virtual environment and probably for reasons like this. - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com
RE: [qmailtoaster] Re: How things can work and do sort themselves out with a little help
Jake, You are probably right, but we are a small ISP, and never will have a large mail server. If this server ever reaches 500 accounts I will be way more than shocked. So this works well for us. The really nice thing is restoral - it's a snap. If I have a few thousand email accounts, it would be a different story. -Original Message- From: Jake Vickers [mailto:j...@qmailtoaster.com] Sent: Saturday, February 27, 2010 8:16 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Re: How things can work and do sort themselves out with a little help On 02/27/2010 12:08 PM, MagicWISP Sales wrote: Yes the old server was a Quad PII - 4 Intel 400Mhz processors, it ran like a champ with QMT, Apache, Bind, and FreeRadius. As soon as the Apache, Bind, and QMT were rolled into VMWare -it became a dog. I used about every tweak I could find on the internet to make it work better. I think the problem is VMWare Server 2 - it's just slow. Unfortunately I could not use Xen or KVM because of the old processors. From what I could research, they would have worked much better. I am pretty anxious to see how this runs on another server with dual Intel 3.Ghz processors. Unfortunately they aren't dual core, but we should still see some marked improvement. I'm actually testing/using VirtualBox for things right now. I think in the long run that running a large mailserver on any virtual environment is a bad idea. I'll need to verify, but I think even the big official mail server packages are not supported if run in virtual environment and probably for reasons like this. - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com - Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com) Vickers Consulting Group offers Qmailtoaster support and installations. If you need professional help with your setup, contact them today! - Please visit qmailtoaster.com for the latest news, updates, and packages. To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com
