[CentOS] RAID on Email Server
I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. on Centos 4.x 32bit. On occasion I have disk I/O problems. Its handling several domains and alot of email. Its currently on a single SATA drive. I am thinking of moving too 3 drives with RAID 1 for redundancy. RAID 1 will help me on reads but do nothing on writes as I understand. I am thinking the majority of my I/O is read though not? I imagine quotta checks and all that being done and everytime a user checks there email every message in the inbox must be read. I guess I am asking if RAID 1 will help my I/O problem much? [EMAIL PROTECTED] ~]# w 12:04:02 up 2:01, 1 user, load average: 7.02, 7.47, 11.84 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0208.92.169.4.ppp 11:250.00s 0.02s 0.00s w [EMAIL PROTECTED] ~]# vmstat procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 0 2 0 1558496 456916 108722400 198 749 795 537 18 4 27 50 The above is when its running pretty good. Matt ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
Matt wrote: I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. on Centos 4.x 32bit. On occasion I have disk I/O problems. Its handling several domains and alot of email. Its currently on a single SATA drive. I am thinking of moving too 3 drives with RAID 1 for redundancy. RAID 1 will help me on reads but do nothing on writes as I understand. I am thinking the majority of my I/O is read though not? I imagine quotta checks and all that being done and everytime a user checks there email every message in the inbox must be read. I guess I am asking if RAID 1 will help my I/O problem much? [EMAIL PROTECTED] ~]# w 12:04:02 up 2:01, 1 user, load average: 7.02, 7.47, 11.84 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0208.92.169.4.ppp 11:250.00s 0.02s 0.00s w [EMAIL PROTECTED] ~]# vmstat procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 0 2 0 1558496 456916 108722400 198 749 795 537 18 4 27 50 The above is when its running pretty good. can you paste the output of `iostat -x 5 5` while its busy ? this will show definateively how busy your disks are... the first sample from vmstat, iostat, etc only shows the AVERAGE since the system booted. the 2nd and beyond samples are the average over the time intervals specified (5 5 means 5 seconds, 5 samples) oh, if you don't have iostat, its part of package sysstat, so `yum install sysstat` ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. on Centos 4.x 32bit. On occasion I have disk I/O problems. Its handling several domains and alot of email. Its currently on a single SATA drive. I am thinking of moving too 3 drives with RAID 1 for redundancy. RAID 1 will help me on reads but do nothing on writes as I understand. I am thinking the majority of my I/O is read though not? I imagine quotta checks and all that being done and everytime a user checks there email every message in the inbox must be read. I guess I am asking if RAID 1 will help my I/O problem much? [EMAIL PROTECTED] ~]# w 12:04:02 up 2:01, 1 user, load average: 7.02, 7.47, 11.84 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0208.92.169.4.ppp 11:250.00s 0.02s 0.00s w [EMAIL PROTECTED] ~]# vmstat procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 0 2 0 1558496 456916 108722400 198 749 795 537 18 4 27 50 The above is when its running pretty good. can you paste the output of `iostat -x 5 5` while its busy ? this will show definateively how busy your disks are... the first sample from vmstat, iostat, etc only shows the AVERAGE since the system booted. the 2nd and beyond samples are the average over the time intervals specified (5 5 means 5 seconds, 5 samples) oh, if you don't have iostat, its part of package sysstat, so `yum install sysstat` Right now its running pretty good but here it is. [EMAIL PROTECTED] ~]# w 13:11:02 up 3:08, 2 users, load average: 4.03, 5.71, 5.51 avg-cpu: %user %nice%sys %iowait %idle 2.800.001.60 58.10 37.50 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.60 142.80 67.20 170.20 678.40 2292.80 339.20 1146.4012.52 118.53 615.66 4.21 99.92 sda1 0.00 0.00 0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.60 142.80 67.20 170.20 678.40 2292.80 339.20 1146.4012.52 118.53 615.66 4.21 99.92 sdb 0.00 0.00 0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 67.40 286.20 678.40 2289.60 339.20 1144.80 8.39 163.02 582.40 2.83 99.94 dm-1 0.00 0.00 0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sar also. 10:02:56 AM LINUX RESTART 10:10:04 AM CPU %user %nice %system %iowait %idle 10:20:01 AM all 13.87 0.00 2.45 59.20 24.49 10:30:03 AM all 22.26 0.00 3.68 53.51 20.54 10:40:01 AM all 20.58 0.00 3.78 55.40 20.24 10:50:04 AM all 22.20 0.00 5.23 52.67 19.91 11:00:05 AM all 21.58 0.00 4.72 51.81 21.89 11:10:01 AM all 18.14 0.00 4.52 56.91 20.43 11:20:03 AM all 21.42 0.00 4.59 47.20 26.79 11:30:02 AM all 19.22 0.00 4.48 53.86 22.44 11:40:04 AM all 17.59 0.00 4.82 51.61 25.98 11:50:02 AM all 15.88 0.00 4.67 45.74 33.71 12:00:01 PM all 13.32 0.00 2.73 25.72 58.23 12:10:02 PM all 16.98 0.00 4.54 53.14 25.35 12:20:01 PM all 17.31 0.00 3.45 47.80 31.44 12:30:01 PM all 19.45 0.00 4.08 36.47 40.00 12:40:01 PM all 13.79 0.00 4.39 44.83 36.99 12:50:01 PM all 12.18 0.00 3.93 30.16 53.73 01:00:01 PM all 11.53 0.00 2.38 20.96 65.12 Average: all 17.49 0.00 4.03 46.29 32.19 A while after the reboot it straightened its self out. Yesterday w was indicatining load average of like 120 or more at times. Today after reboot all is good. Matt ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
Matt wrote: Right now its running pretty good but here it is. Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.60 142.80 67.20 170.20 678.40 2292.80 339.20 1146.4012.52 118.53 615.66 4.21 99.92 You have about 3 times more writing going on than reading. RAID1 isn't going to do much for you. You might want to try four disks in a RAID10 instead. And put your mail folders and spool on partitions mounted using 'noatime'. -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
Matt schrieb: I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. on Centos 4.x 32bit. On occasion I have disk I/O problems. Its handling several domains and alot of email. Its currently on a single SATA drive. I am thinking of moving too 3 drives with RAID 1 for redundancy. RAID 1 will help me on reads but do nothing on writes as I understand. I am thinking the majority of my I/O is read though not? I imagine quotta checks and all that being done and everytime a user checks there email every message in the inbox must be read. I guess I am asking if RAID 1 will help my I/O problem much? [EMAIL PROTECTED] ~]# w 12:04:02 up 2:01, 1 user, load average: 7.02, 7.47, 11.84 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0208.92.169.4.ppp 11:250.00s 0.02s 0.00s w [EMAIL PROTECTED] ~]# vmstat procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 0 2 0 1558496 456916 108722400 198 749 795 537 18 4 27 50 The above is when its running pretty good. Matt ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos How many concurrent users? If you stay with SATA, you will probably have to increase the amount of disks in you storage-array (depending on the number of users). A SATA disk does only 80 I/Os per second or so - and clamav+spamassassin alone will consume a lot of these. Try to put their working-directories on a swap-backed tmpfs. cheers, Rainer ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
Matt wrote: I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. Firstly, I'd drop dovecot completely, cyrus-imapd has, for me, been a lot faster and better optimised for situations where you have more than a few hand full of users. Secondly, exim configs out of the box on CentOS are *really* not optimized for performance at all. If you handle more than a few thousand emails an hour, I dont be surprised if exim is to blame for jamming your i/o pipe just trying to work out which email to attempt delivery for next. -- Karanbir Singh : http://www.karan.org/ : [EMAIL PROTECTED] ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
Matt wrote: Right now its running pretty good but here it is. 10:10:04 AM CPU %user %nice %system %iowait %idle 10:20:01 AM all 13.87 0.00 2.45 59.20 24.49 10:30:03 AM all 22.26 0.00 3.68 53.51 20.54 10:40:01 AM all 20.58 0.00 3.78 55.40 20.24 [ Stuff Deleted ... ] 01:00:01 PM all 11.53 0.00 2.38 20.96 65.12 Average: all 17.49 0.00 4.03 46.29 32.19 A while after the reboot it straightened its self out. Yesterday w was indicatining load average of like 120 or more at times. Today after reboot all is good. Matt Hi Matt, Your %iowait seems high. I had %iowait comparable to you with a single 200 Gigs 7200 RPM IDE drive. Now we've upgraded this server: Opteron 2216 with 4 Gigs RAM, CentOS 64 v5.2) and an Adaptec 3405 plus 4 x 73 Gigs Seagate 15K RPM (RAID 10). No more %iowait! I benched it at about 140 MB/s and random r/w are very good on these drives. We have 55 employees running Outlook connecting to Communicate with MAPI. The dataset is about 100 Gigs. Communigate use MBOX files for storage. Guy Boisvert, ing. IngTegration inc. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID on Email Server
I have an email server running Exim, Dovecot, Spamassassin, Clam, etc. Firstly, I'd drop dovecot completely, cyrus-imapd has, for me, been a lot faster and better optimised for situations where you have more than a few hand full of users. Secondly, exim configs out of the box on CentOS are *really* not optimized for performance at all. If you handle more than a few thousand emails an hour, I dont be surprised if exim is to blame for jamming your i/o pipe just trying to work out which email to attempt delivery for next. This is a Directadmin type server. So I really have no options on changing services. I am running exim version 4.60 and I wander if updating to 4.69 may help out. Matt ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
