On 1/19/2012 5:07 AM, Konrad Rzepecki wrote: > W dniu 19.01.2012 08:15, Stan Hoeppner pisze: > >> To demonstrate that fsync alone shouldn't be a factor here, > > But it is. I've straced sendmail to "fsync" waiting lot of time. It was > 80% or more of queue time. > >> Queuing 15K messages took 6 minutes 30 seconds on a single 7.2K drive, >> again while competing with the delivery agent for spindle time. > > My previous 1 drive 1 cpu box was also quick. > >> If your kernel is using CFQ you may want to try deadline. > > No change. Also try disable NCQ, set noatime, switch to reiser also - no > effect either. Only change I found on 8 disk boot raid1 (no lvm) which > appear even more slower. > >> But that >> alone isn't going to fix a 10x performance deficit. You've probably got >> multiple factors degrading performance. > > Yes, you have right. But I found recently, that disk mounted on my > server are slow 5.9K. My tests on in shows that they do fsync 1.5x-2x > slower than 7.2K with quite often 5x-10x slower peak. Together with > raid10, lvm, ext4, and much heavier load during delivery it may give > effect I'm observing.
5.9K RPM? Bingo. There's the problem. Those are "green" drives, from one manufacturer or another, probably Western Digital EARS 2TB drives. They are not suitable for RAID use, nor server use, nor any high performance use whatsoever. As you've seen first hand their performance is low, unpredictable, and unreliable. In addition, they are "Advanced Format" drives, meaning 4KB hardware sectors but reported to the host as 512 byte sectors. This can cause stripe alignment problems with RAID and the filesystem, which will exacerbate fsync delays. Normally this misalignment is only an issue with parity arrays but it can also affect non parity striped arrays in certain configurations. Needless to say, you're not going to get decent queue spooling performance with these green drives, ever. If you can't wholesale swap these 8 drives for units suitable for mail server duty, consider sticking two small inexpensive 7.2k SATA drives in the box and mirroring them. Move the Postfix spool directory onto them--and any other applications you're running that need higher random IOPS performance than you're going to get from these green drives. This is likely the least expensive way, in both $$ and effort, to solve your problem. -- Stan
