On Thu, 1 Jan 2009, Kostas Georgiou wrote:
> On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:
>
> > On Thu, 1 Jan 2009, James Antill wrote:
> >
> > > On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
> > > > Lets pool some knowledge together because at this point, I'm missing
> > > > something.
> > > >
> > > > I've been doing all measurements with sar as bonnie, etc, causes builds
> > > > to
> > > > timeout.
> > > >
> > > > Problem: We're seeing slower then normal disk IO. At least I think we
> > > > are. This is a PERC5/E and MD1000 array.
> > > >
> > > > When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> > > > around 4-6MBytes/s
> > >
> > > This _might_ not be "IO" in a normal sense, -a to cp means:
> > >
> > > file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
> > >
> > > ...esp. given that you aren't getting large IOWait times, you might want
> > > to strace -T the cp and do some perl/whatever on the result to see what
> > > is eating up the time.
> >
> > Even with non cp type things (like a bacula backup) it just doesn't seem
> > as fast as I would expect it to be. I've never actually done trending at
> > this level / scale on a filesystem / drive before. So I really don't have
> > a good baseline except that it just seems slow to me.
> >
> > Other then the much faster direct block access and the large file reads, I
> > don't have much else to go on that makes me think its slow.
>
> Do writes show the same pattern? If you use selinux/ACLs/xattrs the default
> inode size of 128 can cause slowdowns (#205161 for example).
>
One reason I'm trying to ramp this up now is because the koji share is
still under 50% utilized. If it turns out to be something in the
filesystem, its not too late for us to shrink the main filesystem, create
the new, copy, and grow the new.
> Can you run blktrace+seekwatcher (both in EPEL) to get an idea on
> what is going on? An iostat -x -k /dev/sde 1 output will also be
> helpfull.
>
I'll take a look at those two applications as well, here's the iostat:
Linux 2.6.18-92.1.18.el5xen (xen2.fedora.phx.redhat.com) 01/01/2009
avg-cpu: %user %nice %system %iowait %steal %idle
0.55 0.01 1.35 0.10 6.28 91.71
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz
await svctm %util
sde 1389.22 95.13 161.74 270.46 6693.75 1670.16 38.70 1.09
2.51 1.48 64.04
-Mike
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
