On 05/30/2010 04:22 PM, Richard Elling wrote: > If you want to decouple the txg commit completely, then you might consider > using a buffer of some sort. I use mbuffer for pipes, but that may be tricky > to use in an rsync environment. > -- richard > I initially thought this was I/O but now I think it might be a CPU bottleneck causing the problem.
When I write zero's using DD I can get quite good speed (~500 megabytes/sec) but I think the problem is that it totally maxes both of my CPU's cores when I do so (2-5% idle in top). The problem is that when it does the write burst its taking away CPU usage from rsync which is actually what might be causing the dip during writes (not the I/O activity itself) but the CPU generated from the writes. I actually verified this by making a non raidz2 pool and just added all the disks into a pool with nothing specified: zpool create data c4t5000C500028BD5FCd0p0 c4t5000C50009A4D727d0p0 c4t5000C50009A46AF5d0p0 c4t5000C50009A515B0d0p0 c4t5000C500028A81BEd0p0 c4t5000C500028B44A1d0p0 c4t5000C500028B415Bd0p0 c4t5000C500028B23D2d0p0 c4t5000C5000CC3338Dd0p0 c4t5000C500027F59C8d0p0 c4t5000C50009DBF8D4d0p0 c4t5000C500027F3C1Fd0p0 c4t5000C5000DAF02F3d0p0 c4t5000C5000DA7ED4Ed0p0 c4t5000C5000DAEF990d0p0 c4t5000C5000DAEEF8Ed0p0 c4t5000C5000DAEB881d0p0 c4t5000C5000A121581d0p0 c4t5000C5000DAC848Fd0p0 c4t5000C50002770EE6d0p0 Once I did this I got a nice stable 115 megabyte/ssec over the network so 11.5% better. So the problem appears to be that when it goes to write data it will use 100% of its cpu power (even for x number of ms of time) which stalls the network while it does this. This happens when it does parity calculations but doesn't when using a non parity zpool. I don't think I can throttle parity calculations at all so I don't think there will be a fix to this unfortunately =(. I can live with losing 10% off my rsync speed though. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
