On Thu, May 24, 2012 at 03:34:51PM +0300, Dan Frincu wrote: > Hi, > > On Mon, May 21, 2012 at 4:24 PM, Christoph Bartoschek <bartosc...@gmx.de> > wrote: > > Florian Haas wrote: > > > >>> Thus I would expect to have a write performance of about 100 MByte/s. But > >>> dd gives me only 20 MByte/s. > >>> > >>> dd if=/dev/zero of=bigfile.10G bs=8192 count=1310720 > >>> 1310720+0 records in > >>> 1310720+0 records out > >>> 10737418240 bytes (11 GB) copied, 498.26 s, 21.5 MB/s > >> > >> If you used that same dd invocation for your local test that allegedly > >> produced 450 MB/s, you've probably been testing only your page cache. > >> Add oflag=dsync or oflag=direct (the latter will only work locally, as > >> NFS doesn't support O_DIRECT). > >> > >> If your RAID is one of reasonably contemporary SAS or SATA drives, > >> then a sustained to-disk throughput of 450 MB/s would require about > >> 7-9 stripes in a RAID-0 or RAID-10 configuration. Is that what you've > >> got? Or are you writing to SSDs? > > > > I used the same invocation with different filenames each time. To which page > > cache to you refer? To the one on the client or on the server side? > > > > We are using RAID-1 with 6 x 2 disks. I have repeated the local test 10 > > times with different files in a row: > > > > for i in `seq 10`; do time dd if=/dev/zero of=bigfile.10G.$i bs=8192 > > count=1310720; done > > > > The resulting values on a system that is also used by other programs as > > reported by dd are: > > > > 515 MB/s, 480 MB/s, 340 MB/s, 338 MB/s, 360 MB/s, 284 MB/s, 311 MB/s, 320 > > MB/s, 242 MB/s, 289 MB/s > > > > So I think that the system is capable of more than 200 MB/s which is way > > more what can arrive over the network. > > A bit off-topic maybe. > > Whenever you do these kinds of tests regarding performance on disk > (locally) to test actual speed and not some caching, as Florian said, > you should use oflag=direct option to dd and also echo 3 > > /proc/sys/vm/drop_caches and sync. >
You should sync before you drop caches, or you won't drop those caches that have been dirty at that time. > I usually use echo 3 > /proc/sys/vm/drop_caches && sync && date && > time dd if=/dev/zero of=whatever bs=1G count=x oflag=direct && sync && > date > > You can assess if there is data being flushed if the results given by > dd differ from those obtained by calculating the amount of data > written between the two date calls. It also helps to push more data > than the controller can store. Also, dd is doing one bs sized chunk at a time. fio with appropriate options can be more useful, once you learned all those options, and how to interpret the results... > Regards, > Dan > > > > > I've done the measurements on the filesystem that sits on top of LVM and > > DRBD. Thus I think that DRBD is not a problem. > > > > However the strange thing is that I get 108 MB/s on the clients as soon as I > > disable the secondary node for DRBD. Maybe there is strange interaction > > between DRBD and NFS. Dedicated replication link? Maybe the additional latency is all that kills you. Do you have non-volatile write cache on your IO backend? Did you post your drbd configuration setings already? > > > > After reenabling the secondary node the DRBD synchronization is quite slow. > > > > > >>> > >>> Has anyone an idea what could cause such problems? I have no idea for > >>> further analysis. > >> > >> As a knee-jerk response, that might be the classic issue of NFS > >> filling up the page cache until it hits the vm.dirty_ratio and then > >> having a ton of stuff to write to disk, which the local I/O subsystem > >> can't cope with. > > > > Sounds reasonable but shouldn't the I/O subsystem be capable to write > > anything away that arrives? > > > > Christoph -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
