[DRBD-user] SOLVED - kind of (Fwd: slow drbd over tripple gigabit bonding balance-rr)
turns out, once I did the testing with 1M block size instead of 10M, it was
showing the performance I expected.
sync; echo 3 > /proc/sys/vm/drop_caches # free pagecache, dentries and
inodes
sync;dd if=/dev/zero of=/vz/blob bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 4.04935 seconds, 259 MB/s
I wonder if "max-buffers " and "max-epoch-size" set to 2048 has anything to
do with this.
z
PS: I want to thank Mark for replying to the post.
-- Forwarded message --
From: Zoltan Patay
Date: Thu, Jul 30, 2009 at 3:57 AM
Subject: slow drbd over tripple gigabit bonding balance-rr
To: drbd-user@lists.linbit.com
using "dd if=/dev/zero of=/dev/drbd26 bs=10M count=100" I get:
drbd connected
1048576000 bytes (1.0 GB) copied, 13.6526 seconds, 76.8 MB/s
1048576000 bytes (1.0 GB) copied, 13.4238 seconds, 78.1 MB/s
1048576000 bytes (1.0 GB) copied, 13.2448 seconds, 79.2 MB/s
drbd disconnected
1048576000 bytes (1.0 GB) copied, 4.04754 seconds, 259 MB/s
1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
The three (intel) gigabit PCIe cards are bonded with balance-rr, and iperf
gives me:
iperf 0.0-10.0 sec 2.52 GBytes 2.16 Gbits/sec (276.48MB/s)
So clearly there is enough speed for both on the network and in the backend
to support higher speeds. The boxes are with cross-over back-to-back
no-switch.
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by p...@fat-tyre,
2008-12-18 15:26:13
global { usage-count yes; }
common { syncer { rate 650M; } }
resource OpenVZ_C1C2_B_LVM5 {
protocol C;
startup {degr-wfc-timeout 120;}
disk {on-io-error
detach;no-disk-flushes;no-md-flushes;no-disk-drain;no-disk-barrier;}
net {
cram-hmac-alg sha1;
shared-secret "OpenVZ_C1C2_B";
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
rr-conflict disconnect;
timeout 300;
connect-int 10;
ping-int 10;
max-buffers 2048;
max-epoch-size 2048;
}
syncer {rate 650M;al-extents 257;verify-alg crc32c;}
on c1 {
device /dev/drbd26;
disk /dev/mapper/xenvg-OpenVZ_C1C2_B_LVM5;
address10.0.10.10:7826;
meta-disk /dev/mapper/xenvg-DRBD_MetaDisk[26];
}
on c2 {
device/dev/drbd26;
disk /dev/mapper/xenvg-OpenVZ_C1C2_B_LVM5;
address 10.0.10.20:7826;
meta-disk /dev/mapper/xenvg-DRBD_MetaDisk[26];
}
}
Some of the settings above are unsafe (no-disk-flushes;no-md-flushes), they
were turned on to see if it makes any different (did not)
The two boxes are quad core 3GHz Nehalems, 12GB tripple channel DDR3-1600,
6 western digital caviar black 750GB hdds, in RAID10 with LVM on top of it,
the DRBD backends are carved out of LVM. Three separate Intel gigabit PCIe
cards are bonded with ballance-rr, and connects them back-to-back, with a
forth gigabit card in each box (onboard) toward the outside.
The OS is Debian Etch + Backports with some custom deb packages rolled by
me. The machine is a Xen Dom0, kernel: 2.6.26, xen: 3.2.1, drbd: 8.3.0
Thanks any help / hint in advance,
z
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow drbd over tripple gigabit bonding balance-rr
Mark,
syncer rate does not define the overall drbd performance between the two
nodes, but rather it designates a limit for re-syncing, to assure normal
system performance while it is happening in the backgrounnd (so your re-sync
processes dont eat up all your bandwidth between your drbd nodes, and normal
drbd usage can happen).
It has nothing to do with the working performance of a single drbd device
pair see more here:
http://www.drbd.org/users-guide/s-configure-syncer-rate.html
Originally it was set to be 80M in this case, but probably it should be even
lower, since despite the iperf results, I never saw a drbd go over the
117MB/s (single gigabit link hitting performance wall), so it probably
should be as low as 35MB/s
As I wrote before, the two boxes have three gigabit links dedicated for
DRBD, these are bonded using the balance-rr mode, and with arp ip monitoring
(basically I can unplug any of the cables between the nodes, in any order,
and plug them back in, as long there is a single link, the connection is
uninterrupted, also depending how many gigabit links are allive the
bandwidth scales up with every additional connection, pretty swift actually)
What I would like to see is higher write rates. I know how the different
bondings work, I also know balance-rr is the only where a single connection
cab scale beyond the capacity of a single card, and it clearly happens when
I benchamrk it with iperf, but never saw it in DRBD.
Now, since this is a Xen Dom0, I have been able to do more testing in the
DomU itself (half of the testing was done before I wrote to the mailing
list)
In the paravirtualized DomU, the drbd devices are inported as xvdb to xvdf,
and they were used:
1) as phisical volumes for the LVM in the DomU itseld, setting up the
logical volumes using striping in LVM
2) as part of a raid0 stripe
3) raid0 stripe in above as a phisical volume for LVM
There is no signofocant difference in either mode.
I have even tested a RAID_0 stripe over the drbd volumes in the Xen Dom0,
same results.
I also know about the LVM default block device performance issues, and
"blockdev --setra" is used as a workaround on all levels
Whenever I disconnect drbd, performance is as expected (close to the raid10
performance measured)
The reason this is so annoying, is because on both the DRBD wiki and in the
mail list there are hints that even over a dual gigabit link in ballance-rr
performance is much better, see them yourself:
http://www.drbd.org/home/wiki/?tx_drwiki_pi1[keyword]=performance
http://www.nabble.com/DRBD-Performance-td18745802.html
http://lists.linbit.com/pipermail/drbd-user/2008-July/009893.html
Also if it not clear, this is a nested LVM (with the DRBD's being the
phisical volumes for LVM in the paravirtualized xen instance, DRBD is
running in Dom0):
DRBD<-| |<-xvdb (PV)|
Xen Dom0 6XHDD<-RAID10<-LVM<-DRBD<-|XenDomU|<-xvdc
(PV)|<-LVM<file ystems
DRBD<-| |<-xvdd|(PV)|
As a side note, I am a seasoned sysadmin of fifteen years and use linux
practically for everything for the last ten years, work with it at least
twelve hours a day (mostly more than that, I am lucky to do for living what
I love, and work and fun are the same)
So, anybody knows how those magical numbers were achived under those links?
z
On Thu, Jul 30, 2009 at 9:18 AM, Mark Watts wrote:
> On Thu, 2009-07-30 at 03:57 -0400, Zoltan Patay wrote:
> > using "dd if=/dev/zero of=/dev/drbd26 bs=10M count=100" I get:
> >
> > drbd connected
> > 1048576000 bytes (1.0 GB) copied, 13.6526 seconds, 76.8 MB/s
> > 1048576000 bytes (1.0 GB) copied, 13.4238 seconds, 78.1 MB/s
> > 1048576000 bytes (1.0 GB) copied, 13.2448 seconds, 79.2 MB/s
> >
> > drbd disconnected
> > 1048576000 bytes (1.0 GB) copied, 4.04754 seconds, 259 MB/s
> > 1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
> > 1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
> >
> > The three (intel) gigabit PCIe cards are bonded with balance-rr, and
> > iperf gives me:
> >
> > iperf 0.0-10.0 sec 2.52 GBytes 2.16 Gbits/sec (276.48MB/s)
> >
> > So clearly there is enough speed for both on the network and in the
> > backend to support higher speeds. The boxes are with cross-over
> > back-to-back no-switch.
> >
> > version: 8.3.0 (api:88/proto:86-89)
> > GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by
> > p...@fat-tyre, 2008-12-18 15:26:13
> >
> > global { usage-count yes; }
> >common { syncer { rate 650M; } }
>
> Try actually setting this to a sensible value for 3 x 1Gbit links.
> eg: 300M
>
> > resource OpenVZ_C1C2_B_LVM5 {
> > protocol C;
> > startup {degr-wfc-timeout 120;}
> > disk {on-io-e
[DRBD-user] slow drbd over tripple gigabit bonding balance-rr
using "dd if=/dev/zero of=/dev/drbd26 bs=10M count=100" I get:
drbd connected
1048576000 bytes (1.0 GB) copied, 13.6526 seconds, 76.8 MB/s
1048576000 bytes (1.0 GB) copied, 13.4238 seconds, 78.1 MB/s
1048576000 bytes (1.0 GB) copied, 13.2448 seconds, 79.2 MB/s
drbd disconnected
1048576000 bytes (1.0 GB) copied, 4.04754 seconds, 259 MB/s
1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s
The three (intel) gigabit PCIe cards are bonded with balance-rr, and iperf
gives me:
iperf 0.0-10.0 sec 2.52 GBytes 2.16 Gbits/sec (276.48MB/s)
So clearly there is enough speed for both on the network and in the backend
to support higher speeds. The boxes are with cross-over back-to-back
no-switch.
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by p...@fat-tyre,
2008-12-18 15:26:13
global { usage-count yes; }
common { syncer { rate 650M; } }
resource OpenVZ_C1C2_B_LVM5 {
protocol C;
startup {degr-wfc-timeout 120;}
disk {on-io-error
detach;no-disk-flushes;no-md-flushes;no-disk-drain;no-disk-barrier;}
net {
cram-hmac-alg sha1;
shared-secret "OpenVZ_C1C2_B";
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
rr-conflict disconnect;
timeout 300;
connect-int 10;
ping-int 10;
max-buffers 2048;
max-epoch-size 2048;
}
syncer {rate 650M;al-extents 257;verify-alg crc32c;}
on c1 {
device /dev/drbd26;
disk /dev/mapper/xenvg-OpenVZ_C1C2_B_LVM5;
address10.0.10.10:7826;
meta-disk /dev/mapper/xenvg-DRBD_MetaDisk[26];
}
on c2 {
device/dev/drbd26;
disk /dev/mapper/xenvg-OpenVZ_C1C2_B_LVM5;
address 10.0.10.20:7826;
meta-disk /dev/mapper/xenvg-DRBD_MetaDisk[26];
}
}
Some of the settings above are unsafe (no-disk-flushes;no-md-flushes), they
were turned on to see if it makes any different (did not)
The two boxes are quad core 3GHz Nehalems, 12GB tripple channel DDR3-1600,
6 western digital caviar black 750GB hdds, in RAID10 with LVM on top of it,
the DRBD backends are carved out of LVM. Three separate Intel gigabit PCIe
cards are bonded with ballance-rr, and connects them back-to-back, with a
forth gigabit card in each box (onboard) toward the outside.
The OS is Debian Etch + Backports with some custom deb packages rolled by
me. The machine is a Xen Dom0, kernel: 2.6.26, xen: 3.2.1, drbd: 8.3.0
Thanks any help / hint in advance,
z
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
