Re: [DRBD-user] slow sync speed
Just a quick note .. You are correct, it shouldn't be required (v8.9.10) and I was surprised > with that too. > In the DRBD documentation, it is stated that ... "When multiple DRBD resources share a single replication/synchronization network, synchronization with a fixed rate may not be an optimal approach. So, in DRBD 8.4.0 the variable-rate synchronization was enabled by default." ..and.. "In a few, very restricted situations[4], it might make sense to just use some fixed synchronization rate. In this case, first of all you need to turn the dynamic sync rate controller off, by using c-plan-ahead 0;." ..by observing your configuration, it looks like you added that option since the first time, hence no surprises here, you explicitly decided to disable variable sync rate ... :) ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
You are correct, it shouldn't be required (v8.9.10) and I was surprised with that too. Another evidence of the option being honored is "want: 150,000 k/sec" which I sometimes (not always) see in /proc/drbd On 17/10/18 10:17, Oleksiy Evin wrote: If I'm not wrong, the "syncer" section has been deprecated somewhere around 8.4.0 drbd version. Based on the logs you provided the version you use is 8.4.10, so I don't think that should have any speed impact. But I'm glad you've got it resolved. //OE ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
If I'm not wrong, the "syncer" section has been deprecated somewhere around
8.4.0 drbd version. Based on the logs you provided the version you use is
8.4.10, so I don't think that should have any speed impact. But I'm glad you've
got it resolved.
//OE
-Original Message-
From: Adam Weremczuk
To: Robert Altnoeder
Cc: drbd-user@lists.linbit.com
Subject: Re: [DRBD-user] slow sync speed
Date: Wed, 17 Oct 2018 10:05:10 +0100
"Max-buffers 8k" appear to be the sweet spot for me.I'm now getting 145-150
MB/s transfer rates between nodes which I'm happy with.The biggest problem was
I didn't have "syncer" section defined at all.
Currently my fully working and behaving config looks like below:
global { usage-count no; }common { protocol C; }resource r0 { disk {
on-io-error detach; no-disk-flushes; no-disk-barrier; c-plan-ahead
0; } net { max-buffers 8k; } syncer { rate 150M; al-extents
6400; } on lion { device /dev/drbd0; disk /dev/sdb1; address
192.168.200.1:7788; meta-disk internal; } on tiger { device
/dev/drbd0; disk /dev/sdb1; address 192.168.200.2:7788; meta-disk
internal; }}
On 11/10/18 15:06, Robert Altnoeder wrote:On 10/11/2018 03:56 PM, Oleksiy Evin
wrote:Try to remove the following:
c-fill-target 24M;c-min-rate 80M;c-max-rate 720M;
sndbuf-size 1024k;rcvbuf-size 2048k;
Then gradually increase max-buffers from 4K to 12K checking its impactto the
sync speed. Make sure you have the same config on both nodesand apply the
changes with "drbdadm adjust all" on both nodes too.
___drbd-user mailing
listdrbd-user@lists.linbit.comhttp://lists.linbit.com/mailman/listinfo/drbd-user
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
"Max-buffers 8k" appear to be the sweet spot for me.
I'm now getting 145-150 MB/s transfer rates between nodes which I'm
happy with.
The biggest problem was I didn't have "syncer" section defined at all.
Currently my fully working and behaving config looks like below:
global { usage-count no; }
common { protocol C; }
resource r0 {
disk {
on-io-error detach;
no-disk-flushes;
no-disk-barrier;
c-plan-ahead 0;
}
net {
max-buffers 8k;
}
syncer {
rate 150M;
al-extents 6400;
}
on lion {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.200.1:7788;
meta-disk internal;
}
on tiger {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.200.2:7788;
meta-disk internal;
}
}
On 11/10/18 15:06, Robert Altnoeder wrote:
On 10/11/2018 03:56 PM, Oleksiy Evin wrote:
Try to remove the following:
c-fill-target 24M;
c-min-rate 80M;
c-max-rate 720M;
sndbuf-size 1024k;
rcvbuf-size 2048k;
Then gradually increase max-buffers from 4K to 12K checking its impact
to the sync speed. Make sure you have the same config on both nodes
and apply the changes with "drbdadm adjust all" on both nodes too.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
On 10/11/2018 03:56 PM, Oleksiy Evin wrote: > Try to remove the following: > > c-fill-target 24M; > c-min-rate 80M; > c-max-rate 720M; > > sndbuf-size 1024k; > rcvbuf-size 2048k; > > Then gradually increase max-buffers from 4K to 12K checking its impact > to the sync speed. Make sure you have the same config on both nodes > and apply the changes with "drbdadm adjust all" on both nodes too. > > Also, I guess you probably expect c-min-rate to do something completely different from what it actually does - might want to read the manual for that option. Cheers, Robert ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
Try to remove the following:
c-fill-target 24M;
c-min-rate 80M;
c-max-rate 720M;
sndbuf-size 1024k;
rcvbuf-size 2048k;
Then gradually increase max-buffers from 4K to 12K checking its impact to the
sync speed. Make sure you have the same config on both nodes and apply the
changes with "drbdadm adjust all" on both nodes too.
//OE
On Oct 11, 2018, 21:43, at 21:43, Adam Weremczuk
wrote:
>Hi Oleksiy,
>
>Here is my entire current config:
>
>global { usage-count no; }
>common { protocol C; }
>resource r0 {
> disk {
> on-io-error detach;
> no-disk-flushes ;
> no-disk-barrier;
> c-plan-ahead 0;
> c-fill-target 24M;
> c-min-rate 80M;
> c-max-rate 720M;
> }
> net {
> max-buffers 36k;
> sndbuf-size 1024k;
> rcvbuf-size 2048k;
> }
> on node1 {
> device /dev/drbd0;
> disk /dev/sdb1;
> address 192.168.200.1:7788;
> meta-disk internal;
> }
> on node2 {
> device /dev/drbd0;
> disk /dev/sdb1;
> address 192.168.200.2:7788;
> meta-disk internal;
> }
>}
>
>...and the speed is still dreadfully slow, even though the link can
>easily do 150MB/s:
>
>cat /proc/drbd
>version: 8.4.10 (api:1/proto:86-101)
>srcversion: 17A0C3A0AF9492ED4B9A418
> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
>r-
> ns:46712 nr:0 dw:0 dr:47528 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d
>
>oos:3898265768
> [>] sync'ed: 0.1% (3806900/3806944)M
> finish: 3867:19:37 speed: 264 (260) K/sec
>
>Anything else to try before giving up on DRBD?
>
>Thanks,
>Adam
>
>
>On 11/10/18 07:45, Oleksiy Evin wrote:
>> You may try to disable dynamic sync rate by setting "c-plan-ahead" to
>
>> 0 and increase "max-buffers". That's the only way which helped me to
>> get reasonable sync rate with 100GigE connection.
>>
>> net {
>> ...
>> max-buffers 32K;
>> # max-epoch-size 18K;
>> }
>>
>> disk {
>> ...
>> c-plan-ahead 0;
>> }
>>
>> You can find some more info here:
>>
>https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370
>>
>>
>> //OE
>>
>> -Original Message-
>> *From*: Adam Weremczuk > <mailto:adam%20weremczuk%20%3cad...@matrixscience.com%3e>>
>> *To*: drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
>> *Subject*: [DRBD-user] slow sync speed
>> *Date*: Wed, 10 Oct 2018 14:57:02 +0100
>>
>> Hi all,
>> I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
>> I have 2 identical servers connected with 2 x 1 Gbps links in
>bond_mode
>> balance-rr.
>> The bond is working fine; I get a transfer rate of 150 MB/s with scp.
>> Following this guide:
>>
>https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
>
>> was going smoothly up until:
>> drbdadm -- --overwrite-data-of-peer primary r0/0
>> cat /proc/drbd
>> version: 8.4.10 (api:1/proto:86-101)
>> srcversion: 17A0C3A0AF9492ED4B9A418
>> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
>r-
>> ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
>wo:f
>> oos:3898301536
>> [>] sync'ed: 0.1% (3806932/3806944)M
>> finish: 483:25:13 speed: 2,188 (2,188) K/sec
>> The transfer rate is horribly slow and at this pace it's going to
>take
>> 20 days for two 4 TB volumes to sync!
>> That's almost 15 times slower comparing with the guide video (8:30):
>> https://www.youtube.com/watch?v=WQGi8Nf0kVc
>> The volumes have been zeroed and contain no live data yet.
>> My sdb disks are logical drives (hardware RAID) set up as RAID50 with
>> the defaults:
>> Strip size: 128 KB
>> Access policy: RW
>> Read policy: Normal
>> Write policy: Write Back with BBU
>> IO policy: Direct
>> Drive Cache: Disable
>> Disable BGI: No
>> Performance looks good when tested with hdparm:
>> hdparm -tT /dev/sdb1
>> /dev/sdb1:
>> Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec
>> Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81
>MB/sec
>> The volumes have been zeroed and contain no live data yet.
>> It seems to be a problem with default DRBD settings.
>> Can anybody recommend optimal tweaks specific to my environment?
>> Regards,
>> Adam
>> ___
>> drbd-user mailing list
>> drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
>> http://lists.linbit.com/mailman/listinfo/drbd-user
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
Hi Oleksiy,
Here is my entire current config:
global { usage-count no; }
common { protocol C; }
resource r0 {
disk {
on-io-error detach;
no-disk-flushes ;
no-disk-barrier;
c-plan-ahead 0;
c-fill-target 24M;
c-min-rate 80M;
c-max-rate 720M;
}
net {
max-buffers 36k;
sndbuf-size 1024k;
rcvbuf-size 2048k;
}
on node1 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.200.1:7788;
meta-disk internal;
}
on node2 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.200.2:7788;
meta-disk internal;
}
}
...and the speed is still dreadfully slow, even though the link can
easily do 150MB/s:
cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-
ns:46712 nr:0 dw:0 dr:47528 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d
oos:3898265768
[>] sync'ed: 0.1% (3806900/3806944)M
finish: 3867:19:37 speed: 264 (260) K/sec
Anything else to try before giving up on DRBD?
Thanks,
Adam
On 11/10/18 07:45, Oleksiy Evin wrote:
You may try to disable dynamic sync rate by setting "c-plan-ahead" to
0 and increase "max-buffers". That's the only way which helped me to
get reasonable sync rate with 100GigE connection.
net {
...
max-buffers 32K;
# max-epoch-size 18K;
}
disk {
...
c-plan-ahead 0;
}
You can find some more info here:
https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370
//OE
-Original Message-
*From*: Adam Weremczuk <mailto:adam%20weremczuk%20%3cad...@matrixscience.com%3e>>
*To*: drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
*Subject*: [DRBD-user] slow sync speed
*Date*: Wed, 10 Oct 2018 14:57:02 +0100
Hi all,
I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode
balance-rr.
The bond is working fine; I get a transfer rate of 150 MB/s with scp.
Following this guide:
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
was going smoothly up until:
drbdadm -- --overwrite-data-of-peer primary r0/0
cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-
ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:3898301536
[>] sync'ed: 0.1% (3806932/3806944)M
finish: 483:25:13 speed: 2,188 (2,188) K/sec
The transfer rate is horribly slow and at this pace it's going to take
20 days for two 4 TB volumes to sync!
That's almost 15 times slower comparing with the guide video (8:30):
https://www.youtube.com/watch?v=WQGi8Nf0kVc
The volumes have been zeroed and contain no live data yet.
My sdb disks are logical drives (hardware RAID) set up as RAID50 with
the defaults:
Strip size: 128 KB
Access policy: RW
Read policy: Normal
Write policy: Write Back with BBU
IO policy: Direct
Drive Cache: Disable
Disable BGI: No
Performance looks good when tested with hdparm:
hdparm -tT /dev/sdb1
/dev/sdb1:
Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec
Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec
The volumes have been zeroed and contain no live data yet.
It seems to be a problem with default DRBD settings.
Can anybody recommend optimal tweaks specific to my environment?
Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
http://lists.linbit.com/mailman/listinfo/drbd-user
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
You may try to disable dynamic sync rate by setting "c-plan-ahead" to 0 and
increase "max-buffers". That's the only way which helped me to get reasonable
sync rate with 100GigE connection.
net {
...
max-buffers 32K;
# max-epoch-size 18K;
}
disk {
...
c-plan-ahead 0;
}
You can find some more info here:
https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370
//OE
-Original Message-
From: Adam Weremczuk
To: drbd-user@lists.linbit.com
Subject: [DRBD-user] slow sync speed
Date: Wed, 10 Oct 2018 14:57:02 +0100
Hi all,
I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode
balance-rr.
The bond is working fine; I get a transfer rate of 150 MB/s with scp.
Following this guide:
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/ was
going smoothly up until:
drbdadm -- --overwrite-data-of-peer primary r0/0
cat /proc/drbdversion: 8.4.10 (api:1/proto:86-101)srcversion:
17A0C3A0AF9492ED4B9A418 0: cs:SyncSource ro:Primary/Secondary
ds:UpToDate/Inconsistent C r- ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0
lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3898301536 [>]
sync'ed: 0.1% (3806932/3806944)M finish: 483:25:13 speed: 2,188 (2,188)
K/sec
The transfer rate is horribly slow and at this pace it's going to take 20 days
for two 4 TB volumes to sync!
That's almost 15 times slower comparing with the guide video (8:30):
https://www.youtube.com/watch?v=WQGi8Nf0kVc
The volumes have been zeroed and contain no live data yet.
My sdb disks are logical drives (hardware RAID) set up as RAID50 with the
defaults:
Strip size: 128 KBAccess policy: RWRead policy: NormalWrite policy: Write Back
with BBUIO policy: DirectDrive Cache: DisableDisable BGI: No
Performance looks good when tested with hdparm:
hdparm -tT /dev/sdb1
/dev/sdb1: Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec
Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec
The volumes have been zeroed and contain no live data yet.
It seems to be a problem with default DRBD settings.
Can anybody recommend optimal tweaks specific to my environment?
Regards,Adam
___drbd-user mailing
listdrbd-user@lists.linbit.comhttp://lists.linbit.com/mailman/listinfo/drbd-user
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] slow sync speed
Primary is mounted?. I've got similar behavior on my setup, and just when I've umounted the disk it started to sync very fast. Greetings. El jue., 11 oct. 2018 8:29, Adam Weremczuk escribió: > Hi all, > > I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2 > > I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode > balance-rr. > > The bond is working fine; I get a transfer rate of 150 MB/s with scp. > > Following this guide: > https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/ > was going smoothly up until: > > drbdadm -- --overwrite-data-of-peer primary r0/0 > > cat /proc/drbd > version: 8.4.10 (api:1/proto:86-101) > srcversion: 17A0C3A0AF9492ED4B9A418 > 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r- > ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f > oos:3898301536 > [>] sync'ed: 0.1% (3806932/3806944)M > finish: 483:25:13 speed: 2,188 (2,188) K/sec > > The transfer rate is horribly slow and at this pace it's going to take > 20 days for two 4 TB volumes to sync! > > That's almost 15 times slower comparing with the guide video (8:30): > https://www.youtube.com/watch?v=WQGi8Nf0kVc > > The volumes have been zeroed and contain no live data yet. > > My sdb disks are logical drives (hardware RAID) set up as RAID50 with > the defaults: > > Strip size: 128 KB > Access policy: RW > Read policy: Normal > Write policy: Write Back with BBU > IO policy: Direct > Drive Cache: Disable > Disable BGI: No > > Performance looks good when tested with hdparm: > > hdparm -tT /dev/sdb1 > > /dev/sdb1: > Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec > Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec > > The volumes have been zeroed and contain no live data yet. > > It seems to be a problem with default DRBD settings. > > Can anybody recommend optimal tweaks specific to my environment? > > Regards, > Adam > > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] slow sync speed
Hi all, I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2 I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode balance-rr. The bond is working fine; I get a transfer rate of 150 MB/s with scp. Following this guide: https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/ was going smoothly up until: drbdadm -- --overwrite-data-of-peer primary r0/0 cat /proc/drbd version: 8.4.10 (api:1/proto:86-101) srcversion: 17A0C3A0AF9492ED4B9A418 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r- ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3898301536 [>] sync'ed: 0.1% (3806932/3806944)M finish: 483:25:13 speed: 2,188 (2,188) K/sec The transfer rate is horribly slow and at this pace it's going to take 20 days for two 4 TB volumes to sync! That's almost 15 times slower comparing with the guide video (8:30): https://www.youtube.com/watch?v=WQGi8Nf0kVc The volumes have been zeroed and contain no live data yet. My sdb disks are logical drives (hardware RAID) set up as RAID50 with the defaults: Strip size: 128 KB Access policy: RW Read policy: Normal Write policy: Write Back with BBU IO policy: Direct Drive Cache: Disable Disable BGI: No Performance looks good when tested with hdparm: hdparm -tT /dev/sdb1 /dev/sdb1: Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec The volumes have been zeroed and contain no live data yet. It seems to be a problem with default DRBD settings. Can anybody recommend optimal tweaks specific to my environment? Regards, Adam ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
