linstor-controller 1.28.0 hangs after startup

[Adam Goryachev]

Hi all, hoping to get some ideas on how to investigate this issue 
further, including as many details as I can below...


I've been a bit lax, and had a few issues with my linstor cluster 
recently, one of which was I ended up with some mis-matched versions on 
one or two satellites (ie, 
linstor-client/python-linstor/linstor-satellite) although the 
linstor-satellite versions all matched and all nodes were online and 
working.


Except, I also had an issue where I tried to resize (grow) one volume, 
and this was producing error reports on the satellite about the drbd res 
files not being valid, as well as the return code from a drbdadm command 
as 5 instead of what it should be (this seemed to be due to using a 
newer version of drbdadm compared to linstor-controller, although there 
was no actual error message, just a warning about timeout values was the 
output).


So, I thought I should do the obvious, and upgrade everything to the 
latest versions, to make sure drbd-utils, drbd, linstor-controller, 
linstor-client, python-linstor etc all matched across all nodes. I 
started with the controller node, and it seemed to go smoothly, showing 
all nodes as offline/version mismatch when done. I proceeded with the 
first diskful satellite, and it came online, so I moved onto the next. 
When I checked linstor node list, the controller was not responding, so 
after some time, I restarted the controller service, and it all was 
working as expected (3 nodes online, and the rest offline/version mismatch).


However, after a short amount of time, the linstor controller just stops 
responding again, with basic commands like "linstor node list" timing 
out after 5 minutes.


At this point, I'm not seeing any new logs, or anything at all regarding 
linstor-controller. So I'm not really sure what the issue is, or why it 
seems to be stuck, so any advice is of course appreciated.


I'll include the output from journalctl which shows all the startup 
messages, but not sure what else I should include or where else to look. 
Also, I'm concerned about the status of some of the resources, since I 
can't really see what is happening with them, although using drbdadm 
status I'm seeing mostly what I expect, where all resources are showing 
2 nodes in-sync, except for the original resize one I mentioned above 
which is showing this:


The one diskfull satellite not yet upgraded (san6):

crossbow role:Secondary
  disk:UpToDate
  castle role:Secondary
    peer-disk:UpToDate
  flail connection:Connecting
  san7 connection:Connecting
  xen1 role:Primary
    peer-disk:Diskless peer-client:yes

The controller:

crossbow role:Secondary
  disk:UpToDate
  flail connection:Connecting
  san5 role:Secondary
    peer-disk:Inconsistent
  san6 role:Secondary
    peer-disk:UpToDate
  san7 connection:Connecting
  xen1 role:Primary
    peer-disk:Diskless peer-client:yes

I added san5 in an attempt to get at least a proper second copy of the 
data before attempting to upgrade san6 which is still offline due to 
version mismatch.


output from journalctl -u linstor-controller, including the shutdown 
logs in case that provides some hints on where it was hung:


Jul 13 10:06:01 castle systemd[1]: Stopping linstor-controller.service - 
LINSTOR Controller Service...
Jul 13 10:06:01 castle Controller[30333]: 10:06:01.567 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Shutdown in progress
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: State 
'stop-sigterm' timed out. Killing.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30333 (java) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30345 (Main) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30346 (GC Thread#0) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30347 (G1 Main Marker) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30348 (G1 Conc#0) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30349 (G1 Refine#0) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30350 (G1 Service) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30351 (VM Thread) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30352 (Reference Handl) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30353 (Finalizer) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30354 (Signal Dispatch) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: linstor-controller.service: Killing 
process 30355 (Service Thread) with signal SIGKILL.
Jul 13 10:07:31 castle systemd[1]: 

How to recover linstor-controller node?

[Adam Goryachev]

Possibly a dumb question, but for a number of reasons, I have lost my 
linstor-controller node. I still have all other nodes up and running, 
but am missing any backup of the controller database. The main issue is 
that while the data is still available (both linstor-controller and 
backup systems) but both are offline for at least the next week.


So, is there any other method to "re-create" or otherwise setup a 
controller on another node?


I know, would be a good idea to setup some HA and/or some regular DB 
backup and replication to other nodes, but in the meantime, what are the 
options to get linstor-controller or at least linstor-satellites all up 
and running?

Re: [DRBD-user] Can't shrink backing block device offline

[Adam Goryachev]

On 15/7/21 20:40, Christophorus Reyhan wrote:

Hello,

I'm trying to shrink LVM backing block device offline because I 
use internal metadata, and I'm following this guide 
https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-shrinking-offline 
 
. But it failed at create-md step.


*Here's the log for first machine (called ol1):*
[root@ol1 drbd.d]# e2fsck -f /dev/drbd0
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/drbd0: 12/51200 files (0.0% non-contiguous), 109774/204732 blocks
[root@ol1 drbd.d]# drbdadm status
test role:Primary
  disk:UpToDate
  ol2 role:Secondary
    peer-disk:UpToDate

[root@ol1 drbd.d]# resize2fs /dev/drbd0 180M

So the filesystem inside your DRBD device is going to be 180M

resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/drbd0 to 184320 (1k) blocks.
The filesystem on /dev/drbd0 is now 184320 blocks long.

[root@ol1 drbd.d]# drbdadm down test
[root@ol1 drbd.d]# drbdadm dump-md test > /tmp/metadata
Found meta data is "unclean", please apply-al first
[root@ol1 drbd.d]# drbdadm apply-al test
[root@ol1 drbd.d]# drbdadm dump-md test > /tmp/metadata
[root@ol1 drbd.d]# lvreduce -L 180M /dev/mapper/vgdrbd-vol_drbd
  WARNING: Reducing active logical volume to 180.00 MiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce vgdrbd/vol_drbd? [y/n]: y
  Size of logical volume vgdrbd/vol_drbd changed from 200.00 MiB (50 
extents) to 180.00 MiB (45 extents).
Also, the size of the volume containing both your ext filesystem AND 
your DRBD data is going to be 180M

  Logical volume vgdrbd/vol_drbd successfully resized.
[root@ol1 drbd.d]# vi /tmp/metadata
[root@ol1 drbd.d]# blockdev --getsz /dev/mapper/vgdrbd-vol_drbd
368640
[root@ol1 drbd.d]# vi /tmp/metadata
[root@ol1 drbd.d]# drbdadm create-md test --max-peers=5
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/vgdrbd/vol_drbd at byte offset 188739584

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 188739584
al_offset 188706816
bm_offset 188674048

Found ext3 filesystem
      184320 kB data area apparently used
      184252 kB left usable by current configuration


As per the messages, there isn't enough space to preserve both your 
filesystem data AND add the DRBD data. Either resize your filesystem to 
179M, resize the LV to 180M, and then after everything is working again, 
you can resize the filesystem to take all available space, and it should 
increase to 184252kB or 179.9+ MB


I'm pretty sure the DRBD manual has some details on how to calculate 
exactly how much space you will need to leave available for the DRBD 
metadata, but I usually just do the above (make the filesystem smaller 
than my target, and then after DRBD has been resized, increase the FS to 
take up any extra space).


Hope that helps...

Regards,
Adam

___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] mounting drbd snapshots that contain vdo devices

[Adam Goryachev]

On 21/6/21 12:21, Eric Robinson wrote:


I have a filesystem directly on a drbd disk, and the drbd disk has an 
LVM volumes as a backing device. It looks like this…


lvm volume

    drbd disk

    ext4 filesystem

I can snapshot the backing device, mount it, and explore the filesystems.

However, I recently started experimenting with vdo, so now the stack 
looks like this….


lvm volume

    drbd disk

    vdo disk

ext4 filesystem

When I snapshot the backing device, I can no longer mount it because 
it complains that vdo is an unknown filesystem type. Any ideas how I 
can mount the vdo disk from the snapshot?


Makes sense. I guess you would need to refer to the VDO docs to find 
out what you need to do to make a snapshot of a VDO device appear as a 
proper/normal block device, and then mount the filesystem from that 
block device.


Very similar to if you used encryption between the DRBD disk and the 
filesystem, you would need to make the decrypted block device available, 
and then mount that


Regards,
Adam

___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] linstor failure

[Adam Goryachev]

Reposting the below as I guess early January wasn't the best time to get 
any responses. I'd really appreciate any assistance as I'd prefer to 
avoid rebuilding the VM from scratch (wasted hours, not lost data), but 
also I'd like to know how to resolve or avoid the issue in the future 
when I actually have real data being stored.


Thanks,
Adam


I have a small test setup with 2 x diskless linstor-satellite nodes, and 
4 x diskful linstor-satellite nodes, one of which is the linstor-controller.



The idea is that the diskless node is the compute node (xen, running the 
VM's whose data is on linstor resources).


I have 2 x test VM's, one which was (and still is) working OK (it's an 
older debian linux crossbowold), the other has failed (a Windows 10 VM 
jspiterivm1) while I was installing (attempting) the xen PV drivers (not 
sure if that is relevant or not). The other two resources are unused 
(ns2 and windows-wm).


I have a nothing relevant in the linstor error logs, but the linstor 
controller node has this in it's kern.log:


Dec 30 10:50:44 castle kernel: [4103630.414725] drbd windows-wm 
san6.mytest.com.au: sock was shut down by peer
Dec 30 10:50:44 castle kernel: [4103630.414752] drbd windows-wm 
san6.mytest.com.au: conn( Connected -> BrokenPipe ) peer( Secondary -> 
Unknown )
Dec 30 10:50:44 castle kernel: [4103630.414759] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: pdsk( UpToDate -> DUnknown ) repl( 
Established -> Off )
Dec 30 10:50:44 castle kernel: [4103630.414807] drbd windows-wm 
san6.mytest.com.au: ack_receiver terminated
Dec 30 10:50:44 castle kernel: [4103630.414810] drbd windows-wm 
san6.mytest.com.au: Terminating ack_recv thread
Dec 30 10:50:44 castle kernel: [4103630.445961] drbd windows-wm 
san6.mytest.com.au: Restarting sender thread
Dec 30 10:50:44 castle kernel: [4103630.479708] drbd windows-wm 
san6.mytest.com.au: Connection closed
Dec 30 10:50:44 castle kernel: [4103630.479739] drbd windows-wm 
san6.mytest.com.au: helper command: /sbin/drbdadm disconnected
Dec 30 10:50:44 castle kernel: [4103630.486479] drbd windows-wm 
san6.mytest.com.au: helper command: /sbin/drbdadm disconnected exit code 0
Dec 30 10:50:44 castle kernel: [4103630.486533] drbd windows-wm 
san6.mytest.com.au: conn( BrokenPipe -> Unconnected )
Dec 30 10:50:44 castle kernel: [4103630.486556] drbd windows-wm 
san6.mytest.com.au: Restarting receiver thread
Dec 30 10:50:44 castle kernel: [4103630.486566] drbd windows-wm 
san6.mytest.com.au: conn( Unconnected -> Connecting )
Dec 30 10:50:44 castle kernel: [4103631.006727] drbd windows-wm 
san6.mytest.com.au: Handshake to peer 2 successful: Agreed network 
protocol version 117
Dec 30 10:50:44 castle kernel: [4103631.006735] drbd windows-wm 
san6.mytest.com.au: Feature flags enabled on protocol level: 0xf TRIM 
THIN_RESYNC WRITE_SAME WRITE_ZEROES.
Dec 30 10:50:44 castle kernel: [4103631.006918] drbd windows-wm 
san6.mytest.com.au: Peer authenticated using 20 bytes HMAC
Dec 30 10:50:44 castle kernel: [4103631.006943] drbd windows-wm 
san6.mytest.com.au: Starting ack_recv thread (from drbd_r_windows- [1164])
Dec 30 10:50:44 castle kernel: [4103631.041925] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: drbd_sync_handshake:
Dec 30 10:50:44 castle kernel: [4103631.041932] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: self 
CC647323743B5AE0::: 
bits:0 flags:120
Dec 30 10:50:44 castle kernel: [4103631.041937] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: peer 
CC647323743B5AE0::: 
bits:0 flags:120
Dec 30 10:50:44 castle kernel: [4103631.041941] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: uuid_compare()=no-sync by rule 38
Dec 30 10:50:44 castle kernel: [4103631.229931] drbd windows-wm: 
Preparing cluster-wide state change 1880606796 (0->2 499/146)
Dec 30 10:50:44 castle kernel: [4103631.230424] drbd windows-wm: State 
change 1880606796: primary_nodes=0, weak_nodes=0
Dec 30 10:50:44 castle kernel: [4103631.230429] drbd windows-wm: 
Committing cluster-wide state change 1880606796 (0ms)
Dec 30 10:50:44 castle kernel: [4103631.230480] drbd windows-wm 
san6.mytest.com.au: conn( Connecting -> Connected ) peer( Unknown -> 
Secondary )
Dec 30 10:50:44 castle kernel: [4103631.230486] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: pdsk( DUnknown -> UpToDate ) repl( Off -> 
Established )
Dec 30 10:58:27 castle kernel: [4104093.577650] drbd jspiteriVM1 
xen1.mytest.com.au: peer( Primary -> Secondary )
Dec 30 10:58:27 castle kernel: [4104093.790062] drbd jspiteriVM1/0 
drbd1011: bitmap WRITE of 327 pages took 216 ms
Dec 30 10:58:39 castle kernel: [4104106.278699] drbd jspiteriVM1 
xen1.mytest.com.au: Preparing remote state change 490644362
Dec 30 10:58:39 castle kernel: [4104106.278984] drbd jspiteriVM1 
xen1.mytest.com.au: Committing remote state change 490644362 
(primary_nodes=10)
Dec 30 10:58:39 castle kernel: [4104106.278999] drbd jspiteriVM1 
xen1.mytest.com.au: peer( 

[DRBD-user] linstor failure

[Adam Goryachev]

I have a small test setup with 2 x diskless linstor-satellite nodes, and 
4 x diskful linstor-satellite nodes, one of which is the linstor-controller.


The idea is that the diskless node is the compute node (xen, running the 
VM's whose data is on linstor resources).


I have 2 x test VM's, one which was (and still is) working OK (it's an 
older debian linux crossbowold), the other has failed (a Windows 10 VM 
jspiterivm1) while I was installing (attempting) the xen PV drivers (not 
sure if that is relevant or not). The other two resources are unused 
(ns2 and windows-wm).


I have a nothing relevant in the linstor error logs, but the linstor 
controller node has this in it's kern.log:


Dec 30 10:50:44 castle kernel: [4103630.414725] drbd windows-wm 
san6.mytest.com.au: sock was shut down by peer
Dec 30 10:50:44 castle kernel: [4103630.414752] drbd windows-wm 
san6.mytest.com.au: conn( Connected -> BrokenPipe ) peer( Secondary -> 
Unknown )
Dec 30 10:50:44 castle kernel: [4103630.414759] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: pdsk( UpToDate -> DUnknown ) repl( 
Established -> Off )
Dec 30 10:50:44 castle kernel: [4103630.414807] drbd windows-wm 
san6.mytest.com.au: ack_receiver terminated
Dec 30 10:50:44 castle kernel: [4103630.414810] drbd windows-wm 
san6.mytest.com.au: Terminating ack_recv thread
Dec 30 10:50:44 castle kernel: [4103630.445961] drbd windows-wm 
san6.mytest.com.au: Restarting sender thread
Dec 30 10:50:44 castle kernel: [4103630.479708] drbd windows-wm 
san6.mytest.com.au: Connection closed
Dec 30 10:50:44 castle kernel: [4103630.479739] drbd windows-wm 
san6.mytest.com.au: helper command: /sbin/drbdadm disconnected
Dec 30 10:50:44 castle kernel: [4103630.486479] drbd windows-wm 
san6.mytest.com.au: helper command: /sbin/drbdadm disconnected exit code 0
Dec 30 10:50:44 castle kernel: [4103630.486533] drbd windows-wm 
san6.mytest.com.au: conn( BrokenPipe -> Unconnected )
Dec 30 10:50:44 castle kernel: [4103630.486556] drbd windows-wm 
san6.mytest.com.au: Restarting receiver thread
Dec 30 10:50:44 castle kernel: [4103630.486566] drbd windows-wm 
san6.mytest.com.au: conn( Unconnected -> Connecting )
Dec 30 10:50:44 castle kernel: [4103631.006727] drbd windows-wm 
san6.mytest.com.au: Handshake to peer 2 successful: Agreed network 
protocol version 117
Dec 30 10:50:44 castle kernel: [4103631.006735] drbd windows-wm 
san6.mytest.com.au: Feature flags enabled on protocol level: 0xf TRIM 
THIN_RESYNC WRITE_SAME WRITE_ZEROES.
Dec 30 10:50:44 castle kernel: [4103631.006918] drbd windows-wm 
san6.mytest.com.au: Peer authenticated using 20 bytes HMAC
Dec 30 10:50:44 castle kernel: [4103631.006943] drbd windows-wm 
san6.mytest.com.au: Starting ack_recv thread (from drbd_r_windows- [1164])
Dec 30 10:50:44 castle kernel: [4103631.041925] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: drbd_sync_handshake:
Dec 30 10:50:44 castle kernel: [4103631.041932] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: self 
CC647323743B5AE0::: 
bits:0 flags:120
Dec 30 10:50:44 castle kernel: [4103631.041937] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: peer 
CC647323743B5AE0::: 
bits:0 flags:120
Dec 30 10:50:44 castle kernel: [4103631.041941] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: uuid_compare()=no-sync by rule 38
Dec 30 10:50:44 castle kernel: [4103631.229931] drbd windows-wm: 
Preparing cluster-wide state change 1880606796 (0->2 499/146)
Dec 30 10:50:44 castle kernel: [4103631.230424] drbd windows-wm: State 
change 1880606796: primary_nodes=0, weak_nodes=0
Dec 30 10:50:44 castle kernel: [4103631.230429] drbd windows-wm: 
Committing cluster-wide state change 1880606796 (0ms)
Dec 30 10:50:44 castle kernel: [4103631.230480] drbd windows-wm 
san6.mytest.com.au: conn( Connecting -> Connected ) peer( Unknown -> 
Secondary )
Dec 30 10:50:44 castle kernel: [4103631.230486] drbd windows-wm/0 
drbd1001 san6.mytest.com.au: pdsk( DUnknown -> UpToDate ) repl( Off -> 
Established )
Dec 30 10:58:27 castle kernel: [4104093.577650] drbd jspiteriVM1 
xen1.mytest.com.au: peer( Primary -> Secondary )
Dec 30 10:58:27 castle kernel: [4104093.790062] drbd jspiteriVM1/0 
drbd1011: bitmap WRITE of 327 pages took 216 ms
Dec 30 10:58:39 castle kernel: [4104106.278699] drbd jspiteriVM1 
xen1.mytest.com.au: Preparing remote state change 490644362
Dec 30 10:58:39 castle kernel: [4104106.278984] drbd jspiteriVM1 
xen1.mytest.com.au: Committing remote state change 490644362 
(primary_nodes=10)
Dec 30 10:58:39 castle kernel: [4104106.278999] drbd jspiteriVM1 
xen1.mytest.com.au: peer( Secondary -> Primary )
Dec 30 10:58:40 castle kernel: [4104106.547178] drbd jspiteriVM1/0 
drbd1011 xen1.mytest.com.au: resync-susp( no -> connection dependency )
Dec 30 10:58:40 castle kernel: [4104106.547191] drbd jspiteriVM1/0 
drbd1011 san5.mytest.com.au: repl( PausedSyncT -> SyncTarget ) 
resync-susp( peer -> no )
Dec 30 10:58:40 castle kernel: 

Re: [DRBD-user] drbd does not find metadata any more

[Adam Goryachev]

On 26/11/20 23:08, Michael Schwartzkopff wrote:

Hi,


I tried to upgrade from CentOS 8.0 to 8.1. My installation has a
self-compiled DRBD-9.

After the upgrade my drbd does not start any more. the Status says:

# drbdadm status
r0 role:Secondary
   disk:Diskless
   dev02 connection:StandAlone


Trying to attach to the disk tells me:

# drbdadm attach r0
1: Failure: (119) No valid meta-data signature found.

     ==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command 'drbdsetup attach 1 /dev/vgDev01/drbd /dev/vgDev01/drbd
internal' terminated with exit code 10


The journal says:

Nov 26 13:05:09 dev01.sys4.de kernel: drbd r0/0 drbd1: meta-data IO
uses: blk-bio
Nov 26 13:05:09 dev01.sys4.de kernel: drbd r0/0 drbd1: ambiguous node
id: meta-data: 1, config: 0


As far as I can see I have meta-data on my disk:

# drbdadm dump-md r0 | head -30
# DRBD meta data dump
# 2020-11-26 13:06:58 +0100 [1606392418]
# xxx.sys4.de> drbdmeta 1 v09 /dev/vgDev01/drbd internal dump-md
#

version "v09";

max-peers 1;
# md_size_sect 196680
# md_offset 3298534879232
# al_offset 3298534846464
# bm_offset 3298434183168

node-id 1;
current-uuid 0x45451CABFEB9894F;
flags 0x0093;
peer[0] {
     bitmap-index 0;
     bitmap-uuid 0xF3394DA3B6073B46;
     bitmap-dagtag 0x;
     flags 0x;
}

(...)


Any idea how to bring my DRBD to life again?

Any help appreciated.


Mit freundlichen Grüßen,

Just a thought, did you check if your drbd 9 kernel module is loaded, or 
is the old drbd 8 kernel module loaded? Check /proc/drbd to see this. 
You might need to re-compile the DRBD9 for the new kernel that was 
installed with the upgrade.


Hope that helps...

Regards,
Adam

___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] drbd-9.0.26-rc1

[Adam Goryachev]

Here is my failed compile attempt on Raspberry Pi 4 with Ubuntu 20.10:

root@pi-03:/usr/src/drbd_manual/drbd-9.0.26-0rc1# make
Need a git checkout to regenerate drbd/.drbd_git_revision
make[1]: Entering directory '/usr/src/drbd_manual/drbd-9.0.26-0rc1/drbd'

    Calling toplevel makefile of kernel source tree, which I believe is in
    KDIR=/lib/modules/5.8.0-1006-raspi/build

make -C /lib/modules/5.8.0-1006-raspi/build 
M=/usr/src/drbd_manual/drbd-9.0.26-0rc1/drbd  modules

  COMPAT  __vmalloc_has_2_params
  COMPAT  alloc_workqueue_takes_fmt
  COMPAT  before_4_13_kernel_read
  COMPAT  blkdev_issue_zeroout_discard
  COMPAT  drbd_release_returns_void
  COMPAT  genl_policy_in_ops
  COMPAT  have_SHASH_DESC_ON_STACK
  COMPAT  have_WB_congested_enum
  COMPAT  have_allow_kernel_signal
  COMPAT  have_atomic_dec_if_positive_linux
  COMPAT  have_atomic_in_flight
  COMPAT  have_bd_claim_by_disk
  COMPAT  have_bd_unlink_disk_holder
  COMPAT  have_bdi_congested_fn
  COMPAT  have_bio_bi_bdev
  COMPAT  have_bio_bi_disk
  COMPAT  have_bio_bi_error
  COMPAT  have_bio_bi_opf
  COMPAT  have_bio_bi_status
  COMPAT  have_bio_clone_fast
  COMPAT  have_bio_flush
  COMPAT  have_bio_free
  COMPAT  have_bio_op_shift
  COMPAT  have_bio_rw
  COMPAT  have_bio_set_op_attrs
  COMPAT  have_bio_start_io_acct
  COMPAT  have_bioset_create_front_pad
  COMPAT  have_bioset_init
  COMPAT  have_bioset_need_bvecs
  COMPAT  have_blk_check_plugged
  COMPAT  have_blk_qc_t_make_request
  COMPAT  have_blk_queue_flag_set
  COMPAT  have_blk_queue_make_request
  COMPAT  have_blk_queue_merge_bvec
  COMPAT  have_blk_queue_plugged
  COMPAT  have_blk_queue_split_bio
  COMPAT  have_blk_queue_split_q_bio
  COMPAT  have_blk_queue_split_q_bio_bioset
  COMPAT  have_blk_queue_update_readahead
  COMPAT  have_blk_queue_write_cache
  COMPAT  have_blkdev_get_by_path
  COMPAT  have_d_inode
  COMPAT  have_fallthrough
  COMPAT  have_file_inode
  COMPAT  have_generic_start_io_acct_q_rw_sect_part
  COMPAT  have_generic_start_io_acct_rw_sect_part
  COMPAT  have_genl_family_parallel_ops
  COMPAT  have_ib_cq_init_attr
  COMPAT  have_ib_get_dma_mr
  COMPAT  have_idr_alloc
  COMPAT  have_idr_is_empty
  COMPAT  have_inode_lock
  COMPAT  have_ktime_to_timespec64
  COMPAT  have_kvfree
  COMPAT  have_max_send_recv_sge
  COMPAT  have_netlink_cb_portid
  COMPAT  have_nla_nest_start_noflag
  COMPAT  have_nla_parse_deprecated
  COMPAT  have_nla_put_64bit
  COMPAT  have_part_stat_h
  COMPAT  have_pointer_backing_dev_info
  COMPAT  have_prandom_u32
  COMPAT  have_proc_create_single
  COMPAT  have_ratelimit_state_init
  COMPAT  have_rb_augment_functions
  COMPAT  have_refcount_inc
  COMPAT  have_req_hardbarrier
  COMPAT  have_req_noidle
  COMPAT  have_req_nounmap
  COMPAT  have_req_op_write
  COMPAT  have_req_op_write_same
  COMPAT  have_req_op_write_zeroes
  COMPAT  have_req_prio
  COMPAT  have_req_write
  COMPAT  have_req_write_same
  COMPAT  have_revalidate_disk_size
  COMPAT  have_security_netlink_recv
  COMPAT  have_sendpage_ok
  COMPAT  have_shash_desc_zero
  COMPAT  have_signed_nla_put
  COMPAT  have_simple_positive
  COMPAT  have_struct_bvec_iter
  COMPAT  have_struct_kernel_param_ops
  COMPAT  have_struct_size
  COMPAT  have_submit_bio
  COMPAT  have_submit_bio_noacct
  COMPAT  have_tcp_sock_set_cork
  COMPAT  have_tcp_sock_set_nodelay
  COMPAT  have_tcp_sock_set_quickack
  COMPAT  have_time64_to_tm
  COMPAT  have_timer_setup
  COMPAT  have_void_make_request
  COMPAT  hlist_for_each_entry_has_three_parameters
  COMPAT  ib_alloc_pd_has_2_params
  COMPAT  ib_device_has_ops
  COMPAT  ib_post_send_const_params
  COMPAT  ib_query_device_has_3_params
  COMPAT  kmap_atomic_page_only
  COMPAT  need_make_request_recursion
  COMPAT  queue_limits_has_discard_zeroes_data
  COMPAT  rdma_create_id_has_net_ns
  COMPAT  sock_create_kern_has_five_parameters
  COMPAT  sock_ops_returns_addr_len
  UPD /usr/src/drbd_manual/drbd-9.0.26-0rc1/drbd/compat.5.8.14.h
  UPD /usr/src/drbd_manual/drbd-9.0.26-0rc1/drbd/compat.h
./drbd-kernel-compat/gen_compat_patch.sh: line 12: spatch: command not found
./drbd-kernel-compat/gen_compat_patch.sh: line 45: hash: spatch: not found
  INFO: no suitable spatch found; trying spatch-as-a-service;
  be patient, may take up to 10 minutes
  if it is in the server side cache it might only take a second
  SPAAS    e770d31eb644591d9d0920dfda4bff32
Successfully connected to SPAAS ('d35a4b17210dab1336de2725b997f300e9acd297')
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
 Dload  Upload   Total   Spent Left  Speed
100 11287    0  5767    0  5520    406    388 --:--:--  0:00:14 
--:--:--  1368

  You can create a new .tgz including this pre-computed compat patch
  by calling "make unpatch ; echo 
drbd-9.0.26-0rc1/drbd/drbd-kernel-compat/cocci_cache/e770d31eb644591d9d0920dfda4bff32/compat.patch 
>>.filelist ; make tgz"

  PATCH
patching file ./drbd_int.h
patching file drbd_main.c
patching file drbd_debugfs.c

Re: [DRBD-user] drbd-9.0.25-1 doesn't build against linux 5.8

[Adam Goryachev]

I didn't see any response to this on the archives, and am having the 
same problem. I'm trying to install onto a raspberry pi 4 using ubuntu 
20.10.


The last part of my compile log:

  CC [M] 
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_sender.o
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_bitmap.c: In 
function ‘bm_realloc_pages’:
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_bitmap.c:368:15: 
error: too many arguments to function ‘__vmalloc’

  368 |   new_pages = __vmalloc(bytes,
  |   ^
In file included from 
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_bitmap.c:16:

./include/linux/vmalloc.h:111:14: note: declared here
  111 | extern void *__vmalloc(unsigned long size, gfp_t gfp_mask);
  |  ^
  CC [M] 
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_receiver.o
make[2]: *** [scripts/Makefile.build:290: 
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd/drbd_bitmap.o] Error 1

make[2]: *** Waiting for unfinished jobs
make[1]: *** [Makefile:1783: 
/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd] Error 2

make: *** [Makefile:132: kbuild] Error 2
make: Leaving directory 
'/var/lib/dkms/drbd/9.0.25-1ppa1~focal1/build/src/drbd'


Is there a simple solution to this?

Regards,
Adam

On 12/10/20 00:54, Elimar Riesebieter wrote:

Hi all,

building 9.0.25-1 dkms modul against linux 5.8 gives:

/var/lib/dkms/drbd/9.0.25-1hsh1/build/make.log:

/var/lib/dkms/drbd/9.0.25-1hsh1/build/src/drbd/drbd_bitmap.c: In function 
‘bm_realloc_pages’:
/var/lib/dkms/drbd/9.0.25-1hsh1/build/src/drbd/drbd_bitmap.c:368:15: error:
too many arguments to function ‘__vmalloc’
new_pages = __vmalloc(bytes,
^
In file included from 
/var/lib/dkms/drbd/9.0.25-1hsh1/build/src/drbd/drbd_bitmap.c:16:
/usr/src/linux-headers-5.8.0-0.bpo.2-common/include/linux/vmalloc.h:111:14: 
note: declared here
  extern void *__vmalloc(unsigned long size, gfp_t gfp_mask);

Indeed signatures have been changed:

5.7:
extern void *__vmalloc(unsigned long size, gfp_t gfp_mask, pgprot_t prot);

5.8:
extern void *__vmalloc(unsigned long size, gfp_t gfp_mask);

will there be a short-term update?

Thanks in advance

Elimar


--
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au

___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] linstor Error after adding resource to node

[Adam Goryachev]

On 30/9/20 16:49, Gábor Hernádi wrote:

Hi,

I tried to recreate this issue, but without success.

4 Node setup, all LVM
First create a resource with --auto-place 3,
Create 9 other resources with --auto-place 4
Create the first resource on the 4th (missing) node
Check "linstor volume list"

That means, there has to be something else in your setup.
What else did you do? I see that your "first" resource "windows-wm" 
was more like the second resource, as it got the minor-number 1001, 
instead of 1000. That minor-number 1000 was later reused by "testvm1". 
However, was something broken with the "original" resource using 
minor-number 1000?


Unfortunately, yes, a whole bunch of things have been done on the first 
three nodes. I've been slowly messing around to try and get everything 
working over the last few months. There was another "testvm3" created 
before, which I deleted, and then started again with the further testing



Error report 5F733CD9-0-04 is a NullPointerException, but this 
is most likely just a side-effect of the original issue.


> Since it looks relevant, error reports 1, 2 and 3 are all similar 
for nodes castle, san5 and san6


What about error report 0? Not relevant for this issue?

Ooops, I just didn't think that there was a 0 report... I've included it 
here now:


ERROR REPORT 5F733CD9-0-00



Application:    LINBIT® LINSTOR
Module: Controller
Version:    1.9.0
Build ID: 678acd24a8b9b73a735407cd79ca33a5e95eb2e2
Build time: 2020-09-23T10:27:49+00:00
Error time: 2020-09-30 00:12:19
Node:   castle.websitemanagers.com.au
Peer:   RestClient(192.168.5.207; 
'PythonLinstor/1.4.0 (API1.0.4)')




Reported error:
===

Description:
    Dependency not found

Category:   LinStorException
Class name: LinStorException
Class canonical name: com.linbit.linstor.LinStorException
Generated at:   Method 'checkStorPoolLoaded', Source 
file 'CtrlStorPoolResolveHelper.java', Line #225


Error message:  Dependency not found

Error context:
    The storage pool 'DfltStorPool' for resource 'windows-wm' for 
volume number '0' is not deployed on node 'san7'.


Call backtrace:

    Method   Native Class:Line number
    checkStorPoolLoaded  N 
com.linbit.linstor.CtrlStorPoolResolveHelper:225
    resolveStorPool  N 
com.linbit.linstor.CtrlStorPoolResolveHelper:149
    resolveStorPool  N 
com.linbit.linstor.CtrlStorPoolResolveHelper:65
    createVolumeResolvingStorPool    N 
com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:72
    createResourceDb N 
com.linbit.linstor.core.apicallhandler.controller.CtrlRscCrtApiHelper:396
    createResourceInTransaction  N 
com.linbit.linstor.core.apicallhandler.controller.CtrlRscCrtApiCallHandler:171
    lambda$createResource$2  N 
com.linbit.linstor.core.apicallhandler.controller.CtrlRscCrtApiCallHandler:143
    doInScope    N 
com.linbit.linstor.core.apicallhandler.ScopeRunner:147
    lambda$fluxInScope$0 N 
com.linbit.linstor.core.apicallhandler.ScopeRunner:75
    call N 
reactor.core.publisher.MonoCallable:91
    trySubscribeScalarMap    N 
reactor.core.publisher.FluxFlatMap:126
    subscribeOrReturn    N 
reactor.core.publisher.MonoFlatMapMany:49
    subscribe    N 
reactor.core.publisher.Flux:8311
    onNext   N 
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request  N 
reactor.core.publisher.Operators$ScalarSubscription:2317
    onSubscribe  N 
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe    N 
reactor.core.publisher.MonoCurrentContext:35
    subscribe    N 
reactor.core.publisher.Flux:8325
    onNext   N 
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request  N 
reactor.core.publisher.Operators$ScalarSubscription:2317
    onSubscribe  N 
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe    N 
reactor.core.publisher.MonoCurrentContext:35
    subscribe    N 
reactor.core.publisher.Flux:8325
    onNext   N 

[DRBD-user] linstor Error after adding resource to node

[Adam Goryachev]

I'm still progressively testing linstor on a group of test servers, and 
now came across a new problem.


I had everything working nicely with 4 x nodes, linstor n l showed all 4 
working, machines would automatically come back after a reboot, and it 
all looked good. I deleted all the storage-pools and re-created them 
(same name across all servers), and then followed the docs to create my 
first resource-definition, volume-definition and then used 
auto-placement on 3 nodes.


I then decided to get clever, and started creating another 9 tests with 
auto-placement on 4 nodes. All of this worked well, and I got a status 
like this:


╭──╮
┊ Node   ┊ Resource   ┊ StoragePool ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊  
Allocated ┊ InUse  ┊    State ┊

╞══╡
┊ castle ┊ testvm1    ┊ pool_hdd    ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm1    ┊ pool_hdd    ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm1    ┊ pool_hdd    ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm1    ┊ pool_hdd    ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm2    ┊ pool_hdd    ┊ 0 ┊    1002 ┊ /dev/drbd1002 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm2    ┊ pool_hdd    ┊ 0 ┊    1002 ┊ /dev/drbd1002 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm2    ┊ pool_hdd    ┊ 0 ┊    1002 ┊ /dev/drbd1002 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm2    ┊ pool_hdd    ┊ 0 ┊    1002 ┊ /dev/drbd1002 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm3    ┊ pool_hdd    ┊ 0 ┊    1003 ┊ /dev/drbd1003 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm3    ┊ pool_hdd    ┊ 0 ┊    1003 ┊ /dev/drbd1003 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm3    ┊ pool_hdd    ┊ 0 ┊    1003 ┊ /dev/drbd1003 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm3    ┊ pool_hdd    ┊ 0 ┊    1003 ┊ /dev/drbd1003 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm4    ┊ pool_hdd    ┊ 0 ┊    1004 ┊ /dev/drbd1004 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm4    ┊ pool_hdd    ┊ 0 ┊    1004 ┊ /dev/drbd1004 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm4    ┊ pool_hdd    ┊ 0 ┊    1004 ┊ /dev/drbd1004 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm4    ┊ pool_hdd    ┊ 0 ┊    1004 ┊ /dev/drbd1004 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm5    ┊ pool_hdd    ┊ 0 ┊    1005 ┊ /dev/drbd1005 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm5    ┊ pool_hdd    ┊ 0 ┊    1005 ┊ /dev/drbd1005 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm5    ┊ pool_hdd    ┊ 0 ┊    1005 ┊ /dev/drbd1005 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm5    ┊ pool_hdd    ┊ 0 ┊    1005 ┊ /dev/drbd1005 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm6    ┊ pool_hdd    ┊ 0 ┊    1006 ┊ /dev/drbd1006 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm6    ┊ pool_hdd    ┊ 0 ┊    1006 ┊ /dev/drbd1006 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm6    ┊ pool_hdd    ┊ 0 ┊    1006 ┊ /dev/drbd1006 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm6    ┊ pool_hdd    ┊ 0 ┊    1006 ┊ /dev/drbd1006 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm7    ┊ pool_hdd    ┊ 0 ┊    1007 ┊ /dev/drbd1007 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm7    ┊ pool_hdd    ┊ 0 ┊    1007 ┊ /dev/drbd1007 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm7    ┊ pool_hdd    ┊ 0 ┊    1007 ┊ /dev/drbd1007 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm7    ┊ pool_hdd    ┊ 0 ┊    1007 ┊ /dev/drbd1007 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm8    ┊ pool_hdd    ┊ 0 ┊    1008 ┊ /dev/drbd1008 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm8    ┊ pool_hdd    ┊ 0 ┊    1008 ┊ /dev/drbd1008 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm8    ┊ pool_hdd    ┊ 0 ┊    1008 ┊ /dev/drbd1008 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm8    ┊ pool_hdd    ┊ 0 ┊    1008 ┊ /dev/drbd1008 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ testvm9    ┊ pool_hdd    ┊ 0 ┊    1009 ┊ /dev/drbd1009 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ testvm9    ┊ pool_hdd    ┊ 0 ┊    1009 ┊ /dev/drbd1009 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ testvm9    ┊ pool_hdd    ┊ 0 ┊    1009 ┊ /dev/drbd1009 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ san7   ┊ testvm9    ┊ pool_hdd    ┊ 0 ┊    1009 ┊ /dev/drbd1009 ┊ 
102.42 MiB ┊ Unused ┊ UpToDate ┊
┊ castle ┊ windows-wm ┊ pool_hdd    ┊ 0 ┊    1001 ┊ /dev/drbd1001 ┊  
49.16 MiB ┊ Unused ┊ UpToDate ┊
┊ san5   ┊ windows-wm ┊ pool_hdd    ┊ 0 ┊    1001 ┊ /dev/drbd1001 ┊  
49.16 MiB ┊ Unused ┊ UpToDate ┊
┊ san6   ┊ 

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

On 25/9/20 16:12, Rene Peinthor wrote:
My last idea is that you didn't run with a linstor.toml file until now 
and so user/password would have been empty.
You can comment out user/password in the linstor.toml and restart the 
controller, if this doesn't work it is probably simpler to start fresh.


Interesting, you are correct, it works if I comment out the 
username/password.


So, in order to try and get a more standard installation, I tried 
connecting to the DB and running these commands:


SET PASSWORD='linstor';
ALTER USER "" rename to "linstor";

They seemed to work, but after resetting the linstor.toml file with the 
user/pass, it wouldn't work, and I couldn't connect to the DB  with the 
linstor/linstor nor with the details as blanks.


So, for now, I've restored the DB, and commented out the user/pass from 
the linstor.toml file.


Could you offer any advice on how to fix things so that it will use the 
linstor/linstor credentials normally?


linstor n l
╭──╮
┊ Node   ┊ NodeType  ┊ Addresses  ┊ State  ┊
╞══╡
┊ castle ┊ SATELLITE ┊ 192.168.5.204:3366 (PLAIN) ┊ Online ┊
┊ san5   ┊ SATELLITE ┊ 192.168.5.205:3366 (PLAIN) ┊ Online ┊
┊ san6   ┊ SATELLITE ┊ 192.168.5.206:3366 (PLAIN) ┊ Online ┊
╰──╯

Regards,
Adam

You can delete the DB file and start over again, but your LV/ZFS 
volumes will still be there.
Your data will still be replicated, but the Satellite will delete the 
resource files once restarted.


Cheers,
Rene

On Thu, Sep 24, 2020 at 4:55 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:



On 24/9/20 23:48, Rene Peinthor wrote:

I can only recreate that error, if I actively change the password
in the linstor.toml to something incorrect.
I don't know how this would happen in normal operation, are you
sure you never had another password or changed the password?

if you can connect to the h2 database with the commandline tool,
you might be able to set a new password.


I tried this method:

castle:/usr/share/linstor-server/lib#  java -cp h2*.jar
org.h2.tools.Shell

Welcome to H2 Shell 1.4.197 (2018-03-18)
Exit with Ctrl+C
[Enter]   jdbc:h2:~/test
URL   jdbc:h2:/var/lib/linstor/linstordb
[Enter]   org.h2.Driver
Driver
[Enter]
User  linstor
[Enter]   Hide
Password  linstor
SQL Exception: Wrong user name or password [28000-197]

However, it requires user/pass, which clearly is wrong

Is there a method to confirm that the file is "valid" ? I seem to
have three versions of this file all identical content:

-rw-r--r-- 1 root root 376832 Sep 18 16:45 linstordb.mv.db
-rw-r--r-- 1 root root 376832 Sep 24 02:25
linstordb.mv.db-2020-09-24T02:25+10:00.bak
-rw-r--r-- 1 root root 376832 Sep 24 02:25 linstordb.mv.db.bak

Should I just delete the DB file and start again? Is this data
replicated to other nodes automatically?

Thanks,
Adam



Cheers,
Rene

    On Thu, Sep 24, 2020 at 2:48 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:


On 24/9/20 22:42, Rene Peinthor wrote:

They are.
And how do you start the controller?


It should start automatically at boot from systemd, but I can
start it manually with:

systemctl restart linstor-controller

In case it is your next question, here is the content of
/etc/systemd/system/multi-user.target.wants/linstor-controller.service

[Unit]
Description=LINSTOR Controller Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/share/linstor-server/bin/Controller
--logs=/var/log/linstor-controller
--config-directory=/etc/linstor
# if killed by signal 143 -> SIGTERM, 129 -> SIGHUP
SuccessExitStatus=0 143 129
User=root
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Regards,
Adam


    On Thu, Sep 24, 2020 at 2:19 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:


On 24/9/20 15:30, Rene Peinthor wrote:

Sounds weird, I don't think we changed anything with
the database initialization.

Can you post your /etc/linstor/linstor.toml?

Cheers,
Rene


Hi, please see the contents below (I assume these are
default settings):

# Basic linstor configuration toml file
# For more options check documentation

[db]
  user = "linstor"
  password = "linstor"
  connection_url

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

On 24/9/20 23:48, Rene Peinthor wrote:
I can only recreate that error, if I actively change the password in 
the linstor.toml to something incorrect.
I don't know how this would happen in normal operation, are you sure 
you never had another password or changed the password?


if you can connect to the h2 database with the commandline tool, you 
might be able to set a new password.



I tried this method:

castle:/usr/share/linstor-server/lib#  java -cp h2*.jar org.h2.tools.Shell

Welcome to H2 Shell 1.4.197 (2018-03-18)
Exit with Ctrl+C
[Enter]   jdbc:h2:~/test
URL   jdbc:h2:/var/lib/linstor/linstordb
[Enter]   org.h2.Driver
Driver
[Enter]
User  linstor
[Enter]   Hide
Password  linstor
SQL Exception: Wrong user name or password [28000-197]

However, it requires user/pass, which clearly is wrong

Is there a method to confirm that the file is "valid" ? I seem to have 
three versions of this file all identical content:


-rw-r--r-- 1 root root 376832 Sep 18 16:45 linstordb.mv.db
-rw-r--r-- 1 root root 376832 Sep 24 02:25 
linstordb.mv.db-2020-09-24T02:25+10:00.bak

-rw-r--r-- 1 root root 376832 Sep 24 02:25 linstordb.mv.db.bak

Should I just delete the DB file and start again? Is this data 
replicated to other nodes automatically?


Thanks,
Adam



Cheers,
Rene

On Thu, Sep 24, 2020 at 2:48 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:



On 24/9/20 22:42, Rene Peinthor wrote:

They are.
And how do you start the controller?


It should start automatically at boot from systemd, but I can
start it manually with:

systemctl restart linstor-controller

In case it is your next question, here is the content of
/etc/systemd/system/multi-user.target.wants/linstor-controller.service

[Unit]
Description=LINSTOR Controller Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/share/linstor-server/bin/Controller
--logs=/var/log/linstor-controller --config-directory=/etc/linstor
# if killed by signal 143 -> SIGTERM, 129 -> SIGHUP
SuccessExitStatus=0 143 129
User=root
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Regards,
Adam


On Thu, Sep 24, 2020 at 2:19 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:


On 24/9/20 15:30, Rene Peinthor wrote:

Sounds weird, I don't think we changed anything with the
database initialization.

Can you post your /etc/linstor/linstor.toml?

Cheers,
Rene


Hi, please see the contents below (I assume these are default
settings):

# Basic linstor configuration toml file
# For more options check documentation

[db]
  user = "linstor"
  password = "linstor"
  connection_url = "jdbc:h2:/var/lib/linstor/linstordb"

Thanks,
Adam



On Wed, Sep 23, 2020 at 6:54 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:

Hi,

After upgrading on one of my test nodes I got some
errors, so I thought to upgrade the controller, and now
I am unable to start the controller.

Here is what I get in the system log:

Sep 24 02:33:40 castle systemd[1]: Started LINSTOR
Controller Service.
Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module
Controller
Sep 24 02:33:41 castle Controller[1093]:
Version:    1.9.0
(678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
Sep 24 02:33:41 castle Controller[1093]: Build time:
2020-09-23T10:27:49+00:00
Sep 24 02:33:41 castle Controller[1093]: Java
Version:   11
Sep 24 02:33:41 castle Controller[1093]: Java
VM:    Debian, Version
11.0.8+10-post-Debian-1deb10u1
Sep 24 02:33:41 castle Controller[1093]: Operating
system:   Linux, Version 4.19.0-10-amd64
Sep 24 02:33:41 castle Controller[1093]:
Environment:    amd64, 2 processors, 1988 MiB memory
reserved for allocations
Sep 24 02:33:41 castle Controller[1093]: System
components initialization in progress
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355
[main] INFO LINSTOR/Controller - SYSTEM - ErrorReporter
DB version 1 found.
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359
[main] INFO LINSTOR/Controller - SYSTEM - Log directory
set to: '/var/log/linstor-controller'
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410
[main] WARN io.sentry.dsn.Dsn - *** Couldn't find a
suitable DSN, Sentry operations will do nothing! See
documentation: https

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

On 24/9/20 22:42, Rene Peinthor wrote:

They are.
And how do you start the controller?

It should start automatically at boot from systemd, but I can start it 
manually with:


systemctl restart linstor-controller

In case it is your next question, here is the content of 
/etc/systemd/system/multi-user.target.wants/linstor-controller.service


[Unit]
Description=LINSTOR Controller Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/share/linstor-server/bin/Controller 
--logs=/var/log/linstor-controller --config-directory=/etc/linstor

# if killed by signal 143 -> SIGTERM, 129 -> SIGHUP
SuccessExitStatus=0 143 129
User=root
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Regards,
Adam

On Thu, Sep 24, 2020 at 2:19 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:



On 24/9/20 15:30, Rene Peinthor wrote:

Sounds weird, I don't think we changed anything with the database
initialization.

Can you post your /etc/linstor/linstor.toml?

Cheers,
Rene


Hi, please see the contents below (I assume these are default
settings):

# Basic linstor configuration toml file
# For more options check documentation

[db]
  user = "linstor"
  password = "linstor"
  connection_url = "jdbc:h2:/var/lib/linstor/linstordb"

Thanks,
    Adam



On Wed, Sep 23, 2020 at 6:54 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:

Hi,

After upgrading on one of my test nodes I got some errors, so
I thought to upgrade the controller, and now I am unable to
start the controller.

Here is what I get in the system log:

Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller
Service.
Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module
Controller
Sep 24 02:33:41 castle Controller[1093]: Version:   
1.9.0 (678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
Sep 24 02:33:41 castle Controller[1093]: Build time:
2020-09-23T10:27:49+00:00
Sep 24 02:33:41 castle Controller[1093]: Java Version:   11
Sep 24 02:33:41 castle Controller[1093]: Java VM:   
Debian, Version 11.0.8+10-post-Debian-1deb10u1
Sep 24 02:33:41 castle Controller[1093]: Operating system:  
Linux, Version 4.19.0-10-amd64
Sep 24 02:33:41 castle Controller[1093]: Environment:   
amd64, 2 processors, 1988 MiB memory reserved for allocations
Sep 24 02:33:41 castle Controller[1093]: System components
initialization in progress
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main]
INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB version
1 found.
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main]
INFO  LINSTOR/Controller - SYSTEM - Log directory set to:
'/var/log/linstor-controller'
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main]
WARN  io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN,
Sentry operations will do nothing! See documentation:
https://docs.sentry.io/clients/java/
<https://docs.sentry.io/clients/java/> ***
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main]
INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main]
INFO  LINSTOR/Controller - SYSTEM - API classes loading
finished: 611ms
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main]
INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main]
INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load
of extension module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main]
INFO  LINSTOR/Controller - SYSTEM - Extension module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
is not installed
Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal
reflective access operation has occurred
Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal
reflective access by
com.google.inject.internal.cglib.core.$ReflectUtils$1
(file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to
method

java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Sep 24 02:33:43 castle Controller[1093]: WARNING: Please
consider reporting this to the maintainers of
com.google.inject.internal.cglib.core.$ReflectUtils$1
Sep 24 02:33:43 castle Controller[1093]: WARNING: Use
--illega

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

On 24/9/20 15:30, Rene Peinthor wrote:
Sounds weird, I don't think we changed anything with the database 
initialization.


Can you post your /etc/linstor/linstor.toml?

Cheers,
Rene


Hi, please see the contents below (I assume these are default settings):

# Basic linstor configuration toml file
# For more options check documentation

[db]
  user = "linstor"
  password = "linstor"
  connection_url = "jdbc:h2:/var/lib/linstor/linstordb"

Thanks,
Adam


On Wed, Sep 23, 2020 at 6:54 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:


Hi,

After upgrading on one of my test nodes I got some errors, so I
thought to upgrade the controller, and now I am unable to start
the controller.

Here is what I get in the system log:

Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller Service.
Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module Controller
Sep 24 02:33:41 castle Controller[1093]: Version:    1.9.0
(678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
Sep 24 02:33:41 castle Controller[1093]: Build time:
2020-09-23T10:27:49+00:00
Sep 24 02:33:41 castle Controller[1093]: Java Version:   11
Sep 24 02:33:41 castle Controller[1093]: Java VM:   
Debian, Version 11.0.8+10-post-Debian-1deb10u1
Sep 24 02:33:41 castle Controller[1093]: Operating system:  
Linux, Version 4.19.0-10-amd64
Sep 24 02:33:41 castle Controller[1093]: Environment:   
amd64, 2 processors, 1988 MiB memory reserved for allocations
Sep 24 02:33:41 castle Controller[1093]: System components
initialization in progress
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main] INFO 
LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main] INFO 
LINSTOR/Controller - SYSTEM - Log directory set to:
'/var/log/linstor-controller'
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main] WARN 
io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN, Sentry
operations will do nothing! See documentation:
https://docs.sentry.io/clients/java/
<https://docs.sentry.io/clients/java/> ***
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main] INFO 
LINSTOR/Controller - SYSTEM - Loading API classes started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main] INFO 
LINSTOR/Controller - SYSTEM - API classes loading finished: 611ms
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main] INFO 
LINSTOR/Controller - SYSTEM - Dependency injection started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO 
LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension
module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO 
LINSTOR/Controller - SYSTEM - Extension module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
is not installed
Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal
reflective access operation has occurred
Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal
reflective access by
com.google.inject.internal.cglib.core.$ReflectUtils$1
(file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to method

java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Sep 24 02:33:43 castle Controller[1093]: WARNING: Please consider
reporting this to the maintainers of
com.google.inject.internal.cglib.core.$ReflectUtils$1
Sep 24 02:33:43 castle Controller[1093]: WARNING: Use
--illegal-access=warn to enable warnings of further illegal
reflective access operations
Sep 24 02:33:43 castle Controller[1093]: WARNING: All illegal
access operations will be denied in a future release
Sep 24 02:33:44 castle Controller[1093]: 02:33:44.962 [Main] INFO 
LINSTOR/Controller - SYSTEM - Dependency injection finished: 1915ms
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.533 [Main] INFO 
LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.736 [Main] INFO 
LINSTOR/Controller - SYSTEM - Starting service instance
'TimerEventService' of type TimerEventService
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.740 [Main] INFO 
LINSTOR/Controller - SYSTEM - Initializing the database connection
pool
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.303 [Main] ERROR
LINSTOR/Controller - SYSTEM - Database initialization error
[Report number 5F6B78E5-0-00]
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.343 [Thread-2]
INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.344 [Thread-2]
I

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

Thanks, I saw that, but I'm not sure why it is reporting that. I'm starting the 
service from root, and the config didn't change from the old package to the new 
(I didn't change it, just compile new package and install)...

Do you know where it gets the username/password from?


On 24 September 2020 3:13:38 am AEST, "Mindaugas Milinavičius" 
 wrote:
>Caused by: org.h2.jdbc.JdbcSQLException: *Wrong user name or password*
>[28000-197]
>
>
>
>Pagarbiai
>Mindaugas Milinavičius
>UAB STARNITA
>Direktorius
>http://www.clustspace.com
>LT: +37068882880
>RU: +7913933
>
>Tomorrow's possibilities today
><http://www.clustspace.com/>
>
>   - 1 core CPU, 512MB RAM, 20GB (€2.50)
>   - 1 core CPU, 1GB RAM, 30GB (€5.00)
>   - 2 core CPU, 2GB RAM, 40GB (€10.00)
>   - 2 core CPU, 4GB RAM, 60GB (€20.00)
>   - 4 core CPU, 8GB RAM, 80GB (€40.00)
>
>
>
>On Wed, Sep 23, 2020 at 7:54 PM Adam Goryachev <
>mailingli...@websitemanagers.com.au> wrote:
>
>> Hi,
>>
>> After upgrading on one of my test nodes I got some errors, so I
>thought to
>> upgrade the controller, and now I am unable to start the controller.
>>
>> Here is what I get in the system log:
>>
>> Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller
>Service.
>> Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module Controller
>> Sep 24 02:33:41 castle Controller[1093]: Version:1.9.0
>> (678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
>> Sep 24 02:33:41 castle Controller[1093]: Build time:
>> 2020-09-23T10:27:49+00:00
>> Sep 24 02:33:41 castle Controller[1093]: Java Version:   11
>> Sep 24 02:33:41 castle Controller[1093]: Java VM:Debian,
>> Version 11.0.8+10-post-Debian-1deb10u1
>> Sep 24 02:33:41 castle Controller[1093]: Operating system:   Linux,
>> Version 4.19.0-10-amd64
>> Sep 24 02:33:41 castle Controller[1093]: Environment:amd64, 2
>> processors, 1988 MiB memory reserved for allocations
>> Sep 24 02:33:41 castle Controller[1093]: System components
>initialization
>> in progress
>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main] INFO
>> LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main] INFO
>> LINSTOR/Controller - SYSTEM - Log directory set to:
>> '/var/log/linstor-controller'
>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main] WARN
>> io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN, Sentry
>operations
>> will do nothing! See documentation:
>https://docs.sentry.io/clients/java/
>> ***
>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Loading API classes started.
>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main] INFO
>> LINSTOR/Controller - SYSTEM - API classes loading finished: 611ms
>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Dependency injection started.
>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension
>module
>> "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Extension module
>> "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is
>not
>> installed
>> Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal
>reflective
>> access operation has occurred
>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal reflective
>> access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (
>> file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to method
>>
>java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Please consider
>> reporting this to the maintainers of
>> com.google.inject.internal.cglib.core.$ReflectUtils$1
>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Use
>> --illegal-access=warn to enable warnings of further illegal
>reflective
>> access operations
>> Sep 24 02:33:43 castle Controller[1093]: WARNING: All illegal access
>> operations will be denied in a future release
>> Sep 24 02:33:44 castle Controller[1093]: 02:33:44.962 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Dependency injection finished: 1915ms
>> Sep 24 02:33:45 castle Controller[1093]: 02:33:45.533 [Main] INFO
>> LINSTOR/Controller - SYSTEM - Initializing authentication s

Re: [DRBD-user] linstor-server 1.9.0 release

[Adam Goryachev]

Hi,

After upgrading on one of my test nodes I got some errors, so I thought 
to upgrade the controller, and now I am unable to start the controller.


Here is what I get in the system log:

Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller Service.
Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module Controller
Sep 24 02:33:41 castle Controller[1093]: Version:    1.9.0 
(678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
Sep 24 02:33:41 castle Controller[1093]: Build time: 
2020-09-23T10:27:49+00:00

Sep 24 02:33:41 castle Controller[1093]: Java Version:   11
Sep 24 02:33:41 castle Controller[1093]: Java VM: Debian, Version 
11.0.8+10-post-Debian-1deb10u1
Sep 24 02:33:41 castle Controller[1093]: Operating system: Linux, 
Version 4.19.0-10-amd64
Sep 24 02:33:41 castle Controller[1093]: Environment: amd64, 2 
processors, 1988 MiB memory reserved for allocations
Sep 24 02:33:41 castle Controller[1093]: System components 
initialization in progress
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main] INFO 
LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main] INFO 
LINSTOR/Controller - SYSTEM - Log directory set to: 
'/var/log/linstor-controller'
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main] WARN 
io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN, Sentry operations 
will do nothing! See documentation: https://docs.sentry.io/clients/java/ ***
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main] INFO 
LINSTOR/Controller - SYSTEM - Loading API classes started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main] INFO 
LINSTOR/Controller - SYSTEM - API classes loading finished: 611ms
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main] INFO 
LINSTOR/Controller - SYSTEM - Dependency injection started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO 
LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension 
module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO 
LINSTOR/Controller - SYSTEM - Extension module 
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not 
installed
Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal reflective 
access operation has occurred
Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal reflective 
access by com.google.inject.internal.cglib.core.$ReflectUtils$1 
(file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to method 
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Sep 24 02:33:43 castle Controller[1093]: WARNING: Please consider 
reporting this to the maintainers of 
com.google.inject.internal.cglib.core.$ReflectUtils$1
Sep 24 02:33:43 castle Controller[1093]: WARNING: Use 
--illegal-access=warn to enable warnings of further illegal reflective 
access operations
Sep 24 02:33:43 castle Controller[1093]: WARNING: All illegal access 
operations will be denied in a future release
Sep 24 02:33:44 castle Controller[1093]: 02:33:44.962 [Main] INFO 
LINSTOR/Controller - SYSTEM - Dependency injection finished: 1915ms
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.533 [Main] INFO 
LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.736 [Main] INFO 
LINSTOR/Controller - SYSTEM - Starting service instance 
'TimerEventService' of type TimerEventService
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.740 [Main] INFO 
LINSTOR/Controller - SYSTEM - Initializing the database connection pool
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.303 [Main] ERROR 
LINSTOR/Controller - SYSTEM - Database initialization error [Report 
number 5F6B78E5-0-00]
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.343 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Shutdown in progress
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.344 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Shutting down service instance 
'TaskScheduleService' of type TaskScheduleService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Waiting for service instance 
'TaskScheduleService' to complete shutdown
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Shutting down service instance 
'DatabaseService' of type DatabaseService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.349 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Waiting for service instance 
'DatabaseService' to complete shutdown
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.350 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Shutting down service instance 
'TimerEventService' of type TimerEventService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.351 [Thread-2] INFO  
LINSTOR/Controller - SYSTEM - Waiting for service instance 
'TimerEventService' to 

Re: [DRBD-user] linstor issues

[Adam Goryachev]

Hi all,

I've finally had another crack at this, and eventually got it working.

I think I had all of the following issues at various stages, so just 
mentioning them here for other future people who find they have similar 
issues.


1) Initially, I had a firewall blocking connections between servers. I 
fixed this early on, but not sure if it messed up some of the config and 
never recovered.


2) Somehow, some of the servers ended up with old 8.4 kernel module 
version installed. I think this was related to installing new kernels 
from debian, but for some reason it wouldn't update matching header 
files, and so dkms couldn't recompile the module for the new kernel. 
Thus, reverting to the in-kernel 8.4 version.


3) Finally, I think the hardest issue to track down was that there was a 
copy of drbdadm and associated drbd-utils installed under /usr/local and 
these were taking priority over the package installed version. Thus, 
drbdadm couldn't find the drbd resource config files because linstor was 
putting them in /etc/drbd.d and drbdadm was looking in /usr/local/etc/drbd.d


Question: Is this something that linstor should be aware of (ie, storing 
the resource files in the same location as the installed/used drbdadm 
executable expects them to go)? I guess it's a pretty rare issue, but 
curious if it could be avoided easily.


After a few complete wipe/reinstalls, I found and fixed these issues, 
and now looks like I have a working installation.


linstor node list
╭──╮
┊ Node   ┊ NodeType  ┊ Addresses  ┊ State  ┊
╞══╡
┊ castle ┊ SATELLITE ┊ 192.168.5.204:3366 (PLAIN) ┊ Online ┊
┊ san5   ┊ SATELLITE ┊ 192.168.5.205:3366 (PLAIN) ┊ Online ┊
┊ san6   ┊ SATELLITE ┊ 192.168.5.206:3366 (PLAIN) ┊ Online ┊
╰──╯

linstor storage-pool list
╭───╮
┊ StoragePool  ┊ Node   ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ 
TotalCapacity ┊ CanSnapshots ┊ State ┊

╞═══╡
┊ DfltDisklessStorPool ┊ castle ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san5   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san6   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ pool_hdd ┊ castle ┊ LVM  ┊ vg_hdd   ┊ 2.95 TiB 
┊  3.44 TiB ┊ False    ┊ Ok    ┊
┊ pool_hdd ┊ san5   ┊ LVM  ┊ vg_hdd   ┊ 3.87 TiB 
┊  4.36 TiB ┊ False    ┊ Ok    ┊
┊ pool_hdd ┊ san6   ┊ LVM  ┊ vg_hdd   ┊ 1.26 TiB 
┊  1.75 TiB ┊ False    ┊ Ok    ┊

╰───╯

Thank you for those that replied and helped out along the way, I'm 
finally feeling a lot more confident in moving forward with this 
project. I'll no doubt be asking a few more questions along the way, but 
hopefully there won't be anything quite as unexpected as this (#3) 
happening.


Regards,
Adam


On 14/7/20 16:43, Adam Goryachev wrote:


I'm having another crack at this, I think it will be worth it once it 
works.


Firstly, another documentation error:

https://www.linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-using_the_linstor_client

In case anything goes wrong with the storage pool’s VG/zPool, e.g. 
the VG having been renamed or somehow became invalid you can delete 
the storage pool in LINSTOR with the following command, given that 
only resources with all their volumes in the so-called ‘lost’ storage 
pool are attached. This feature is available since LINSTOR v0.9.13.


# linstor storage-pool lost alpha pool_ssd

linstor storage-pool lost castle vg_hdd
usage: linstor storage-pool [-h]
    {create, delete, list, list-properties,
    set-property} ...
linstor storage-pool: error: argument {create, delete, list, 
list-properties, set-property}: invalid choice: 'lost' (choose from 
'create', 'c', 'delete', 'd', 'list', 'l', 'list-properties', 'lp', 
'set-property', 'sp')


Changing to use delete instead of lost:

castle:~# linstor storage-pool delete castle vg_hdd
ERROR:
Description:
    Storage pool definition 'vg_hdd' not found.
Cause:
    The specified storage pool definition 'vg_hdd' could not be found 
in the database

Correction:
    Create a storage pool definition 'vg_hdd' first.
Details:
    Node: castle, Storage pool name: vg_hdd
Show reports:
    linstor error-reports show 5F0D500C-0-00
castle:~# linstor storage-pool list
╭───╮
┊ StoragePool

Re: [DRBD-user] linstor issues

[Adam Goryachev]

I'm having another crack at this, I think it will be worth it once it works.

Firstly, another documentation error:

https://www.linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-using_the_linstor_client

In case anything goes wrong with the storage pool’s VG/zPool, e.g. the 
VG having been renamed or somehow became invalid you can delete the 
storage pool in LINSTOR with the following command, given that only 
resources with all their volumes in the so-called ‘lost’ storage pool 
are attached. This feature is available since LINSTOR v0.9.13.


# linstor storage-pool lost alpha pool_ssd

linstor storage-pool lost castle vg_hdd
usage: linstor storage-pool [-h]
    {create, delete, list, list-properties,
    set-property} ...
linstor storage-pool: error: argument {create, delete, list, 
list-properties, set-property}: invalid choice: 'lost' (choose from 
'create', 'c', 'delete', 'd', 'list', 'l', 'list-properties', 'lp', 
'set-property', 'sp')


Changing to use delete instead of lost:

castle:~# linstor storage-pool delete castle vg_hdd
ERROR:
Description:
    Storage pool definition 'vg_hdd' not found.
Cause:
    The specified storage pool definition 'vg_hdd' could not be found 
in the database

Correction:
    Create a storage pool definition 'vg_hdd' first.
Details:
    Node: castle, Storage pool name: vg_hdd
Show reports:
    linstor error-reports show 5F0D500C-0-00
castle:~# linstor storage-pool list
╭───╮
┊ StoragePool  ┊ Node   ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ 
TotalCapacity ┊ CanSnapshots ┊ State ┊

╞═══╡
┊ DfltDisklessStorPool ┊ castle ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san5   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san6   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ pool ┊ castle ┊ LVM  ┊ vg_hdd   ┊ 2.95 TiB 
┊  3.44 TiB ┊ False    ┊ Ok    ┊
┊ pool ┊ san5   ┊ LVM  ┊ vg_hdd   ┊ 3.87 TiB 
┊  4.36 TiB ┊ False    ┊ Ok    ┊
┊ pool ┊ san6   ┊ LVM  ┊ vg_ssd   ┊ 1.26 TiB 
┊  1.75 TiB ┊ False    ┊ Ok    ┊

╰───╯

I was hoping I could just remove the storage pool from castle (since it 
doesn't seem to be working properly), and then destroy it, re-create it, 
and then re-add it and see if that solves the problem. However, while it 
seems to exist, it also doesn't (can't delete it).


Possibly part of the cause of my original problem is that I have a 
script that automatically creates a snapshot for each LV, and this 
created a snapshot of testvm1_0 named 
backup_testvm1_0_blahblah I've now manually deleted that, and 
fixed my script to avoid messing with the VG allocated to linstor, but 
so far, there is no change in the current status (as per below).


Would appreciate any suggestions on what might be going wrong, and/or 
how to fix it?


Regards,
Adam



On 24/6/20 11:46, Adam Goryachev wrote:



On 23/6/20 21:53, Gábor Hernádi wrote:

Hi,

apparently something is quite broken... maybe it's somehow your setup 
or environment, I am not sure...


linstor resource list

╭╮
┊ ResourceName ┊ Node   ┊ Port ┊ Usage  ┊ Conns  
┊    State ┊

╞╡
┊ testvm1  ┊ castle ┊ 7000 ┊ ┊ ┊ 
Unknown ┊
┊ testvm1  ┊ san5   ┊ 7000 ┊ ┊ ┊ 
Unknown ┊
┊ testvm1  ┊ san6   ┊ 7000 ┊ Unused ┊ Connecting(san5,castle)
┊ UpToDate ┊

╰╯

This looks like some kind of network issues.

# linstor storage-pool list --groupby Size

However, the second command produces a usage error (documentation
bug perhaps).


Thanks for reporting, we will look into this.

WARNING:
Description:
    No active connection to satellite 'san5'
Details:
    The controller is trying to (re-) establish a connection to
the satellite. The controller stored the changes and as soon the
satellite is connected, it will receive this update.


So Linstor has obviously no connection to satellite 'san5'.

[95078.599813] drbd testvm1 castle: conn( Unconnected -> Connecting )
[95078.604454] drbd testvm1 san5: conn( Unconnected -> Connecting )


... and DRBD apparently also has troubles connecting...

linst

Re: [DRBD-user] linstor issues

[Adam Goryachev]

On 23/6/20 21:53, Gábor Hernádi wrote:

Hi,

apparently something is quite broken... maybe it's somehow your setup 
or environment, I am not sure...


linstor resource list

╭╮
┊ ResourceName ┊ Node   ┊ Port ┊ Usage  ┊ Conns  
┊    State ┊

╞╡
┊ testvm1  ┊ castle ┊ 7000 ┊ ┊ ┊ 
Unknown ┊
┊ testvm1  ┊ san5   ┊ 7000 ┊ ┊ ┊ 
Unknown ┊
┊ testvm1  ┊ san6   ┊ 7000 ┊ Unused ┊ Connecting(san5,castle)
┊ UpToDate ┊

╰╯

This looks like some kind of network issues.

# linstor storage-pool list --groupby Size

However, the second command produces a usage error (documentation
bug perhaps).


Thanks for reporting, we will look into this.

WARNING:
Description:
    No active connection to satellite 'san5'
Details:
    The controller is trying to (re-) establish a connection to
the satellite. The controller stored the changes and as soon the
satellite is connected, it will receive this update.


So Linstor has obviously no connection to satellite 'san5'.

[95078.599813] drbd testvm1 castle: conn( Unconnected -> Connecting )
[95078.604454] drbd testvm1 san5: conn( Unconnected -> Connecting )


... and DRBD apparently also has troubles connecting...

linstor n l
╭───╮
┊ Node   ┊ NodeType  ┊ Addresses  ┊ State   ┊
╞═══╡
┊ castle ┊ SATELLITE ┊ 192.168.5.204:3366
 (PLAIN) ┊ Unknown ┊
┊ san5   ┊ SATELLITE ┊ 192.168.5.205:3366
 (PLAIN) ┊ Unknown ┊
┊ san6   ┊ SATELLITE ┊ 192.168.5.206:3366
 (PLAIN) ┊ Unknown ┊
╰───╯


Now  this is really strange. I will spare you with some details, but I 
assume you have triggered some bad exception in Linstor which somehow 
killed a necessary thread.

You should check
   linstor err list
and see if you can find some related error reports.
Also, restarting the controller might help you here.


Thank you!

linstor err list showed a list of errors, but the contents didn't make a 
lot of sense to me. Let me know if you are interested in them, and I can 
send them.


I did a systemctl restart linstor-controller.service on san6, and things 
started looking much better.


linstor n l
╭──╮
┊ Node   ┊ NodeType  ┊ Addresses  ┊ State  ┊
╞══╡
┊ castle ┊ SATELLITE ┊ 192.168.5.204:3366 (PLAIN) ┊ Online ┊
┊ san5   ┊ SATELLITE ┊ 192.168.5.205:3366 (PLAIN) ┊ Online ┊
┊ san6   ┊ SATELLITE ┊ 192.168.5.206:3366 (PLAIN) ┊ Online ┊
╰──╯

So, all nodes agree that they are now online and talking to each other. 
I assume this proves there is no network issues.


linstor resource list
╭─╮
┊ ResourceName ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊  State ┊
╞═╡
┊ testvm1  ┊ castle ┊ 7000 ┊    ┊ ┊    Unknown ┊
┊ testvm1  ┊ san5   ┊ 7000 ┊ Unused ┊ Connecting(castle) ┊ 
SyncTarget(12.67%) ┊
┊ testvm1  ┊ san6   ┊ 7000 ┊ Unused ┊ Connecting(castle) ┊   
UpToDate ┊

╰─╯

From this, it looks like san6 (the controller) thinks it has the up to 
date data, probably based on the fact it was created there first or 
something. The data is syncing to san5 (in progress, and progressing 
steadily), so that is good also. However, castle doesn't seem to be 
syncing/connecting.


On castle, I see this:

Jun 24 11:01:55 castle Satellite[7499]: 11:01:55.177 [DeviceManager] 
ERROR LINSTOR/Satellite - SYSTEM - Failed to create meta-data for DRBD 
volume testvm1/0 [Report number 5EF2A316-31431-02]


linstor err show give this:

ERROR REPORT 5EF2A316-31431-02



Application:    LINBIT® LINSTOR
Module: Satellite
Version:    1.7.1
Build ID: 6760637d6fae7a5862103ced4ea0ab0a758861f9
Build time: 2020-05-14T13:14:11+00:00
Error time: 2020-06-24 11:01:55
Node:   castle



Reported error:
===

Description:
    Failed to 

[DRBD-user] linstor issues

[Adam Goryachev]

I've tried to follow the limited documentation on installing DRBD 9 and 
linstor, and sort of managed to get things working. I have three nodes 
(castle, san5 and san6). I've re-built the various ubuntu packages under 
debian, and installed on debian buster on all three machines:


drbd-dkms_9.0.22-1ppa1~bionic1_all.deb
drbd-utils_9.13.1-1ppa1~bionic1_amd64.deb
linstor-controller_1.7.1-1ppa1~bionic1_all.deb
linstor-satellite_1.7.1-1ppa1~bionic1_all.deb
linstor-common_1.7.1-1ppa1~bionic1_all.deb
python-linstor_1.1.1-1ppa1~bionic1_all.deb
linstor-client_1.1.1-1ppa1~bionic1_all.deb

After adding the three nodes I had this output:
linstor node list
╭──╮
┊ Node   ┊ NodeType  ┊ Addresses  ┊ State  ┊
╞══╡
┊ castle ┊ SATELLITE ┊ .204:3366 (PLAIN) ┊ Online ┊
┊ san5   ┊ SATELLITE ┊ .205:3366 (PLAIN) ┊ Online ┊
┊ san6   ┊ SATELLITE ┊ .206:3366 (PLAIN) ┊ Online ┊
╰──╯

Then I added some storage pools:
linstor storage-pool list
╭───╮
┊ StoragePool  ┊ Node   ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ 
TotalCapacity ┊ CanSnapshots ┊ State ┊

╞═══╡
┊ DfltDisklessStorPool ┊ castle ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san5   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ san6   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok    ┊
┊ pool ┊ castle ┊ LVM  ┊ vg_hdd   ┊ 3.44 TiB 
┊  3.44 TiB ┊ False    ┊ Ok    ┊
┊ pool ┊ san5   ┊ LVM  ┊ vg_hdd   ┊ 4.36 TiB 
┊  4.36 TiB ┊ False    ┊ Ok    ┊
┊ pool ┊ san6   ┊ LVM  ┊ vg_ssd   ┊ 1.75 TiB 
┊  1.75 TiB ┊ False    ┊ Ok    ┊

╰───╯

Again, everything was looking pretty good.

So, I tried to create a resource, and then I got this:

linstor resource list
╭╮
┊ ResourceName ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊    State ┊
╞╡
┊ testvm1  ┊ castle ┊ 7000 ┊    ┊ ┊  Unknown ┊
┊ testvm1  ┊ san5   ┊ 7000 ┊    ┊ ┊  Unknown ┊
┊ testvm1  ┊ san6   ┊ 7000 ┊ Unused ┊ Connecting(san5,castle) ┊ 
UpToDate ┊

╰╯

There hasn't been any change in over 24 hours, so I'm guessing there is 
something stuck/not working, but I don't seem to have many clues on what 
it might be.


I've checked through the docs at: 
https://www.linbit.com/drbd-user-guide/linstor-guide-1_0-en/ and found 
these two commands in section 2.7 Checking the state of your cluster:


# linstor node list
# linstor storage-pool list --groupby Size

However, the second command produces a usage error (documentation bug 
perhaps). Editing the command to something valid produces:

linstor storage-pool list --groupby Node
╭─╮
┊ StoragePool  ┊ Node   ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ 
TotalCapacity ┊ CanSnapshots ┊ State   ┊

╞═╡
┊ DfltDisklessStorPool ┊ castle ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok  ┊
┊ pool ┊ castle ┊ LVM  ┊ vg_hdd   ┊ 3.44 TiB 
┊  3.44 TiB ┊ False    ┊ Ok  ┊
┊ DfltDisklessStorPool ┊ san5   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Warning ┊
┊ pool ┊ san5   ┊ LVM  ┊ vg_hdd ┊  
┊   ┊ False    ┊ Warning ┊
┊ DfltDisklessStorPool ┊ san6   ┊ DISKLESS ┊ ┊  
┊   ┊ False    ┊ Ok  ┊
┊ pool ┊ san6   ┊ LVM  ┊ vg_ssd   ┊ 1.26 TiB 
┊  1.75 TiB ┊ False    ┊ Ok  ┊

╰─╯
WARNING:
Description:
    No active connection to satellite 'san5'
Details:
    The controller is trying to (re-) establish a connection to the 
satellite. The controller stored the changes and as soon the satellite 
is connected, it will receive this update.


Note, after waiting approx 20hours, san5 was shutdown cleanly, so is 
currently offline.


dmesg on san6 includes this:
[95078.272184] drbd testvm1: Starting worker thread (from drbdsetup [2398])
[95078.285272] drbd testvm1 castle: Starting sender 

Re: [DRBD-user] drbd & multicast?

[Adam Goryachev]

On 7/4/20 02:44, Volodymyr Litovka wrote:


Hi Robert,

please see below

On 06.04.2020 17:23, Robert Altnoeder wrote:

On 06 Apr 2020, at 10:17, Volodymyr Litovka  wrote:

To avoid this, I'd propose to add additional layer like proxy, which will:

- reside on every satellite
- receive data over unicast
** thus, drbd code will get minimal changes (now - it sends unicast data
to 1+ neighbors, after changes - it will send the same unicast to single
neighbor)
** to minimize delay - use local sockets
- resend it over multicast
- but manage control traffic (e.g. acknowledgments from remote peers)
over unicast

This would probably still require many changes in the DRBD kernel module, add 
another layer of complexity, another component that can fail independently, and 
makes the system as a whole harder to maintain and troubleshoot.

Delay would probably also be rather unpredictable, because different threads in 
kernel- and user-space must be activated and paused frequently for the IPC to 
work, and Linux, as a monolithical kernel, does not offer any specialized 
mechanisms for direct low-latency context switches/thread activation in a chain 
of I/O servers like those mechanisms that are found in most microkernels, or at 
least something in the general direction like e.g. “door calls” in the SunOS 
kernel (the kernel of the Solaris OS).


Well, I fully believe in what you're saying and my try to find "a 
better solution" doesn't look too convincing :-)



Multicast in DRBD would certainly make sense in various scenarios, but it would 
probably have to be implemented directly in DRBD.


Nice to hear this ;-)


Anyway, I don’t see that much difference between diskless nodes and nodes with 
storage. Any one of these nodes always sends write requests to all connected 
storage nodes, the only difference with diskless nodes is that they also use 
the replication link for reading data, which storage nodes rather do locally 
(load-balancing may cause read requests over the network too). So the only 
thing that would make write performance on a diskless node worse than write 
performance on a node with local storage would be network saturation due to 
lots of read requests putting load on the network.


I see a bit different picture, in fact. I'm using VM (disk is 
/dev/drbd/by-res/m1/0) to produce load. I launch test VM (using virsh) 
on different nodes and get corresponding resource usage like this:


# linstor resource list
╭─╮
┊ ResourceName ┊ Node  ┊ Port ┊ Usage  ┊ Conns ┊State ┊
╞═╡
┊ m1   ┊ stor1 ┊ 7000 ┊ Unused ┊ Ok┊ UpToDate ┊
┊ m1   ┊ stor2 ┊ 7000 ┊ Unused ┊ Ok┊ UpToDate ┊
┊ m1   ┊ stor3 ┊ 7000 ┊ InUse  ┊ Ok┊ Diskless ┊
╰─╯
# linstor resource list-volumes
╭───╮
┊ Node  ┊ Resource ┊ StoragePool  ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊ 
Allocated ┊ InUse  ┊    State ┊
╞═══╡
┊ stor1 ┊ m1   ┊ drbdpool ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
50.01 GiB ┊ Unused ┊ UpToDate ┊
┊ stor2 ┊ m1   ┊ drbdpool ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊ 
50.01 GiB ┊ Unused ┊ UpToDate ┊
┊ stor3 ┊ m1   ┊ DfltDisklessStorPool ┊ 0 ┊    1000 ┊ /dev/drbd1000 ┊   
    ┊ InUse  ┊ Diskless ┊
╰───╯

when m1 is InUse on stor2 ("disk" node) and I launch 'dd' there, I see 
the following tcpdump output on host (stor2):


# tcpdump -i eno4 'src host stor2 and dst port 7000'
[ ... lot of similar packets to stor1 and no packets to stor3 ... ]
18:59:22.640966 IP stor2.39897 > stor1.afs3-fileserver: Flags [.], seq 
1073298560:1073363720, ack 1, win 18449, options [nop,nop,TS val 2730290186 ecr 
2958297760], length 65160
18:59:22.641495 IP stor2.39897 > stor1.afs3-fileserver: Flags [P.], seq 
1073363720:1073426344, ack 1, win 18449, options [nop,nop,TS val 2730290186 ecr 
2958297761], length 62624
18:59:22.642053 IP stor2.39897 > stor1.afs3-fileserver: Flags [.], seq 
1073426344:1073491504, ack 1, win 18449, options [nop,nop,TS val 2730290187 ecr 
2958297761], length 65160
18:59:22.642606 IP stor2.39897 > stor1.afs3-fileserver: Flags [.], seq 
1073491504:1073556664, ack 1, win 18449, options [nop,nop,TS val 2730290187 ecr 
2958297761], length 65160

when m1 is InUse on stor3 ("diskless" node) and I launch 'dd' there, I 
see the following tcpdump output on host (stor3):


# tcpdump -i eno4 'src host stor3 and dst port 7000'
[ ... lot of similar packets to both stor1 and stor2 ... ]
19:05:56.451425 IP stor3.59577 > stor1.afs3-fileserver: Flags [.], seq 
1069351304:1069416464, ack 16425, win 11444, 

Re: [DRBD-user] PingAck did not arrive in time.

[Adam Goryachev]

On 25/6/19 12:58 am, Robert Altnoeder wrote:

On 6/24/19 3:39 PM, Oleksiy Evin wrote:

At this moment we're able to achieve about 1.2GB while replication
which is little bit slower then a single enclosure speed, but enough
for all our needs. Potentially we may grow up above PB next year or two.

I don't think so, at least not with a single volume on the current
version of DRBD, because 1 PiB is currently the maximum gross size of a
single DRBD volume.


Do you advise any different setup?

I would certainly recommend splitting the data if it's possible. I know
that 4k/8k video material can be very large in size, but I guess those
~600 TiB of data is not the working set size for a single movie
editing/finish/effects/... project, or is it?
If it is rather something like an archive, or a single volume with mixed
content, then partitioning by either date or content type could be a
possible solution.


Is it possible to use Linux MD on top of DRBD devices? ie, use 
/dev/drbd[0-3] to create a RAID0 array?


Or I guess using them as PV's and then creating a single LV across all 
of them?


Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] COnfused about shared partitions

[Adam Goryachev]

On 5/4/19 5:09 am, JCA wrote:
I believe that you are right in your assessment. Unfortunately, after 
this I seem to be stuck in a situation in which both nodes are not 
connected through DRBD:


In A:

# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by 
mockbuild@, 2018-11-03 01:26:55


 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-
    ns:0 nr:0 dw:4 dr:2161 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:336

#  drbdadm status
my-data role:Primary
  disk:UpToDate
  peer connection:Connecting

In B:

# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by 
mockbuild@, 2018-11-03 01:26:55


 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:24

# drbdadm status
my-data role:Secondary
  disk:UpToDate
  peer connection:Connecting

I have tried numerous things - including rebooting both VMs where the 
nodes live - to no avail.


Sounds like both sides have been a primary and so now they both think 
the other is out of date (ie, split brain). What you will need to do is 
tell DRBD which one should discard it's data and become secondary, you 
can then reconnect them and DRBD will re-copy the data, re-sync, and get 
back into a working state.


This is a good lesson to work out why you ended up with a split brain 
situation it sounds like:


1. server A was up while server B was down

2. server A went down, and then server B came up

3. server A then came up without any valid network connection to server B

or, you have a non-standard or incorrect config. DRBD (in my experience) 
would notice, catch, and prevent you from getting into this situation.


You should also refer to your log files, they will contain lots of 
useful information, as well as details that you should post to the list 
so we can see what mess you have created


Hope it helps..

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] COnfused about shared partitions

[Adam Goryachev]

     Daemon Status:
              corosync: active/enabled
              pacemaker: active/enabled
              pcsd: active/enabled

On listing the contents of /var/lib/my-data on A, nothing is found 
there. However, on listing the contents of /var/lib/my-data on B, I 
find files f1 and f2. So, B has taken over and is now accessing, 
thanks to DRDB,  the filesystem created in A earlier on.


Next I started and stopped both A and B (the nodes, not the VMs) until 
I got back the original situation - namely, A is active, B is passive, 
/var/log/my-data contains f1 and f2 in A, and nothing in B.


Next I did the following: In A I removed both f1 and f2, and added new 
files, f3, and f4. I did this while A is the active node. After this I 
stopped the A node from A as above. Having done this, the output from 
pcs status in B is as described above. Here's the thing:


In this situation, when I list the contents of /var/lib/my-data in A, 
I find it to be empty - as expected, for A has been stopped. When I 
list the contents of /var/lib/my-data in B, however, what I see is 
files f1 and f2, not f3 and f4. I was expecting for any changes made 
on /var/lib/my-data while A was active to be made visible to B when it 
takes over.


Is my expectation misplaced? Or perhaps the changes to 
/var/lib/my-data have to be made by some resource explicitly managed 
by PaceMaker, while changes done by hand are just dismissed? Notice, 
however, that if I bring the A node up again, and make it active, on 
listing /var/lib/my-data on A I see files f3 and f4: the changes I 
made by hand have not just been dismissed.


As you can see, I am very confused.


I have no idea what all your cluster management commands do, but it 
sounds like you are mounting the underlying device on A instead of the 
DRBD device. Thus, the changes you make are not replicated to B.


While A is active and you see f3 and f4, run df /var/lib/my-data and 
provide the output here, or just note that if the device is not 
/dev/drbd then you have made this mistake.


Hope that this helps...

Regards,
Adam

--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 disconnect, double primary -> connect again -> Sometimes "initial Split Brain", sometimes not

[Adam Goryachev]

On 31/12/18 01:12, Jochen Kamuf wrote:


Hi,

I have two identical bare metal machines "server1" and "server2" 
running DRBD_KERNEL_VERSION=9.0.13 on Opensuse Leap15.

I have one resource "res1"

Double Primary is NOT allowed.

If I run the following scenario:

0. "server1" stops the VM that uses the res1 for the virtual hard disk
(to ensure a consistent vm state of the backup and have no read/writes 
on the drbd resource )


1. Set "res1" on both nodes "server1" and "server2" to secondary and 
disconnect

2. Set "res1" on both nodes to primary in standalone mode
3. "server1" starts the VM that uses the res1 for the virtual hard disk
4. "server2"  run the command "ddrescue /dev/drbd/by-res/bb 
/backup/backup.img"


5. When the backup is finished I set "res1" on node"server2" to 
secondary ( while on server1 the res1 is already on primary and gets 
reads/writes )

6. Connect both nodes again

My suggestion, (which doesn't take into account any of your actual 
needs, but takes what you have and adds IMHO a small improvement):


1. Set "res1" on both nodes "server1" and "server2" to secondary and 
disconnect

2. Set "res1" on "server1" ONLY to primary in standalone mode
3. "server1" starts the VM that uses the res1 for the virtual hard disk
4. "server2"  run the command "ddrescue /dev/underlying/device 
/backup/backup.img"
5. When the backup is finished I set "res1" on node "server2" to connect 
( while on server1 the res1 is already on primary and gets reads/writes )


The reason this will work is because DRBD doesn't store any data at the 
beginning of the underlying block device, so you can simply use the 
restored data as a direct disk image. Also, if you ever needed to, you 
could restore the image back to the DRBD device, which would "run out of 
space" but all of the disk content would be restored successfully.


ie, dd if=/backup/backup.img of=/dev/drbd/by-res/bb

You will get an out of space when trying to write the DRBD internal 
data, but that can be ignored, since DRBD will manage/update it's 
internal data anyway (or in reality, you have probably just re-created 
the DRBD data as well).


This avoids any potential for split brain.

An additional consideration is to use LVM below DRBD, so at step 3 you 
would take a snapshot of the underlying DRBD volume, then you can simply 
reconnect DRBD in primary/secondary (step 5), then start the VM (step 3) 
and finally do the backup from the snapshot. Obviously, you would remove 
the snapshot afterwards.


Regards,
Adam

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Offline Resize

[Adam Goryachev]

On 7/12/18 10:15 am, G C wrote:

Not really clear to me but no we're not using an SSD.

(I'm not an expert on LVM so bear with me)
Are you saying that I would add some of the new PV to the existing LV 
so it can be utilized and then perform a pvmove later?
If I'm using part of the new PV as part of the LV how would I ensure 
that it's not overwritten when performing the pvmove later?


Here are my scenarios below maybe you can add what you're thinking 
into it and it will make more sense to me.


Original Scenario:
DISK_1   2TB
pvcreate /dev/sdd
vgcreate vg01 /dev/sdd
lvcreate -n 01lv -l 100%FREE vg01
Device: /dev/vg01/01lv
# drbd setup
drbdadm create-md clusterdb
drbdadm up clusterdb
drbdadm primary --force clusterdb
mkfs.ext4 /dev/drbd0
mkdir /01
mount /dev/drbd0 /01

RESIZE (ONLINE) - really slow currently takes over 20hours for the 
migration, final outcome is that we have only one Physical disk after 
the move

DISK_2   3TB
pvcreate /dev/sde
vgextend vg01 /dev/sde
pvmove /dev/sdd /dev/sde
vgreduce vg01 /dev/sdd
pvremove /dev/sdd
lvextend -l +100%FREE /dev/vg01/01lv
drbdadm -- --assume-clean resize clusterdb
resize2fs /dev/drbd0


For me, it is like this (simple case with SSD's)
pvcreate /dev/sde
vgextend vg01 /dev/sde
lvextend -L+900G /dev/vg01/01lv
drbdadm resize clusterdb

A somewhat simple solution to do what you want might be this (untested, 
so check it on a VM or test env first):

pvcreate /dev/sde
vgextend vg01 /dev/sde
lvcreate -n temp -l  vg01

Note, the size xxx is exactly equal to the size of 01lv which you can 
check with lvdisplay. This will allocate the first part of sde to a new 
LV called temp, it is just here to use up the space.


lvextend -l+100%FREE /dev/vg01/01lv

Note, I don't usually use this 100%FREE method, but I assume it will 
work right...


drbdadm resize clusterdb

This effectively means your LV 01lv is split across sdd and sde, which 
might improve performance, but increases risk if either disk dies.


At any time, you can now lvremove /dev/vg01/temp and can then do your 
pvmove which will move the first 2TB of 01lv from sdd to the beginning 
of sde.


There is a more complicated method, but this is what I would do if faced 
with your problem. I've also successfully used this script:

https://bisqwit.iki.fi/source/lvm2defrag.html

Which can assist you to ensure that the entire disk ends up contiguous 
(though you would need to have at least some temporary space, so better 
not to allocate 100% of the space in the step after you create the temp LV).


Also, why do you allocate 100% of the space? Why not leave some space 
for snapshots, or for future growth? If you had left 100G spare, then 
right now you would simply lvextend to use up the last 100G, and then 
have some time to deal with arranging the upgrade. Just my suggestions, 
but I don't understand your needs.


Regards,
Adam


--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Offline Resize

[Adam Goryachev]

On 7/12/18 9:44 am, G C wrote:
Correct we're adding a new physical device with larger capacity then 
using pvmove to relocate to new device, etc...


The problem is that it takes a long time to complete the pvmove and it 
out of space on the original device (hence the move) we wouldn't be 
able to use the system until the migration was completed.




I am pretty sure you could just allocate space from the new PV to the 
existing LV, that way, the LV will span both the old PV and the new PV. 
If you are using SSD storage, then this won't matter, if you are using 
spinning disks, then you might want to specifically allocate blocks from 
the new PV in the physical position where it will end up when finished, 
and then when you do the pvmove later, it will be contiguous with the 
old and new data all on the new PV.


Does that make sense?

Regards,
Adam




On Thu, Dec 6, 2018 at 4:35 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:


On 7/12/18 9:30 am, G C wrote:

Thank you for your response.

In our scenario we're limited to a single PV on the LVM and while
online does work it takes a long time for the pvmove to complete
from say 2TB to a 3TB PV.

We're trying the offline in hopes that it would speed things up a
lot.


Sorry, I still don't understand. If you have a single PV, what are
you using pvmove for?

In my scenario, I use lvextend and then drbdadm resize.

Or do you mean you are adding additional physical devices with
larger capacity, and then using pvmove to relocate to the new
device, and then using lvextend to increase the size, and then
drbdadm resize? and instead you are planning to drbdadm down
 and then fiddle with the underlying devices, and then
drbdadm up ?

Is the problem that pvmove takes a long time to complete, or that
it affects performance too much during the move? Just curious...

Regards,
Adam




On Thu, Dec 6, 2018 at 4:26 PM Adam Goryachev
mailto:mailingli...@websitemanagers.com.au>> wrote:


On 7/12/18 8:54 am, G C wrote:

Version:
drbd84-utils.x86_64  8.9.2-1.0.1.el7

cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 75EFA3DCEBEF5C771A0FCFC

Following instructions located here for offline with
internal metadata
https://docs.linbit.com/docs/users-guide-8.4/#s-resizing
I always get this error, I've found several issues that are
similar but haven't found anyway to get past this issue.
Architecture
LVM > DRBD

# drbdadm down 

# drbdadm dump-md clusterdb > /tmp/metadata

Found meta data is "unclean", please apply-al first

Command 'drbdmeta 0 v08 /dev/drbd/clusterdb internal
dump-md' terminated with exit code 255




Out of interest, why do you need to use offline resize? I
always have used the online resize with no issues (LVM
underneath). Simply increase the LV (on both nodes) and then
issue a "drbdadm resize " command.

Not sure if that helps, or if you have a reason for using
offline resize, maybe someone else can assist with that.

PS, using 8.4.x versions.

Regards,
    Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

<http://www.websitemanagers.com.au>

-- The information in this e-mail is confidential and may be
legally privileged. It is intended solely for the addressee.
Access to this e-mail by anyone else is unauthorised. If you
are not the intended recipient, any disclosure, copying,
distribution or any action taken or omitted to be taken in
reliance on it, is prohibited and may be unlawful. If you
have received this message in error, please notify us
immediately. Please also destroy and delete the message from
your computer. Viruses - Any loss/damage incurred by
receiving this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
http://lists.linbit.com/mailman/listinfo/drbd-user



-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

<http://www.websitemanagers.com.au>

-- The information in this e-mail is confidential and may be
legally privileged. It is intended solely for the addressee.
Access to this e-mail by anyone else is unauthorised. If you are
not the intended recipient, any disclosure, copying, distribution
or any action taken or omitted to be taken in reliance on it, is
prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and
  

Re: [DRBD-user] Offline Resize

[Adam Goryachev]

On 7/12/18 9:30 am, G C wrote:

Thank you for your response.

In our scenario we're limited to a single PV on the LVM and while 
online does work it takes a long time for the pvmove to complete from 
say 2TB to a 3TB PV.


We're trying the offline in hopes that it would speed things up a lot.


Sorry, I still don't understand. If you have a single PV, what are you 
using pvmove for?


In my scenario, I use lvextend and then drbdadm resize.

Or do you mean you are adding additional physical devices with larger 
capacity, and then using pvmove to relocate to the new device, and then 
using lvextend to increase the size, and then drbdadm resize? and 
instead you are planning to drbdadm down  and then fiddle with 
the underlying devices, and then drbdadm up ?


Is the problem that pvmove takes a long time to complete, or that it 
affects performance too much during the move? Just curious...


Regards,
Adam




On Thu, Dec 6, 2018 at 4:26 PM Adam Goryachev 
<mailto:mailingli...@websitemanagers.com.au>> wrote:



On 7/12/18 8:54 am, G C wrote:

Version:
drbd84-utils.x86_64  8.9.2-1.0.1.el7

cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 75EFA3DCEBEF5C771A0FCFC

Following instructions located here for offline with internal
metadata
https://docs.linbit.com/docs/users-guide-8.4/#s-resizing
I always get this error, I've found several issues that are
similar but haven't found anyway to get past this issue.
Architecture
LVM > DRBD

# drbdadm down 

# drbdadm dump-md clusterdb > /tmp/metadata

Found meta data is "unclean", please apply-al first

Command 'drbdmeta 0 v08 /dev/drbd/clusterdb internal
dump-md' terminated with exit code 255




Out of interest, why do you need to use offline resize? I always
have used the online resize with no issues (LVM underneath).
Simply increase the LV (on both nodes) and then issue a "drbdadm
resize " command.

Not sure if that helps, or if you have a reason for using offline
resize, maybe someone else can assist with that.

PS, using 8.4.x versions.

Regards,
    Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

<http://www.websitemanagers.com.au>

-- The information in this e-mail is confidential and may be
legally privileged. It is intended solely for the addressee.
Access to this e-mail by anyone else is unauthorised. If you are
not the intended recipient, any disclosure, copying, distribution
or any action taken or omitted to be taken in reliance on it, is
prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and
delete the message from your computer. Viruses - Any loss/damage
incurred by receiving this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
http://lists.linbit.com/mailman/listinfo/drbd-user



--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Offline Resize

[Adam Goryachev]

On 7/12/18 8:54 am, G C wrote:

Version:
drbd84-utils.x86_64     8.9.2-1.0.1.el7

cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 75EFA3DCEBEF5C771A0FCFC

Following instructions located here for offline with internal metadata
https://docs.linbit.com/docs/users-guide-8.4/#s-resizing
I always get this error, I've found several issues that are similar 
but haven't found anyway to get past this issue.

Architecture
LVM > DRBD

# drbdadm down 

# drbdadm dump-md clusterdb > /tmp/metadata

Found meta data is "unclean", please apply-al first

Command 'drbdmeta 0 v08 /dev/drbd/clusterdb internal dump-md'
terminated with exit code 255




Out of interest, why do you need to use offline resize? I always have 
used the online resize with no issues (LVM underneath). Simply increase 
the LV (on both nodes) and then issue a "drbdadm resize " command.


Not sure if that helps, or if you have a reason for using offline 
resize, maybe someone else can assist with that.


PS, using 8.4.x versions.

Regards,
Adam

--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] drbdmanage multiple VGs not supported

[Adam Goryachev]

On 03/09/18 19:57, abdollah karimnia wrote:

Dear all,

Is there any way to replicate  different VGs using drbdmanage?
Currently we can add only one VG name (drbdpool) into 
/etc/drbdmanaged.cfg file. Seems that it is not possible to have 
different VGs to replicate.


kind Regards.


I'm not entirely sure what you are asking, but I think you can achieve 
what you want by using the DRBD managed VG to create multiple devices 
(drbd0, drbd1, drbdn...) and each drbd device could be used as a PV for 
a different VG.


drbdpool
drbd0, drbd1, drbd2, drbdn
VG0 VG1 VG2 VGn

You can then create multiple LV's from each of the VG0/VG1/VG2/VGn as 
needed.


I hope that is clear and you can understand it.

Note, you will likely need to ensure that LVM doesn't try to access the 
VG0/VG1/VG2/VGn using the underlying LV's within drbdpool, but will only 
use the devices exported by DRBD. You might also need some extra work to 
have LVM activate those LV's after DRBD has started, depending on how 
you are using them. See the filter options in the lvm.conf.


Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Some info

[Adam Goryachev]

On 12/10/17 06:52, Gandalf Corvotempesta wrote:

2017-10-11 21:22 GMT+02:00 Adam Goryachev <mailingli...@websitemanagers.com.au>:

You can also do that with raid + lvm + drbd... you just need to create a new
drbd as you add a new LV, and also resize the drbd after you resize the LV.

I prefere to keep drbd as minimum. I'm much more familiar with LVM.
If not an issue, i prefere to keep the number of drbd resources as bare minimum.

Except that you should become familiar with DRBD so that when something 
goes wrong, you will be better placed to fix it. If you set it up once 
and don't touch it for three years, then it breaks, you will have no 
idea on what to do or even where to start. You will probably have 
forgotten how it was configured/supposed to work.

If both drives fail on one node, then raid will pass the disk errors up to
DRBD, which will mark the local storage as down, and yes, it will read all
needed data from remote node (writes are always sent to the remote node).
You would probably want to migrate the remote node to primary as quickly as
possible, and then work on fixing the storage.

Why should I migrate the remote node to primary? Any advantage?
Yes, avoids reads from going over the network, reducing latency and 
increasing throughput (depending on bandwidth between nodes). I guess 
this is not a MUST, but just an easy optimisation.



Yes, it is not some bizarre configuration that has never been seen before.
You also haven't mentioned the size of your proposed raid, nor what size you
are planning on growing it to?

Currently, I'm planning to start with 2TB disks. I don't think to go
over 10-12TB

That is a significant growth. I would advise to plan how you will 
achieve that growth now. For example, create a 200GB array, with DRBD + 
LVM etc, then try to grow the array (add extra 200GB partitions to the 
drive) and make sure everything works as expected. A good idea to 
document the process while you are doing this, so that when you need it, 
you have a very good idea on how to proceed. (You should still re-test 
it at that time in case tools have changed/etc).


One thing you have ignored is that DRBD will behave differently with a 
single resource as opposed to multiple resources. For me, this was 
enough of a difference that it made a horrible solution into a viable 
solution (from the end users that were using it, performance was 
terrible with the single resource, and possible with multiple, other 
changes were also made to convert it to highly useful).

Yes, you will always want multiple network paths between the two nodes, and
also fencing. bonding can be used to improve performance, but you should
*also* have an additional network or serial or other connection between the
two nodes which is used for fencing.

Ok.

Any "bare-metal" distribution with DRBD or detailed guide on how to
implement HA?
Something like FreeNAS, or similiar.


No, I just use debian and then configure things as required, for me that 
is the best way to become familiar with the system, and be prepared for 
when things break. I would also strongly advise to read the very good 
documentation, try and read through all of it at least once. (Another 
thank you to linbit for this documentation!)


Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Some info

[Adam Goryachev]

On 12/10/17 07:55, Gandalf Corvotempesta wrote:

For this project I'll use v8
As I would like to use just one big resource, I don't think v9 would 
be able to rebalance a single resource across 4 or 5 nodes


v9 would allow for a 3 node mirror, which improves redundancy and 
resiliency, and I assume makes split-brain avoidance much simpler/more 
reliable to stonith the right node.


For me, I mainly still use v8 in production.

Regards,
Adam

Il 11 ott 2017 10:48 PM, "Yannis Milios" <yannis.mil...@gmail.com 
<mailto:yannis.mil...@gmail.com>> ha scritto:


Are you planning to use DRBD8 or DRBD9?

DRBD8 is limited to 2 nodes(max 3).
DRBD9 can scale to multiple nodes.

For DRBD8 the most common setup is RAID -> DRBD -> LVM  or  RAID
-> LVM -> DRBD
It’s management is way easier than DRBD9.

The most common DRBD9 setups are RAID -> LVM (thin or thick) ->
DRBD  or  HDD  ->  ZFS (thin or thick)  ->  DRBD.
Complicated management...

On Wed, 11 Oct 2017 at 20:52, Gandalf Corvotempesta
<gandalf.corvotempe...@gmail.com
<mailto:gandalf.corvotempe...@gmail.com>> wrote:

2017-10-11 21:22 GMT+02:00 Adam Goryachev
<mailingli...@websitemanagers.com.au
<mailto:mailingli...@websitemanagers.com.au>>:
> You can also do that with raid + lvm + drbd... you just need
to create a new
> drbd as you add a new LV, and also resize the drbd after you
resize the LV.

I prefere to keep drbd as minimum. I'm much more familiar with
LVM.
If not an issue, i prefere to keep the number of drbd
resources as bare minimum.

> If both drives fail on one node, then raid will pass the
disk errors up to
> DRBD, which will mark the local storage as down, and yes, it
will read all
> needed data from remote node (writes are always sent to the
remote node).
> You would probably want to migrate the remote node to
primary as quickly as
> possible, and then work on fixing the storage.

Why should I migrate the remote node to primary? Any advantage?

> Yes, it is not some bizarre configuration that has never
been seen before.
> You also haven't mentioned the size of your proposed raid,
nor what size you
> are planning on growing it to?

Currently, I'm planning to start with 2TB disks. I don't think
to go
over 10-12TB

> Yes, you will always want multiple network paths between the
two nodes, and
> also fencing. bonding can be used to improve performance,
but you should
> *also* have an additional network or serial or other
connection between the
> two nodes which is used for fencing.

Ok.

Any "bare-metal" distribution with DRBD or detailed guide on
how to
implement HA?
Something like FreeNAS, or similiar.
___
drbd-user mailing list
drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
http://lists.linbit.com/mailman/listinfo/drbd-user
<http://lists.linbit.com/mailman/listinfo/drbd-user>

-- 
Sent from Gmail Mobile




--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Some info

[Adam Goryachev]

On 12/10/17 06:14, Gandalf Corvotempesta wrote:

So, let's assume a raid -> drbd -> lvm

starting with a single RAID1, what If I would like to add a second
raid1 converting the existing one to a RAID10 ? drbdadm resize would
be enoguh ?
Correct, assuming you can convert the raid1 to raid10. You might need to 
start with a 2 device RAID10, best to check that procedure now and 
ensure mdadm will properly support this.

keeping lvm as the upper layer would be best, I think, because will
allow me to create logical volumes, snapshot and so on.
You can also do that with raid + lvm + drbd... you just need to create a 
new drbd as you add a new LV, and also resize the drbd after you resize 
the LV.

what happens if local raid totally fails ? the upper layer will stay
up thanks to DRBD fetching data from the other node ?
If both drives fail on one node, then raid will pass the disk errors up 
to DRBD, which will mark the local storage as down, and yes, it will 
read all needed data from remote node (writes are always sent to the 
remote node). You would probably want to migrate the remote node to 
primary as quickly as possible, and then work on fixing the storage.

is "raid -> drbd -> lvm" a standard configuration or something bad? I
don't want to put in production something "custom" and not supported.
Yes, it is not some bizarre configuration that has never been seen 
before. You also haven't mentioned the size of your proposed raid, nor 
what size you are planning on growing it to?



How to prevent splitbrains ? Would be enough to bond the cluster
network ? Any qdevice or fencing to configure ?
Yes, you will always want multiple network paths between the two nodes, 
and also fencing. bonding can be used to improve performance, but you 
should *also* have an additional network or serial or other connection 
between the two nodes which is used for fencing.


Regards,
Adam


2017-10-11 21:07 GMT+02:00 Adam Goryachev <mailingli...@websitemanagers.com.au>:


On 12/10/17 05:10, Gandalf Corvotempesta wrote:

Previously i've asked about DRBDv9+ZFS.
Let's assume a more "standard" setup with DRBDv8 + mdadm.

What I would like to archieve is a simple redundant SAN. (anything
preconfigured for this ?)

Which is best, raid1+drbd+lvm or drbd+raid1+lvm?

Any advantage by creating multiple drbd resources ? I think that a
single DRBD resource is better for administrative point of view.

A simple failover would be enough, I don't need master-master
configuration.

In my case, the best option was raid + lvm + drbd
It allows me to use lvm tools to resize each exported resource as required
easily:
lvmextend...
drbdadm resize ...

However, the main reason was to improve drbd "performance" so that it will
use different counters for each resource instead of a single set of counters
for one massive resource.

BTW, how would you configure drbd + raid + lvm ?

If you do DRBD with a raw drive on each machine, then use raid1 on top
within each local machine, then your raw drbd drive dies, the second raid
member will not contain or participate with DRBD anymore, so the whole node
is failed. This only adds DR ability to recover the user data. I would
suggest this should not be a considered configuration at all (unless I'm
awake to early and am overlooking something).

Actually, assuming machine1 with disk1 + disk2, and machine2 with disk3 +
disk4, I guess you could setup drbd1 between disk1 + disk3, and a drbd2 with
disk2 + disk4, and then create raid on machine 1 with drbd1+drbd2 and raid
on machine2 with drbd1+drbd2 and then use the raid device for lvm. You would
need double the write bandwidth between the two machines. When machine1 is
primary, and a write for the LV, it will be sent to raid which will send the
write to drbd1 and also drbd2. Locally, they are written to disk1 + disk2,
but also those 2 x writes will need to send over the network to machine2, so
it can be written to disk3 (drbd1) and disk4 (drbd2). Still not a sensible
option IMHO.

The two valid options would be raid + drbd + lvm or raid + lvm + drbd (or
just lvm + drbd if you use lvm to handle the raid as well).

Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user


___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Some info

[Adam Goryachev]

On 12/10/17 05:10, Gandalf Corvotempesta wrote:

Previously i've asked about DRBDv9+ZFS.
Let's assume a more "standard" setup with DRBDv8 + mdadm.

What I would like to archieve is a simple redundant SAN. (anything
preconfigured for this ?)

Which is best, raid1+drbd+lvm or drbd+raid1+lvm?

Any advantage by creating multiple drbd resources ? I think that a
single DRBD resource is better for administrative point of view.

A simple failover would be enough, I don't need master-master configuration.

In my case, the best option was raid + lvm + drbd
It allows me to use lvm tools to resize each exported resource as 
required easily:

lvmextend...
drbdadm resize ...

However, the main reason was to improve drbd "performance" so that it 
will use different counters for each resource instead of a single set of 
counters for one massive resource.


BTW, how would you configure drbd + raid + lvm ?

If you do DRBD with a raw drive on each machine, then use raid1 on top 
within each local machine, then your raw drbd drive dies, the second 
raid member will not contain or participate with DRBD anymore, so the 
whole node is failed. This only adds DR ability to recover the user 
data. I would suggest this should not be a considered configuration at 
all (unless I'm awake to early and am overlooking something).


Actually, assuming machine1 with disk1 + disk2, and machine2 with disk3 
+ disk4, I guess you could setup drbd1 between disk1 + disk3, and a 
drbd2 with disk2 + disk4, and then create raid on machine 1 with 
drbd1+drbd2 and raid on machine2 with drbd1+drbd2 and then use the raid 
device for lvm. You would need double the write bandwidth between the 
two machines. When machine1 is primary, and a write for the LV, it will 
be sent to raid which will send the write to drbd1 and also drbd2. 
Locally, they are written to disk1 + disk2, but also those 2 x writes 
will need to send over the network to machine2, so it can be written to 
disk3 (drbd1) and disk4 (drbd2). Still not a sensible option IMHO.


The two valid options would be raid + drbd + lvm or raid + lvm + drbd 
(or just lvm + drbd if you use lvm to handle the raid as well).


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Moving TeraBytes to drbd device

[Adam Goryachev]

ta, or setup a matching sized storage
device and configure external metadata. With that extra space, you
can setup DRBD and it will see the existing FS and data fine. You'll
need a full resync to the new peer, of course. Also, test/practice
outside prod thoroughly to be certain you have the steps down pat.

There are docs on how to do this openly available on LINBIT's website.
If you get stuck on certain steps, post specific questions and we'll help.

--
Digimer
Papers and Projects: https://alteeve.com/w/ "I am, somehow, less
interested in the weight and convolutions of Einstein’s brain than in
the near certainty that people of equal talent have lived and died in
cotton fields and sweatshops." - Stephen Jay Gould



--
Digimer
Papers and Projects: https://alteeve.com/w/ "I am, somehow, less
interested in the weight and convolutions of Einstein’s brain than in
the near certainty that people of equal talent have lived and died in
cotton fields and sweatshops." - Stephen Jay Gould



--
Digimer
Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the 
weight and convolutions of Einstein’s brain than in the near certainty that people of 
equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay 
Gould





--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBDv9 with iSCSI as scaleout SAN

[Adam Goryachev]

Note, all the below relates to my uses of DRBD 8.4 in production. I'm 
assuming most of it will be equally applicable to DRBD9.



On 3/10/17 19:52, Gandalf Corvotempesta wrote:

Just trying to figure out if drbd9 can do the job.

Requirement: a scale-out storage for VMs image hosting (and other
services, but they would be made by creating, in example, an NFS VM on
top of DRBD)

Let's assume a 3-nodes DRBDv9 cluster.
I would like to share this cluster by using iSCSI (or better protocol, if any).
Multiple proxmox nodes sharing this drbd cluster.

Probably, one drbd resource is created for each VM.

Now, some question:

how can I ensure that my iSCSI target is redundant across all nodes in
the cluster ?
What do you mean by redundant? You only have a single iscsi server, this 
is the current DRBD primary server. You would use heartbeat or similar 
to automatically stop the iscsi server, change primary to a different 
server, and then start iscsi server on that machine. Your iscsi clients 
will get no response during this time, which looks like a disk stall. 
Note, it's important to ensure you do this in the correct order:
1) Remove IP address (or firewall so that no response is sent back, no 
ICMP port closed message, no TCP packets, nothing at all).

2) Stop iscsi service
3) Change to secondary
4) Change other server to primary
5) Start iscsi service on new primary server
6) Add IP address, or fix firewall to allow traffic in/out.

When I have to add a fourth or fifth node to drbd cluster, should I
replicate the iscsi target configuration on both ?
Yes, you must ensure the iscsi config is identical on every server which 
could potentially become primary.

Will the drbd resources automatically rebalanced across the new nodes ?
I'm not sure, I suspect you are considering to make one of your DRBD 
nodes primary for some of the resources, and another primary for a 
different group of resources, and then somehow your peers will work out 
which primary to talk to for their iscsi service. This could be possible 
(thinking, definitely you will want to test this first).


Consider if each DRBD resource will have a dedicated IP address. You 
will need to somehow dynamically configure the iscsi service (it is 
possible with iet and messing around in /proc) to listen on this extra 
IP, and serve this extra resource. Doing this individually for each 
resource (ie, the above 6 steps would be repeated once for each 
resource). However, I wonder if this would get you any significant 
benefit? All data will still need to be written to all servers, though I 
suppose reads will be better balanced than an all on one primary system.



Should I change something in the iscsi/proxmox configuration after the
rebalance or is it transparent?
I'm thinking yes... I suspect your heartbeat layer will need to manage 
these changes for you.

Any pitfalls or drawbacks ?
Lots make sure you test a lot... including any and all failure 
modes you can think of, as well as a complete failure (all nodes die and 
recover).


Hopefully someone with more hands on experience with DRBD9 can comment 
further


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBDv9 - some questions

[Adam Goryachev]

On 28/9/17 22:31, Gandalf Corvotempesta wrote:

Hi to all,
Some questions about DRBDv9 (i'm really new to DRBD and DRBDv9 seems
to be a major refactor):

Let's assume a 3-node cluster

a) should I use RAID or creating resources on raw-disks would be ok?
The choice is yours. If you build on raw disks, then the failure of a 
disk will cause the failure of the node. If that meets your data 
protection needs, then OK.
Equally, just because you use RAID, doesn't mean it improves your data 
protection. RAID0 will make it worse, but might improve performance. 
Personally, RAID10 or RAID6 would be my preferred options. A small 
premium for additional drives with a massive improvement in data 
resilience, especially when you factor in the 3 x servers, each with 
RAID10 for example.

b) can I use ZFS on top of DRBD ?

Yes

c) what if I have to aggragate multiple resources to increase spaces ?
Can I use LVM with something like this:

pvcreate /dev/drbd0
pvcreate /dev/drbd1
vgcreate vg0 /dev/drbd0 /dev/drbd1
lvcreate ...

to create a single LV on top of 2 drbd resources ?
I've never tried, but I don't see why not. DRBD provides a block device, 
it doesn't "care" what you are storing on it, whether a FS or LVM...


d) what If node0.disk0 on resource0 will fail and node0.disk1 on
resource1 don't?
My LV spawning both resource will still work as expect but in a little
bit slow mode as drbd has to fetch missing data (due to failed disk0)
from one of the other peer ?
Normally, the resource0 would be on disk0 on both servers, and resource1 
on disk1 on both servers
So if node0.disk0 fails then node1.disk0 will still have the data. You 
could move resource0 to primary on node1 which would solve any 
performance issue, in fact you might see performance improve, since you 
no longer need to write data across to node0.disk0.


e) AFAIK drbd is a network block device, to access this "block device"
should I put an NFS/iSCSI/Whatever in front of it or are there
something integrated in drbd ? (Like NFS inside Gluster)
AFAIK, no, thought it depends on your needs/wants. DRBD9 changes things 
a bit in that you can have multiple satellite nodes which do not have 
local storage, but do "use" the DRBD devices.
In 8.4 which I still use in production, I used iscsi to export the DRBD 
devices to their clients, but I expect if/when I move to DRBD9, I would 
use diskless satellite nodes, and then pass the raw DRBD device to the 
application. This will remove the iscsi layer in my system, which I hope 
might improve performance slightly, but certainly will reduce complexity 
(one less software to go wrong).

f) a 3node cluster is still subject to splitbrain ?

I expect not, there are three* possibilities given node1 node2 and node3
1) node1 by itself, node2 + node3 together
2) all nodes together
3) all nodes alone

* ignoring reshuffles of the same thing, eg, you could have node1 + 
node3 and node2 alone, but basically it is just saying 2 nodes together, 
and one alone.


Clearly, in scenario 1, node1 should shutdown (stonith), and leave nodes 
2 + 3 operational, so no split brain.

In scenario 2, everything is working well, so no SB
In scenario 3, all nodes are alone, so they should all stonith and die. 
This is why you need to build the interconnects to ensure that the 
connections between the nodes is more resilient. Don't just plug a 
single ethernet from each to a single switch, or the switch will die 
killing all your storage. They should have direct connections to each 
other in addition to multiple connections to multiple switches.


Of course, doing this properly can be difficult and expensive.

Regards,
Adam

--
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] fence (unfence really) versus corosync (slow startup)

[Adam Goryachev]

On 10/5/17 18:01, Robert Altnoeder wrote:

On 05/10/2017 04:43 AM, Christian Balzer wrote:

Have Pacemaker start DRBD, then the CIB will be available for unfence.


Easier said than done, as Debian only supplies the LSB init script, which
a "systemctl disable" will NOT disable.
For now brutal "exit 0" solves that, bug report to Debian in progress...
Since it's not required anyway, because Pacemaker will start and stop 
DRBD using its own resource agent:

Sysadmin's bag of tricks says,
rm /etc/rcX.d/drbd
  and maybe
rm /etc/init.d/drbd

(or whatever the script and its symlinks are called)

Checking for DRBD being started upon system boot should probably also 
go on the system update SOP just in case that any update reinstalls or 
otherwise reenables the automatic start of DRBD.
Would a possible solution to this be for corosync to install some file 
in /etc/default/corosync such as DRBDSTART=manual and for DRBD to read 
(if it exists) /etc/default/corosync and simply exit unless 
DRBDSTART=auto ...


The other option is for /etc/default/drbd to include an option for 
UseCorosync=true/false etc..


Well, something along those lines.

Ultimately, this would need to be proposed as a patch to debian, and 
would need to be accepted and added for the packages there, which is not 
likely for at least a few years (at least for a stable release)


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Ultra slowness

[Adam Goryachev]

On 9/5/17 16:31, Steven Maddox wrote:

Hey,

We use DRBD at work in a very simple way... but since rebuilding it
we've found it runs slower than before.
When you say slower than before, how fast was it before, and how fast is 
it now? Do you have actual specific tests/numbers? Also, what changed 
from "before" to "now"?

Setup is as follows...

Sites A and B both have a Debian 8 VM and they also both have a physical
SAN each.

These VM's use Open-iSCSI / Multipath to connect to their respective
SAN's (where 12 real disks are used in raid 6 to make one 30TB virtual
disk, and that virtual disk just has one partition with ext4 fs).

The VM's replicate that partition between them using DRBD.  Site A is
always primary and Site B always secondary (unless a dragon takes out
Site A - long story).

Protocol 'A' was chosen as I want writes to happen at Site A immediately
as the data goes to Site B over a WAN link.

The only reason the below config doesn't look like a WAN link is because
there's a VPLS between them (think of it like a layer 2 tunnel).  Using
iperf3 I can see general throughput is very good, around 700 to 900 mbps
region... but we only get about 800 K/sec showing in drbd-overview.
Remember, the replication speed is based more on the latency rather than 
the bandwidth. So look at the ping time over the link, compare that to 
the ping time on a local ethernet connection. Most likely, it will 
be a *lot* slower.

Also I've chosen DRBD 8.9.2 that comes with stock Debian 8.  I'm willing
to change that but only if someone points out some massive performance
hampering bug with it.  I prefer to stick to what Debian stable comes
with unless there's a reason not to.
I would suggest to use at least the latest version from linbit within 
the 8.4.x release. Debian won't collect all of the fixes that are 
released (because they aren't just security related or major bugs), but 
you will definitely solve many bugs and also gain many performance 
improvements.
Note: I was in the same camp as you a couple of years ago, but since 
then I've recognised that a very small number of packages can/should be 
using latest release from upstream. DRBD is one of those for me.

Also also... you'll notice there's no security on this (e.g. password)
as it's an entirely private link... I've only put the crc32c in there as
I'm under the impression you're supposed to have some minimal way (at
least) of checking the data was sent correctly?
You might want to check CPU consumption at both ends, especially on each 
individual CPU. crc32c is unlikely to stress any decent CPU, but it's 
worth checking.
Also, given you are using VM's, you might want to check the various 
other factors like network performance (offload, zero copy buffers, 
etc), CPU and disk overheads. In a VM, there can be a lot of overhead 
that you can easily forget about, or ignore. You could also try booting 
on raw metal (if possible) at least to get a comparison.

I've kept the stock config files the way they come with Debian 8, which
is to say there is nothing in drbd.conf and global_common.conf (as
everything is commented out by default - except for 'usage-count no')

Just added in my own as drbd.d/sanvirtualdisk.res ...

resource archive1 {
 protocol A;
 device /dev/drbd1;
 meta-disk internal;
 startup {
 wfc-timeout 0;
 degr-wfc-timeout 60;
 become-primary-on site-a;
 }
 net {
 verify-alg crc32c;
 }
 on site-a {
 disk /dev/mapper/sanvirtualdisk-part1;
 address 10.0.0.1:7789;
 }
 on site-b {
 disk /dev/mapper/sanvirtualdisk-part1;
 address 10.0.0.2:7789;
 }
}

Any suggestions for performance/integrity improvements to this config
would be greatly appreciated - keep in mind my scenario.
Have you looked at the performance tuning chapter in the documentation? 
That is probably the first place to look.

On last thing as well (something else I've been researching for weeks
and come up with no concrete answer) I'm wondering how I can tell Site A
to tell Site B which parts of the filesystem are marked as empty.. that
way speed up the initial sync.  I keep changing my config and
invalidating Site B to retest sync speeds - but it always seems to want
to sync the whole 30TB.  The virtual disks (on both sides) were
completely zero'd recently - so in theory DRBD should realise there is
large amounts of emptiness it could summarise (assuming it uses some
kind of compression) to help it copy quicker?  I've read TRIM would also
help - but I'm not sure that ability survives when you're using an iSCSI
disk.
AFAIK, you can't. What you can do is disconnect the secondary, make some 
number of changes on the primary, then reconnect. DRBD will only re-sync 
the changed blocks.
However, you should keep in mind that the resync performance is not an 
indication of the 

Re: [DRBD-user] Limit Syncer Speed

[Adam Goryachev]

On 15/12/16 13:48, Jasmin J. wrote:

Hello Lars!

> Maybe this nice post from 2012,
> helps to realize what congestion is?
> Pasted here for your convenience, even though it is in the archives.
THX for answering, it explained it pretty good.

> But even with massive buffers (funnels),
> the sustained mid term/long term average write rate
> obviously cannot exceed the minimum bandwith within the whole system.
This depends how you implement Protocol A. It seems that it slows down 
the

write speed when the "funnel" is full. It seems the main goal is to keep
both in sync, even if this costs write speed.

The original from 2012 poster asked this:
>> I was expecting that if I switched to protocol A, I would be able 
to let
>> the SSD drive write at it's full speed (e.g. 250MB/s) only at the 
price of

>> the secondary potentially falling a little bit behind
I guess he meant here the peer going out of sync.
I know (because of the explanation) it isn't implemented that way. But
wouldn't it be possible to implement Protocol A2 (or D), which simply 
writes
with full speed to the local disk regardless if the peer can follow, 
get out
of sync and let the already implemented sync mechanism resync the peer 
with

the configured sync parameter settings during the heavy write load and
also continue syncing until it is consistent again?

There is already a bitmap of inconsistent blocks implemented. Instead of
trying to update the peer with each write the driver could simply set 
the bit
in the bitmap, if the buffer (funnel) gets full and change back to the 
already
implemented sync mode (which is currently used only after a peer 
connecting again).
I would advise that you refer to the documentation, what you are asking 
for already exists:

https://www.drbd.org/en/doc/users-guide-84/re-drbdconf
Specifically look for |on-congestion|

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Limit Syncer Speed

[Adam Goryachev]

On 8/12/16 22:00, Jasmin J. wrote:

Hi!

I am using DRBD 8.4.

I want to limit the syncer speed to approx. 300MBit/s .

During intial sync after creating the disk this worked as expected. The
parameter controlling this was "c-max-rate".
But during normal operation DRBD seems to sync always with the fastest 
speed

possible.

I tried also protocol versions A and C and didn't see significant 
differences

concerning the used network bandwidth.

This is my currently used setup:

disk {
# in units of 0.1 seconds
# 0,1s * 5 => 0,5s
c-plan-ahead 5;

# in units of 0.1 seconds
# 2 seconds max sync delay
c-delay-target 20;

# in Units of KiB/s
# ca. 300MBit/s
c-max-rate 35M;

# in Units of KiB/s
# 32MBit/s
c-min-rate 4M;

# will overrule c-delay-target, but seems to be better
# concerning linearity of the used network bandwith
c-fill-target 18M;
}

net {
#protocol C;
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512k;
cram-hmac-alg sha1;
shared-secret "X";
verify-alg md5;

protocol A;
on-congestion pull-ahead;
congestion-fill 2G;
congestion-extents 2000;
}

I commented "c-fill-target" -> no change
I also tried this (on both sides):
   drbdadm disk-options --c-plan-ahead=0 --c-max-rate=1M 
 -> no change

I test the used network bandwidth with the tool
   speedometer -s -r eth0 -t eth0
It shows always 940MBit/s on a 1GBit ETH link, which is the whole 
bandwidth

of this link.

@Linbit:
Is it possible, that the driver ignore this settings after initial sync?

Any ideas?

I'm not an expert, but I am pretty sure that all those values are only 
related to the initial sync, or a re-sync (ie, after a period of 
disconnected primary/secondary).


To limit the bandwidth consumed while both nodes are online and in sync, 
you will either need to limit the data being written to the primary, or 
limit (through external means) the bandwidth between the nodes (eg, 
perhaps linux kernel traffic shaping will work).


However, limiting the bandwidth will limit the write performance of your 
storage (depending if this is important to you or not).


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] I shall become SyncTarget, but I am primary!

[Adam Goryachev]

On 06/12/16 09:52, Yannis Milios wrote:
>> Dec  5 10:20:07 localhost kernel: block drbd0: Writing the whole 
bitmap, full sync required after drbd_sync_handshake.

>>Dec  5 10:20:07 localhost kernel: block drbd0: meta connection shut down by 
peer.

It seems that the secondary node needs a full resync of the data. Is 
this a drbd setup inside VMs? If yes, then I guess that drbd doesn't 
like much rolling back to previous snapshots...




I think this document might help:
https://www.drbd.org/en/doc/users-guide-83/s-initial-full-sync

or specifically, this command:

*|drbdadm -- --overwrite-data-of-peer primary /|resource |/|*

Just make sure before the command, one is primary disconnected, and the 
other is secondary disconnected, run the above command on the primary.


It should start a full sync, and when completed, they should both be 
UpToDate


PS, not a DRBD expert, but hope it might help you.

Regards,
Adam

--
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 sync speed

[Adam Goryachev]

On 16/11/16 00:46, Lars Ellenberg wrote:

On Mon, Nov 14, 2016 at 10:03:46AM +, Enrica Ruedin wrote:

I tried "drbdmanage peer-device-options --common --c-max-rate ..." as Matt 
descriped on 06/22/16.

The Problem is that this function only accept numbers within a range of 250 to 
4194304 bytes/second!

Uhm, no.
That is *k*Bytes.
And no, you won't get 720 Gigabyte/s through a 10 GigaBit/s link.


I have 10GBit connection, so I need c-max-rate of about 700Mbyte. I can't set 
this with drbdmanage. Drbdmanage doesn't accept 720M also.
How can I set the correct values with drbdmanage?
Just confirming, so maximum configured sync speed is approx 42Gbps, no 
problem until this:

http://www.extremetech.com/extreme/192929-255tbps-worlds-fastest-network-could-carry-all-the-internet-traffic-single-fiber

Of course, resync speed is also limited by the speed of the backend 
disks  so still probably some time before it is the limiting factor :)


Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] questions about protocol c

[Adam Goryachev]

On 02/11/16 00:44, adream wrote:

hi guys
I'm curious about implementation of drbd's protocol c, is there 
any document describe the detail of the protocol c ?

giving an situation,
I have two node, node A works as primary, and node B works as 
secondary, the drbd works in protocol c.
then I write some date in node A, for example i vi create file named 
t.txt, and write "hello drbd" into the node A.
In the node B, the drbd has just created t.txt and receive "hello" 
from node A, then some terrible thing happened, the the connections 
between node A and node B broken, and node B will never receive the 
rested message.

so in this situation, if i mount the drbd in the node B, what will i see?
a t.txt file with "hello " message, or nothing, or something else?
looking forward to any reply.
thanks



You mean like in the documentation:
https://www.drbd.org/en/doc/users-guide-84/s-replication-protocols
Or in the source code?
Either way, I think this is pretty well documented, but I'll try and 
explain further by example...
In your scenario, if you create a file (touch t.txt) and then write 
"hello drbd" into the file (echo "hello drbd" > t.txt) then you have 
three possible outcomes. Either Node B will not have the file (if the 
crash happened before your touch command returned the success message), 
or it will have an empty file if the crash happened after the touch 
command returned the success message and before the echo command 
returned, or it will contain the content "hello drbd" if the crash was 
after the echo command completed.


So, whatever Node A says has been done, will have been done on Node B. 
In actual fact, this would be the same for Protocol B as well (assuming 
by crash you mean Node A vanished at that instant).


Hope that helps...

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Strange behaviour of DRBD

[Adam Goryachev]

On 05/10/16 01:58, Romain Pelissier wrote:

Hi,
So first, sorry if you foud this question basic but I am very new to 
drbd. Second, maybe it's perfectly normal but in this case, I will 
need some advices from you gyus :)


Here is my situation:

I use 8.4.7-1 on RHEL 6.5 (2.6.32-642.4.2.el6.x86_64) on my 2 nodes. I 
use drbd with this scenario:


I have an lvm partition formatted as xfs.
 This partition contain ma application database (several Gigs).
I want to use drbd to have a copy of the partition on the remote node.

Here is my conf:

cat /etc/drbd.d/backup.res
global { usage-count no; }
common { syncer { rate 1000M; } }
resource rc0 {
protocol C;
startup {
wfc-timeout  15;
degr-wfc-timeout 60;
}
net {
cram-hmac-alg sha1;
shared-secret "secret2secret";
}
on esm65 {
device /dev/drbd0;
disk /dev/vg0/lv_arc;
address 172.31.254.27:7788 <http://172.31.254.27:7788>;
#meta-disk internal;
#flexible-meta-disk /dev/vg0/lv_drdb_md;
meta-disk /dev/vg0/lv_drdb_md;
}
on phpesmbck01 {
device /dev/drbd0;
disk /dev/vg0/lv_arc;
address 172.31.254.19:7788 <http://172.31.254.19:7788>;
#meta-disk internal;
#flexible-meta-disk /dev/vg0/lv_drdb_md;
meta-disk /dev/vg0/lv_drdb_md;
}
}

The first sync works fine. To test it once completed, I stop drbd on 
thr second node, mount /dec/vg0/lv_arc on a folder and I can see the 
folders.

Perferct!
But..
For the fun of it, I have renoved a folder on the secondary node drbd 
partition (drbd was stop at that time). then umount it and start drbd 
process.


I check this

cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by 
mockbuild@Build64R6, 2016-01-12 13:27:11

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-
ns:72054328 nr:0 dw:19012784 dr:59035232 al:1170 bm:0 lo:0 pe:0 
ua:0 ap:0 ep:1 wo:f oos:0


So, everything seems to be up to date.
But If I stop the seocndary node, mount the lvm in a folder, the 
folder that I have deleted earlier is not there but it exists on the 
primary node.


Is it normal in this case that the status is UpToDate/UpToDate? How 
can I be sure that the 2 partition are really in sync?



Never, EVER, modify the underlying data.

If you are going to insist on doing that, then you must know what you 
are doing, and shouldn't be asking about it here.
If you really want to do this, then you might end up with a totally 
broken system (as you have discovered). You *might* be lucky and get a 
working system by

1) discard the "other" data
2) make sure the modified copy is started as primary, and then do a 
verify and resync to properly bring the secondary up to date, and 
remember that it contains invalid data until the resync is completed.
You would have to ignore the status from DRBD, you have lied to DRBD, so 
it will lie to you.


What you *should* do:
1) Switch primary
2) Mount the new primary
3) Make changes as needed
4) Switch primary
5) Mount and check

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Drbd/pacemaker active/passive san failover

[Adam Goryachev]

On 19/09/2016 19:06, Marco Marino wrote:



2016-09-19 10:50 GMT+02:00 Igor Cicimov 
>:


On 19 Sep 2016 5:45 pm, "Marco Marino" > wrote:
>
> Hi, I'm trying to build an active/passive cluster with drbd and
pacemaker for a san. I'm using 2 nodes with one raid controller
(megaraid) on each one. Each node has an ssd disk that works as
cache for read (and write?) realizing the CacheCade proprietary
tecnology.
>
Did you configure the CacheCade? If the write cache was enabled in
write-back mode then suddenly removing the device from under the
controller would have caused serious problems I guess since the
controller expects to write to the ssd cache firts and then flush
to the hdd's. Maybe this explains the read only mode?

Good point. It is exactly as you wrote. How can I mitigate this 
behavior in a clustered (active/passive) enviroment??? As I told in 
the other post, I think the best solution is to poweroff the node 
using local-io-error and switch all resources on the other node 
But please give me some suggestions





> Basically, the structure of the san is:
>
> Physycal disks -> RAID -> Device /dev/sdb in the OS -> Drbd
resource (that use /dev/sdb as backend) (using pacemaker with a
master/slave resource) -> VG (managed with pacemaker) -> Iscsi
target (with pacemaker) -> Iscsi LUNS (one for each logical volume
in the VG, managed with pacemaker)
>
> Few days ago, the ssd disk was wrongly removed from the primary
node of the cluster and this caused a lot of problems: drbd
resource and all logical volumes went in readonly mode with a lot
of I/O errors but the cluster did not switched to the other node.
All filesystem on initiators went to readonly mode. There are 2
problems involved here (I think): 1) Why removing the ssd disk
cause a readonly mode with I/O errors? This means that the ssd is
a single point of failure for a single node san with megaraid
controllers and CacheCade tecnology. and 2) Why drbd not
worked as espected?
What was the state in /proc/drbd ?


I think you will need to examine the logs to find out what happened. It 
would appear (just making a wild guess) that either the cache is 
happening between DRBD and iSCSI instead of between DRBD and RAID. If it 
happened under DRBD then DRBD should see the read/write error, and 
should automatically fail the local storage. It wouldn't necessarily 
failover to the secondary, but it would do all read/write from the 
secondary node. The fact this didn't happen makes it look like the 
failure happened above DRBD.


At least that is my understanding of how it will work in that scenario.

Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9: full-mesh and managed resources

[Adam Goryachev]

On 18/08/2016 17:47, Roberto Resoli wrote:


Hello,

I'm currently running a three nodes cluster with drbd 9.0.3 and
drbdmanage 0.97.

It happens that in my setup i can dedicate two phisical interfaces per
node to the storage network, and possibly create a "full mesh" network
as of Chapter 5.1.4 of DRBD9 manual
(http://drbd.linbit.com/doc/users-guide-90/ch-admin-manual#s-drbdconf-conns).


The goal is to use only dedicated links (no network switch) for the
storage network connections.

I understand that this network topology is currently not supported by
drbdmanage, and I'm asking if it would be possible to configure the
three storage nodes as usual (one ip address per node) and change the
configuration of the network connections afterwards.

In particular I see that currently is not possible to dedicate an IP for
every different link betwen a managed resource and its peer node.

Am I wrong? Any advice/suggestion?

I can't comment on the DRBD related portions, but can't you add both 
interfaces on each machine to a single bridge, and then configure the IP 
address on the bridge. Hence each machine will only have one IP address, 
and the other machines will use their dedicated network to connect to 
it. I would assume the overhead of the bridge inside the kernel would be 
minimal, but possibly not, so it might be a good idea to test it out.


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Basic DRBD link not syncing

[Adam Goryachev]

On 16/08/2016 07:09, Craig Thompson wrote:

Basic DRBD link not syncing

First post to the list.  I've tried using search engines to find this 
info but haven't so far.



I have two servers.  They each run Xen hypervisor to support guest 
virtual machines.  They each have multiple LVMs for the underlying 
disk images of the virtual machines.


I have set up Machine A to be the Primary for Guest1.  Machine B is 
the Secondary for Guest1.



The config I have for each looks like this:


resource guest1 {
net { protocol C; }
device/dev/drbd3;
disk  /dev/vg0/guest3;
meta-disk internal;
syncer { verify-alg "md5" ; }

  on primary.host.domain {
address   192.168.1.10:7789;
  }

  on secondary.host.domain {
address   192.168.1.20:7789;
  }
}


I can use drbdadm and bring the resources up.  I can force a primary. 
This initiates a full sync the first time.



However, if I emulate an outage on Machine B (secondary) for minutes 
or hours and then bring it back online, the syncing doesn't continue 
automatically.



I don't know what I'm missing in configuration(s).  I may be missing 
something in the overall picture.  But what I WANT to have happen is 
that DRBD sees when the link is down, pauses and then picks up syncing 
after an outage on the remote end.



If I run a 'drbdadm verify guest1', then it will initiate a review, 
sync the out of sync blocks and then say it's up to date.



But if I go to the Machine A and create a new file on guest1, then 
shut down guest1 so that no other changes are made, and then I go to 
Machine B and bring guest1 up, the newly created file is NOT there.



So, I'm missing some key component that KEEPS things syncing after the 
initial sync is completed.



Help? Documentation pointers?  FAQs?  How-Tos?


Thanks in advance.


There was a similar question on the list just a few days ago, not sure 
if that was you.
In any case, the best thing to check/verify is that your VM is accessing 
/dev/drbd3 and that nothing (except DRBD itself) is accessing 
/dev/vg0/guest3 (on both primary and secondary).


Any changes made to /dev/vg0/guest3 are unknown to DRBD, and so DRBD 
will not know that they should be synced to the other node. It is the 
same as creating a MD array with /dev/sda1 and /dev/sdb1 as /dev/md1 and 
then writing to /dev/sda1 and complaining there is not consistent data 
on /dev/sdb1.


I hope that helps, if not, please see if you can include a step by step 
process, with all commands you entered, all output generated from each 
step, which will show the problem.


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] no sync within VM with Xen

[Adam Goryachev]

On 14/08/2016 00:12, Clément Février wrote:

Hello,

I run into a problem. I have two node with Ubuntu, LVM2, DRBD 8.4 and
Xen. I set up a Logical Volume with DRBD and it seems to work:
On the primary node (named server),
% cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: 5A4F43804B37BB28FCB1F47
  0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-
 ns:4259836 nr:8 dw:212 dr:4269307 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
wo:f oos:0
and on the secondary node (server2),
% cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
  0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-
 ns:8 nr:4259836 dw:4259844 dr:1645 al:1 bm:0 lo:0 pe:0 ua:0 ap:0
ep:1 wo:f oos:0

So everything looks fine. I made an initial sync between nodes. If I
mount the partition /deb/drbd0 on the primary node and make changes, it
is sync properly (checked by umnounting on server, switching to
secondary, setting primary on server2, then mounting the partition).

This partition is actually the root of a VM. If I launch the VM, make
some changes, stop the VM, and mounting the partition on the other node,
then I can see that the partition is unchanged, ie in the state before
the modifications within the VM.

Is there a specific configuration in Xen or DRBD to make?
What am I missing?


Make sure your VM configuration (xen config file) references the drbd 
resource and not the underlying disk. eg: use /dev/drbd0 or /dev/drbd/0 
and not /dev/sda2 or whatever...


In the meantime, since you have already modified the underlying device, 
you should do a full verify and re-sync on the DRBD device.


Regards,
Adam


___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] raid recovery and drbd

[Adam Goryachev]

On 10/07/16 04:34, Mariusz Mazur wrote:

Raid 5 is dead because on multi-terabyte discs I'm almost guaranteed
some unrecoverable sectors, which results in data loss. (With
sufficiently large disks this is also a risk on a two disk raid1.)
I'd disagree with your ideas of RAID1 being a problem. The chance of a 
URE on the *same sector* on *both* drives in RAID1 *at the same time* is 
infinitesimally small.
Also, assuming you are using RAID1 with only two drives, you can use 3 
mirrors, or more...


However, yes, you do potentially have a problem with one drive failed, 
and trying to resync from the second drive, and having a URE.

Thing is, I shouldn't really care when I'm using drbd for everything.
My best case scenario would be:

1. A drive in a raid fails.
2. I put in a new drive.
3. I tell mdadm to rebuild the array, but not abort on unrecoverable
sectors, just zero them out and log them in a usable manner.

How do you do that?
Lets assume you have a 4 drive RAID5 array, one drive died, and is being 
replaced, and you now have a URE on one of the three active devices.
RAID5 can't recover the data, because it only has 2 out of 4 pieces of 
the data.
Will it mark the sector bad on the disk with the URE and return an 
error, or will it fail the entire array?
Please feel free to test this (perhaps with some test devices and/or in 
a VM).

4. I tell drbd to refetch data from these here sectors as provided by mdadm.
How do you do that? Assuming mdadm has simple "marked" the sector as 
un-readable, you could:
1) Do a write to that equivalent sector in DRBD, and DRBD will then 
attempt to write it, and mdadm can now ask the drive to write the new 
data to the sector, and the drive will either succeed, remap it, or fail 
and mdadm will add it to the badblocks list (assuming you have that 
enabled).
2) Tell DRBD to do a verify, wait for it to notice the data on that 
sector is different, and then do a resync from the "other source" (make 
sure you do the verify in the right direction, as DRBD itself doesn't 
know which source is correct).
3) Just invalidate the entire array on this node, and let DRBD resync 
from the other node (without the failed drive).

5. Array fully back up, no data missing.

Am I correct that neither mdadm nor drbd are anywhere near supporting
this workflow?


My understanding is that DRBD + RAID5 is equivalent to RAID51 which is 
worse than RAID15.
Potentially, you could use those same 4 drives, create 4 DRBD resources, 
then use those 4 resources to create a RAID5 array.
Not entirely sure how well that would work, but I expect it would work 
as RAID15 would, potentially providing a more resilient result.
If one disk failed, then you would simply replace it, and resync from 
the other node, if you encounter a URE, then I'm not sure what DRBD will 
do... probably best to test this scenario. If DRBD will just pass the 
URE up the chain, then you can do a RAID5 verify/sync which will 
re-construct the data from the other drives, re-write the data to DRBD, 
which will re-write to the disks, and generally succeed/remap the sector...


It sounds like an interesting area to run some tests. There is a block 
driver which will insert errors, you might find that useful in 
generating the "URE"...


I'd be quite interested in hearing the results of your tests.

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 sync speed

[Adam Goryachev]

On 21/06/2016 22:10, Tytus Rogalewski wrote:

almost solved.
After reboot, file */var/lib/drbd.d/drbdmanage_global_common.conf i*s 
overwritten and empty.

So settings are not working after rebot.
Anybody know how to make it permanent ?
*
*
I seem to recall a post to the list about this, you need to ask drbd to 
make the change for you, then it will update the file automatically on 
all hosts. Maybe try to check the list archives, or see if there is any 
mention in the docs on providing custom options.


Hope that does help, even though it isn't the answer.

Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 sync speed

[Adam Goryachev]

On 20/06/16 22:47, Emmanuel Florac wrote:

Le Sun, 19 Jun 2016 08:56:18 +0200
Piero Baudino <pbaud...@formatnet.it> écrivait:


Hi all,

I have a cluster with 2 PVE servers with DRBD9.
The replicated volume is about 7 Tb, the disks are 7200 rpm near line
SAS. The communications channel for DRBD9 is done via 4 Gbit
dedicated network cards in bond configuration (round robbin).
I have created a VM with 1 Tb disk for test purpuse in the DRBD
volume. After that, DRBD starts sync...
The sync speed is about 10 Mbyte/sec
The effective network speed misured with IPERF is about 3,6 Gbit/sec

Is there a way to increse sync speed ?
All the docs about this topic are related to 8.x.y DRBD version


See this thread maybe:
http://www.gossamer-threads.com/lists/drbd/users/27675



Stupid question perhaps, but doesn't that thread also relate to 8.4.x 
rather than 9.0.x ?



--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Problems getting Satellite node working (debian testing)

[Adam Goryachev]

On 8/06/2016 18:01, Roland Kammerer wrote:

On Wed, Jun 08, 2016 at 01:14:07PM +1000, Adam Goryachev wrote:

 From my previous thread which started talking about problems compiling the
kernel module, now talking about problems with a satellite node.

On 08/06/16 00:14, Roland Kammerer wrote:

it seems my satellite node can't connect to any other nodes...

They don't connect, they listen for connections from their control node.


This is one of the other nodes:

For all list-* commands, use the "-p" option, it generates
readable/pastable output for mailing lists/bug reports.

To make sure the satellite node starts as satellite, add this to
/etc/drbdmanaged.cfg:
ctrl-volume-access-mode = satellite

Then "drbdmanage restart -q" on the satellite

Did that and got this:
root@xen6:/etc# drbdmanage restart -q
ERROR:dbus.proxies:Introspect error on :1.5:/interface:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Message
recipient disconnected from message bus without replying
Attempting to startup the server through D-Bus activation...
ERROR:dbus.proxies:Introspect error on :1.6:/interface:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Message
recipient disconnected from message bus without replying

D-Bus connection FAILED -- the D-Bus server may have been unable to activate
the drbdmanage service.
Review the syslog for error messages logged by the D-Bus server
or the drbdmanage server.

Hi,

please make sure that there is no leftover drbdmanaged process running.
- drbdmanage shutdown -q
- ps aux | drbdmange # and kill if necessary.
-drbdmanage uninit -q # get rid of the control volume if for whatever
reason it was created
- drbdmanage startup


root@xen6:~# drbdmanage shutdown -q
ERROR:dbus.proxies:Introspect error on :1.9:/interface: 
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: 
Message recipient disconnected from message bus without replying

root@xen6:~# ps axu | grep drbdmanage
root  1309  0.0  0.0  12744  2200 pts/1S+   22:02   0:00 grep 
drbdmanage

root@xen6:~# drbdmanage uninit -q
  Failed to find logical volume "drbdpool/.drbdctrl_0"
  Failed to find logical volume "drbdpool/.drbdctrl_1"
root@xen6:~# drbdmanage startup
Attempting to startup the server through D-Bus activation...
ERROR:dbus.proxies:Introspect error on :1.11:/interface: 
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: 
Message recipient disconnected from message bus without replying


D-Bus connection FAILED -- the D-Bus server may have been unable to activate
the drbdmanage service.
Review the syslog for error messages logged by the D-Bus server
or the drbdmanage server.

Here are the logs from syslog
Jun  8 22:02:58 xen6 drbdmanaged[1330]: INFO   DRBDmanage server, 
version 0.96.1 -- initializing on node 'xen6'
Jun  8 22:02:58 xen6 DRBDmanage:1333: spawning ['drbdadm', 'adjust', 
'.drbdctrl']no resources defined!

Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: .drbdctrl: No such resource
Jun  8 22:02:58 xen6 drbdmanaged[1330]: INFO   DRBDManage starting 
as satellite node
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: Traceback (most recent 
call last):
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/bin/dbus-drbdmanaged-service", line 27, in 

Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: drbdmanage_server.main()
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/dist-packages/drbdmanage_server.py", line 38, in main

Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: server.run()
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/dist-packages/drbdmanage/server.py", line 547, in run

Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: self._proxy.start()
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/dist-packages/drbdmanage/proxy.py", line 216, in start
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: self._tcp_server = 
ThreadedTCPServer((self._host, self._port), ThreadedTCPRequestHandler)
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/SocketServer.py", line 417, in __init__

Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: self.server_bind()
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/SocketServer.py", line 431, in server_bind
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: 
self.socket.bind(self.server_address)
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]:   File 
"/usr/lib/python2.7/socket.py", line 228, in meth
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: return 
getattr(self._sock,name)(*args)
Jun  8 22:02:58 xen6 org.drbd.drbdmanaged[478]: socket.error: [Errno 98] 
Address already in use


root@xen6:/var/log# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address 
State  

[DRBD-user] Problems getting Satellite node working (debian testing)

[Adam Goryachev]

From my previous thread which started talking about problems compiling 
the kernel module, now talking about problems with a satellite node.


On 08/06/16 00:14, Roland Kammerer wrote:

>it seems my satellite node can't connect to any other nodes...

They don't connect, they listen for connections from their control node.


>This is one of the other nodes:

For all list-* commands, use the "-p" option, it generates
readable/pastable output for mailing lists/bug reports.

To make sure the satellite node starts as satellite, add this to
/etc/drbdmanaged.cfg:
ctrl-volume-access-mode = satellite

Then "drbdmanage restart -q" on the satellite

Did that and got this:
root@xen6:/etc# drbdmanage restart -q
ERROR:dbus.proxies:Introspect error on :1.5:/interface: 
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: 
Message recipient disconnected from message bus without replying

Attempting to startup the server through D-Bus activation...
ERROR:dbus.proxies:Introspect error on :1.6:/interface: 
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: 
Message recipient disconnected from message bus without replying


D-Bus connection FAILED -- the D-Bus server may have been unable to activate
the drbdmanage service.
Review the syslog for error messages logged by the D-Bus server
or the drbdmanage server.

Looks like it isn't running on the satellite node properly.

On the control node (you find it with):
drbdmanage list-nodes -s CTRL_Node
execute:
drbdmanage restart -q

Then the control node should connect to the satellite and send it the
necessary information.
I guess this doesn't make any difference until the above problem is 
fixed. I did it anyway, and it just said it was restarted ok.


Thanks,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Problems compiling kernel module

[Adam Goryachev]

On 6/06/2016 22:16, Roland Kammerer wrote:

On Mon, Jun 06, 2016 at 09:24:00PM +1000, Adam Goryachev wrote:

Still looks like the same problem, starting from a completely clean

Hi,

sorry, it also requires a change in the main repo which was not pushed
public yet.

git clone --recursive git://git.linbit.com/drbd-9.0.git
cd drbd-9.0/
make -j



Thanks so much, that now works, and so I reach the next problem, it 
seems my satellite node can't connect to any other nodes...

This is the satellite node:
root@xen6:~# drbdmanage list-nodes
No nodes defined

This is one of the other nodes:
san2:~# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free | 
|  State |

||
| castle  |   5403308 |   4017368 | 
| ok |
| san2|   3777040 |   2432060 | 
| ok |
| san3|   1830932 |   1821384 | 
| ok |
| san5|   2764172 |   2745084 | 
| ok |
| xen6| 0 | 0 | | 
satellite node, no storage |

++


san2:~# drbdadm status
.drbdctrl role:Secondary
  volume:0 disk:UpToDate
  volume:1 disk:UpToDate
  castle role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
  san3 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
  san5 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate

filesrv role:Secondary
  disk:UpToDate
  castle role:Secondary
peer-disk:UpToDate
  xen6 connection:Connecting

So, what is missing/wrong now? I would think xen6 machine needs to know 
it should connect to the other machines, but there is no config or 
storage to tell it to do that. Where/how do I configure this?

san2:~# drbdmanage howto-join xen6
Error: Node "xen6" is a satellite node!
Just make sure drbdmanaged is running on xen6
Operation completed successfully

root@xen6:~# ps axf|grep  drbd
 2976 ?S< 0:00  \_ [drbd-reissue]
 3013 pts/0S+ 0:00  |   \_ grep drbd
 1052 ?Sl 0:04 /usr/bin/python 
/usr/bin/dbus-drbdmanaged-service

 1068 ?S  0:00  \_ drbdsetup events2 all

Looks like drbdmanaged is running
root@xen6:~# dmesg|grep drbd
[ 2714.519639] drbd: initialized. Version: 9.0.2-1 (api:2/proto:86-111)
[ 2714.519642] drbd: GIT-hash: 14d2eead95f99a7de1c0b3e3f335a30b8d486bcb 
build by root@xen6, 2016-06-06 22:20:37

[ 2714.519644] drbd: registered as block device major 147
[89849.902882] drbd: module cleanup done.
[89863.813918] drbd: initialized. Version: 9.0.2-1 (api:2/proto:86-111)
[89863.813921] drbd: GIT-hash: 14d2eead95f99a7de1c0b3e3f335a30b8d486bcb 
build by root@xen6, 2016-06-06 22:20:37

[89863.813923] drbd: registered as block device major 147


Following the advice I usually hand out, "have you rebooted" I 
re-started the xen6 machine and get this:

root@xen6:~# drbdmanage list-nodes
ERROR:dbus.proxies:Introspect error on :1.3:/interface: 
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: 
Message recipient disconnected from message bus without replying


Error: Cannot connect to the drbdmanaged process using DBus
The DBus subsystem returned the following error description:
org.freedesktop.DBus.Error.ServiceUnknown: The name :1.3 was not 
provided by any .service files

root@xen6:~# cat /proc/drbd
version: 9.0.2-1 (api:2/proto:86-111)
GIT-hash: 14d2eead95f99a7de1c0b3e3f335a30b8d486bcb build by root@xen6, 
2016-06-06 22:20:37

Transports (api:14):
root@xen6:~# ps axf| grep drbd
 1083 ?S< 0:00  \_ [drbd-reissue]
 1087 pts/0S+ 0:00  \_ grep drbd
root@xen6:~# systemctl start drbd.service
root@xen6:~# systemctl status drbd.service
● drbd.service - DRBD -- please disable. Unless you are NOT using a 
cluster manager.
   Loaded: loaded (/lib/systemd/system/drbd.service; disabled; vendor 
preset: enabled)

   Active: active (exited) since Tue 2016-06-07 23:43:23 AEST; 15s ago
  Process: 1090 ExecStart=/lib/drbd/drbd start (code=exited, 
status=0/SUCCESS)

 Main PID: 1090 (code=exited, status=0/SUCCESS)

Jun 07 23:43:23 xen6 systemd[1]: Starting DRBD -- please disable. Unless 
you are NOT using a cluster manager
Jun 07 23:43:23 xen6 drbd[1090]: Starting DRBD resources:no resources 
defined!

Jun 07 23:43:23 xen6 drbd[1090]: no resources defined!
Jun 07 23:43:23 xen6 drbd[1090]: WARN: stdin/stdout is not a TTY; using 
/dev/consoleWARN: stdin/stdout is not a TTY; using /dev/consoledrbdadm: 
Unknown command 'sh-b-pri'

Jun 07 23:43

Re: [DRBD-user] Problems compiling kernel module

[Adam Goryachev]

On 3/06/2016 18:30, Roland Kammerer wrote:

On Fri, Jun 03, 2016 at 09:05:23AM +1000, Adam Goryachev wrote:

On 02/06/16 23:44, Roland Kammerer wrote:

On Thu, Jun 02, 2016 at 11:02:34PM +1000, Adam Goryachev wrote:

I'm trying to use debian testing and install drbd9.

Using the packages from
http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu/pool/main/d/,
specifically the xenial packages:
error: redefinition of ‘struct ib_cq_init_attr’
  struct ib_cq_init_attr {
 ^

Known and the compat code for 4.5 is added in git.

If you fankenstein your Debian with a xenial ppa that compiles on linux
4.4 (xenial) because it does not require the compat code for 4.5, then
good luck on linux 4.5 which requires new compat code. This will never
work.

I'm in a difficult position, I'm hoping debian testing will move to stable
soon,

Depends on you definition of "soon", but hm, no.


Yes, well, history should have told me about that, but chatter around 6 
months or more ago made me think it might be soon I thought simply 
using testing was a simpler solution, but perhaps I'm now paying the 
price for taking the easy road...



OK, so as I said, I got the latest code with git and got a similar error,

I'm fairly certain I'm still doing something wrong, but I'm just not so sure
what that might be so any assistance would be greatly appreciated.

git submodule update --remote



Still looks like the same problem, starting from a completely clean 
directory:

root@xen6:/usr/src/drbd9# mkdir git2
root@xen6:/usr/src/drbd9# cd git2
root@xen6:/usr/src/drbd9/git2# git clone git://git.drbd.org/drbd-9.0.git
Cloning into 'drbd-9.0'...
remote: Counting objects: 48836, done.
remote: Compressing objects: 100% (12447/12447), done.
remote: Total 48836 (delta 37610), reused 46626 (delta 35750)
Receiving objects: 100% (48836/48836), 11.70 MiB | 2.04 MiB/s, done.
Resolving deltas: 100% (37610/37610), done.
Checking connectivity... done.
root@xen6:/usr/src/drbd9/git2# git submodule update --remote
fatal: Not a git repository (or any of the parent directories): .git
root@xen6:/usr/src/drbd9/git2# cd drbd-9.0/
root@xen6:/usr/src/drbd9/git2/drbd-9.0# git submodule update --remote
root@xen6:/usr/src/drbd9/git2/drbd-9.0# make
Submodule 'drbd-headers' (git://git.linbit.com/drbd-headers.git) 
registered for path 'drbd-headers'
Submodule 'drbd/drbd-kernel-compat' 
(git://git.linbit.com/drbd-kernel-compat.git) registered for path 
'drbd/drbd-kernel-compat'

Cloning into 'drbd-headers'...
remote: Counting objects: 300, done.
remote: Compressing objects: 100% (186/186), done.
remote: Total 300 (delta 195), reused 176 (delta 114)
Receiving objects: 100% (300/300), 62.83 KiB | 28.00 KiB/s, done.
Resolving deltas: 100% (195/195), done.
Checking connectivity... done.
Submodule path 'drbd-headers': checked out 
'2a06010c1b6d04f5d461a388efc47ba9809f89a3'

Cloning into 'drbd/drbd-kernel-compat'...
remote: Counting objects: 608, done.
remote: Compressing objects: 100% (366/366), done.
remote: Total 608 (delta 314), reused 462 (delta 231)
Receiving objects: 100% (608/608), 89.09 KiB | 29.00 KiB/s, done.
Resolving deltas: 100% (314/314), done.
Checking connectivity... done.
Submodule path 'drbd/drbd-kernel-compat': checked out 
'7018d547c68fb0e2b8cd927799650fcc037eac49'

make[1]: Entering directory '/usr/src/drbd9/git2/drbd-9.0/drbd'

Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/lib/modules/4.5.0-2-amd64/build

make -C /lib/modules/4.5.0-2-amd64/build 
SUBDIRS=/usr/src/drbd9/git2/drbd-9.0/drbd  modules

  COMPAT  have_atomic_dec_if_positive
  COMPAT  have_bdev_discard_alignment
  COMPAT  have_kref_get_unless_zero
  COMPAT  have_idr_for_each
  COMPAT  have_simple_positive
  COMPAT  have_three_paramater_hlist_for_each_entry
  COMPAT  have_bioset_create_front_pad
  COMPAT  have_sock_shutdown
  COMPAT  need_genlmsg_multicast_wrapper
  COMPAT  have_find_next_zero_bit_le
  COMPAT  blkdev_issue_zeroout_discard
  COMPAT  use_blk_queue_max_sectors_anyways
  COMPAT  have_genl_lock
  COMPAT  have_kvfree
  COMPAT  have_prandom_u32
  COMPAT  drbd_release_returns_void
  COMPAT  kmap_atomic_page_only
  COMPAT  have_bio_free
  COMPAT  have_rb_augment_functions
  COMPAT  have_dst_groups
  COMPAT  rdma_create_id_has_net_ns
  COMPAT  have_f_path_dentry
  COMPAT  blkdev_issue_zeroout_blkdev_ifl_wait
  COMPAT  have_ctrl_attr_mcast_groups
  COMPAT  have_vzalloc
  COMPAT  have_blk_check_plugged
  COMPAT  have_blkdev_get_by_path
  COMPAT  have_void_make_request
  COMPAT  have_atomic_in_flight
  COMPAT  have_proc_pde_data
  COMPAT  have_fmode_t
  COMPAT  have_linux_byteorder_swabb_h
  COMPAT  have_blk_queue_merge_bvec
  COMPAT  have_open_bdev_exclusive
  COMPAT  have_struct_bvec_iter
  COMPAT  have_blk_queue_max_hw_sectors
  COMPAT  have_signed_nla_put
  COMPAT  have_bd_unlink_disk_holder
  COMPAT  sock_create_kern_has_five_parameters
  COMPAT  have_blk_queue_max_segments
  COMPAT  have_rcu_dereference_protecte

Re: [DRBD-user] Problems compiling kernel module

[Adam Goryachev]

On 02/06/16 23:44, Roland Kammerer wrote:

On Thu, Jun 02, 2016 at 11:02:34PM +1000, Adam Goryachev wrote:

I'm trying to use debian testing and install drbd9.

Using the packages from
http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu/pool/main/d/,
specifically the xenial packages:
error: redefinition of ‘struct ib_cq_init_attr’
  struct ib_cq_init_attr {
 ^

Known and the compat code for 4.5 is added in git.

If you fankenstein your Debian with a xenial ppa that compiles on linux
4.4 (xenial) because it does not require the compat code for 4.5, then
good luck on linux 4.5 which requires new compat code. This will never
work.
OK, so as I said, I got the latest code with git and got a similar 
error, which explicitly is this:

root@xen6:/usr/src/drbd9/git/drbd-9.0# make
make[1]: Entering directory '/usr/src/drbd9/git/drbd-9.0/drbd'

Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/lib/modules/4.5.0-2-amd64/build

make -C /lib/modules/4.5.0-2-amd64/build 
SUBDIRS=/usr/src/drbd9/git/drbd-9.0/drbd  modules

  CC [M]  /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.o
In file included from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50:0,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1535:8: 
error: redefinition of ‘struct ib_cq_init_attr’

 struct ib_cq_init_attr {
^
In file included from 
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1533:0,

 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/linux-headers-4.5.0-2-common/include/rdma/ib_verbs.h:260:8: 
note: originally defined here

 struct ib_cq_init_attr {
^
In file included from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50:0,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h: In 
function ‘drbd_ib_create_cq’:
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1549:8: 
warning: passing argument 5 of ‘ib_create_cq’ makes pointer from integer 
without a cast [-Wint-conversion]

cq_attr->cqe, cq_attr->comp_vector);
^
In file included from 
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1533:0,

 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/linux-headers-4.5.0-2-common/include/rdma/ib_verbs.h:2557:15: 
note: expected ‘const struct ib_cq_init_attr *’ but argument is of type 
‘unsigned int’

 struct ib_cq *ib_create_cq(struct ib_device *device,
   ^
In file included from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50:0,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1548:9: 
error: too many arguments to function ‘ib_create_cq’

  return ib_create_cq(device, comp_handler, event_handler, cq_context,
 ^
In file included from 
/usr/src/drbd9/git/drbd-9.0/drbd/drbd-kernel-compat/drbd_wrappers.h:1533:0,

 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_int.h:50,
 from /usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.c:10:
/usr/src/linux-headers-4.5.0-2-common/include/rdma/ib_verbs.h:2557:15: 
note: declared here

 struct ib_cq *ib_create_cq(struct ib_device *device,
   ^
/usr/src/linux-headers-4.5.0-2-common/scripts/Makefile.build:263: recipe 
for target '/usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.o' failed

make[5]: *** [/usr/src/drbd9/git/drbd-9.0/drbd/drbd_debugfs.o] Error 1
/usr/src/linux-headers-4.5.0-2-common/Makefile:1408: recipe for target 
'_module_/usr/src/drbd9/git/drbd-9.0/drbd' failed

make[4]: *** [_module_/usr/src/drbd9/git/drbd-9.0/drbd] Error 2
Makefile:146: recipe for target 'sub-make' failed
make[3]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make[2]: *** [all] Error 2
Makefile:103: recipe for target 'kbuild' failed
make[1]: *** [kbuild] Error 2
make[1]: Leaving directory '/usr/src/drbd9/git/drbd-9.0/drbd'
Makefile:103: recipe for target 'module' failed
make: *** [module] Error 2
root@xen6:/usr/src/drbd9/git/drbd-9.0#

I'm in a difficult position, I'm hoping debian testing will move to 
stable soon, debian stable (with xen) crashes on bootup on these 
machines, while debian testing works properly. I actually tried git 
before the ubuntu packages, but thought maybe the packaged version might 
work better.


I'm fairly certain I'm still doing something wrong, but I'm just not so 
sure what that might be so any assistance would be greatly appreciated.


Regards,
Adam


--
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.

[DRBD-user] Problems compiling kernel module

[Adam Goryachev]

I'm trying to use debian testing and install drbd9.

Using the packages from 
http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu/pool/main/d/, 
specifically the xenial packages:

Here is what happens:
Setting up drbd-dkms (9.0.2-1ppa1~xenial1) ...
Loading new drbd-9.0.2-1ppa1~xenial1 DKMS files...
First Installation: checking all kernels...
Building only for 4.5.0-2-amd64
Building initial module for 4.5.0-2-amd64
Error! Bad return status for module build on kernel: 4.5.0-2-amd64 (x86_64)
Consult /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/make.log for more 
information.

Processing triggers for libc-bin (2.22-9) ...
Processing triggers for menu (2.1.47) ...
End...

Looking at why the kernel module compile failed I see this:

# cat /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/make.log
DKMS make.log for drbd-9.0.2-1ppa1~xenial1 for kernel 4.5.0-2-amd64 (x86_64)
Thursday 2 June  18:27:34 AEST 2016
make: Entering directory 
'/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd'


Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/lib/modules/4.5.0-2-amd64/build

make -C /lib/modules/4.5.0-2-amd64/build 
SUBDIRS=/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd modules

  COMPAT  have_atomic_dec_if_positive
  COMPAT  have_bdev_discard_alignment
  COMPAT  have_kref_get_unless_zero
  COMPAT  have_idr_for_each
  COMPAT  have_simple_positive
  COMPAT  have_three_paramater_hlist_for_each_entry
  COMPAT  have_bioset_create_front_pad
  COMPAT  have_sock_shutdown
  COMPAT  need_genlmsg_multicast_wrapper
  COMPAT  have_find_next_zero_bit_le
  COMPAT  blkdev_issue_zeroout_discard
  COMPAT  use_blk_queue_max_sectors_anyways
  COMPAT  have_genl_lock
  COMPAT  have_kvfree
  COMPAT  have_prandom_u32
  COMPAT  drbd_release_returns_void
  COMPAT  kmap_atomic_page_only
  COMPAT  have_bio_free
  COMPAT  have_rb_augment_functions
  COMPAT  have_dst_groups
  COMPAT  rdma_create_id_has_net_ns
  COMPAT  have_f_path_dentry
  COMPAT  blkdev_issue_zeroout_blkdev_ifl_wait
  COMPAT  have_ctrl_attr_mcast_groups
  COMPAT  have_vzalloc
  COMPAT  have_blk_check_plugged
  COMPAT  have_blkdev_get_by_path
  COMPAT  have_void_make_request
  COMPAT  have_atomic_in_flight
  COMPAT  have_proc_pde_data
  COMPAT  have_fmode_t
  COMPAT  have_linux_byteorder_swabb_h
  COMPAT  have_blk_queue_merge_bvec
  COMPAT  have_open_bdev_exclusive
  COMPAT  have_struct_bvec_iter
  COMPAT  have_blk_queue_max_hw_sectors
  COMPAT  have_signed_nla_put
  COMPAT  have_bd_unlink_disk_holder
  COMPAT  sock_create_kern_has_five_parameters
  COMPAT  have_blk_queue_max_segments
  COMPAT  have_rcu_dereference_protected
  COMPAT  have_blk_qc_t_make_request
  COMPAT  have_genlmsg_new
  COMPAT  have_bool_type
  COMPAT  bio_split_has_bio_split_pool_parameter
  COMPAT  have_generic_start_io_acct
  COMPAT  have_IS_ERR_OR_NULL
  COMPAT  queue_limits_has_discard_zeroes_data
  COMPAT  have_WB_congested_enum
  COMPAT  bioset_create_has_three_parameters
  COMPAT  have_is_vmalloc_addr
  COMPAT  have_umh_wait_proc
  COMPAT  have_genlmsg_reply
  COMPAT  have_genlmsg_put_reply
  COMPAT  have_bio_bi_error
  COMPAT  have_blk_set_stacking_limits
  COMPAT  hlist_for_each_entry_has_three_parameters
  COMPAT  have_genlmsg_msg_size
  COMPAT  have_security_netlink_recv
  COMPAT  have_blk_queue_split
  COMPAT  have_proc_create_data
  COMPAT  have_nlmsg_hdr
  COMPAT  have_clear_bit_unlock
  COMPAT  have_netlink_cb_portid
  COMPAT  have_ib_cq_init_attr
  COMPAT  have_bd_claim_by_disk
  COMPAT  have_kref_sub
  COMPAT  have_cn_netlink_skb_parms
  COMPAT  have_idr_alloc
  CHK /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/compat.h
  UPD /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/compat.h
  CC [M] 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_debugfs.o
  CC [M] 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_bitmap.o

  CC [M] /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_proc.o
  CC [M] 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_sender.o
  CC [M] 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_receiver.o

  CC [M] /var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_req.o
In file included from 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_int.h:50:0,
 from 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_bitmap.c:35:
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd-kernel-compat/drbd_wrappers.h:1535:8: 
error: redefinition of ‘struct ib_cq_init_attr’

 struct ib_cq_init_attr {
^
In file included from 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd-kernel-compat/drbd_wrappers.h:1533:0,
 from 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_int.h:50,
 from 
/var/lib/dkms/drbd/9.0.2-1ppa1~xenial1/build/src/drbd/drbd_bitmap.c:35:
/usr/src/linux-headers-4.5.0-2-common/include/rdma/ib_verbs.h:260:8: 
note: originally defined here

 struct ib_cq_init_attr {

Re: [DRBD-user] Problems upgrading drbdmanage

[Adam Goryachev]

On 29/04/2016 23:37, Roland Kammerer wrote:

On Fri, Apr 29, 2016 at 09:36:12PM +1000, Adam Goryachev wrote:


On 29/04/2016 18:40, Roland Kammerer wrote:

On Fri, Apr 29, 2016 at 09:21:17AM +1000, Adam Goryachev wrote:

When upgrading from python-drbdmanage 0.94-1 to 0.95-1 using the packages
available at:
http://oss.linbit.com/drbdmanage/packages/drbdmanage-0.95/debian-jessie/

The following is the output:
dpkg -i python-drbdmanage_0.95-1_all.deb
(Reading database ... 48347 files and directories currently installed.)
Preparing to unpack python-drbdmanage_0.95-1_all.deb ...
Unpacking python-drbdmanage (0.95-1) over (0.94-1) ...
Setting up python-drbdmanage (0.95-1) ...
Job for drbdmanaged.socket failed. See 'systemctl status drbdmanaged.socket'
and 'journalctl -xn' for details.
Processing triggers for dbus (1.8.20-0+deb8u1) ...
Processing triggers for man-db (2.7.0.2-5) ...

systemctl status drbdmanaged.socket
● drbdmanaged.socket - DRBDManage Service
Loaded: loaded (/lib/systemd/system/drbdmanaged.socket; enabled)
Active: inactive (dead) since Fri 2016-04-29 09:15:04 AEST; 46s ago
Listen: [::]:6996 (Stream)

drbdmanage still shows that it is running and working (presumably the old
version). Doing a reboot solves the problem, but I would think the package
upgrade should handle this more smoothly. Any suggestions or ideas?

Is this a satellite node or a control node?

Control node.

Then you don't need socket activation at all. systemd disable/stop it.

Can you confirm which part/command I should be running?
Here is another machine (in the same "farm"):
root@san3:/usr/src/drbd9# systemctl |grep -i drbd
sys-devices-virtual-block-drbd0.device loaded active plugged   
/sys/devices/virtual/block/drbd0
sys-devices-virtual-block-drbd1.device loaded active plugged   
/sys/devices/virtual/block/drbd1

● drbd.service loaded failed failedLSB: Control drbd resources.
drbdmanaged.socket loaded active listening DRBDManage Service

This machine is still running python-drbdmanage 0.94-1 (ie, I haven't 
tried to upgrade it yet).
It seems to suggest the socket (that you say is not needed on a control 
node, I'm guessing it is needed on a satellite node) is working 
normally. However, the service is not working.


Right now:
root@san3:/usr/src/drbd9# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
| |  State |

||
| castle  |   5403308 |   4017368 
| | ok |
| san2.websitemanagers.com.au |   3777040 |   2432060 
| | ok |
| san3|   1830932 |   1830924 
| | ok |
| san5|   2764172 |   2745084 
| | ok |
| xen6| 0 | 0 
| | satellite node, no storage |

++

So, it seems to be working, but there are definitely some problems going 
on which I'll detail in my next post (since I think they are unrelated).



I repeated the same upgrade on a second control node, with identical
results.

How was that enabled at all in the first place? By you or was it picked
up "magically" by systemd?



I didn't do anything more than install the python-drbdmanage package, so
either the package did something to make it work, or else systemd
"magically" picked it up.

Hm, should not happen, but I will check that in a clean environment.
Just to be sure about the test case:
- installed 0.94
- played around with it
- upgraded to 0.95
- *never ever* touched any sytemd foo manually

correct?


Yes, I don't really know anything about systemd, except it is default on 
newer debian, and adds yet another thing I will need to learn about, 
instead of the older and simpler scripts + symlinks


So, I've definitely not manually enabled or otherwise touched the 
systemd configs.


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Problems upgrading drbdmanage

[Adam Goryachev]

On 29/04/2016 18:40, Roland Kammerer wrote:

On Fri, Apr 29, 2016 at 09:21:17AM +1000, Adam Goryachev wrote:

When upgrading from python-drbdmanage 0.94-1 to 0.95-1 using the packages
available at:
http://oss.linbit.com/drbdmanage/packages/drbdmanage-0.95/debian-jessie/

The following is the output:
dpkg -i python-drbdmanage_0.95-1_all.deb
(Reading database ... 48347 files and directories currently installed.)
Preparing to unpack python-drbdmanage_0.95-1_all.deb ...
Unpacking python-drbdmanage (0.95-1) over (0.94-1) ...
Setting up python-drbdmanage (0.95-1) ...
Job for drbdmanaged.socket failed. See 'systemctl status drbdmanaged.socket'
and 'journalctl -xn' for details.
Processing triggers for dbus (1.8.20-0+deb8u1) ...
Processing triggers for man-db (2.7.0.2-5) ...

systemctl status drbdmanaged.socket
● drbdmanaged.socket - DRBDManage Service
Loaded: loaded (/lib/systemd/system/drbdmanaged.socket; enabled)
Active: inactive (dead) since Fri 2016-04-29 09:15:04 AEST; 46s ago
Listen: [::]:6996 (Stream)

drbdmanage still shows that it is running and working (presumably the old
version). Doing a reboot solves the problem, but I would think the package
upgrade should handle this more smoothly. Any suggestions or ideas?

Is this a satellite node or a control node?

Control node.

I repeated the same upgrade on a second control node, with identical 
results.

How was that enabled at all in the first place? By you or was it picked
up "magically" by systemd?


I didn't do anything more than install the python-drbdmanage package, so 
either the package did something to make it work, or else systemd 
"magically" picked it up.


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] Problems upgrading drbdmanage

[Adam Goryachev]

When upgrading from python-drbdmanage 0.94-1 to 0.95-1 using the 
packages available at:

http://oss.linbit.com/drbdmanage/packages/drbdmanage-0.95/debian-jessie/

The following is the output:
dpkg -i python-drbdmanage_0.95-1_all.deb
(Reading database ... 48347 files and directories currently installed.)
Preparing to unpack python-drbdmanage_0.95-1_all.deb ...
Unpacking python-drbdmanage (0.95-1) over (0.94-1) ...
Setting up python-drbdmanage (0.95-1) ...
Job for drbdmanaged.socket failed. See 'systemctl status 
drbdmanaged.socket' and 'journalctl -xn' for details.

Processing triggers for dbus (1.8.20-0+deb8u1) ...
Processing triggers for man-db (2.7.0.2-5) ...

systemctl status drbdmanaged.socket
● drbdmanaged.socket - DRBDManage Service
   Loaded: loaded (/lib/systemd/system/drbdmanaged.socket; enabled)
   Active: inactive (dead) since Fri 2016-04-29 09:15:04 AEST; 46s ago
   Listen: [::]:6996 (Stream)

drbdmanage still shows that it is running and working (presumably the 
old version). Doing a reboot solves the problem, but I would think the 
package upgrade should handle this more smoothly. Any suggestions or ideas?


Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 - drbdmanage wrong free pool size

[Adam Goryachev]

On 29/04/16 05:16, Ml Ml wrote:

Hello List,

i am running proxmox on top of drbd9. My problem is that drbdmanage
claims that i have no free disk space.

Here is the paste: https://paste.fedoraproject.org/360820/70757146/

I wonder what that is. If i add up all the resources i get to 1,8TB of 2,45TB.
Cloud anyone point me to a direction to debug or fix this?


I'm confused. Which command did you run which told you that there is no 
free disk space?

What is the output of this command:
drbdmanage free-space 2

It should be something like this:
The maximum size for a 2x redundant volume is 1002109952 kiB
(Aggregate cluster storage size: 2346769408 kiB)
Operation completed successfully

Based on your numbers from drbdmanage list-nodes

Equally, the output of:
drbdmanage free-space 3
Should be something like this:
The maximum size for a 3x redundant volume is 0 kiB
(Aggregate cluster storage size: 2346769408 kiB)
Operation completed successfully

Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 Satellite nodes

[Adam Goryachev]

On 22/04/2016 18:04, Roland Kammerer wrote:

On Fri, Apr 22, 2016 at 01:40:39AM +1000, Adam Goryachev wrote:

Hi,

Thanks for the quick response. That is really my question though. I have a
DRBD Client, ie, no local storage, receiving the drbdmanage control volume
via tcp/ip, but how do I mount the DRBD volume? eg, on the node which has
the data locally, I can simply mount /dev/drbd101 /mnt, but now I want to do
something similar from the DRBD client machine (not at the same time, not
dual primary, just one node at a time should use the volume).

Also on clients you simply mount it, no difference... On the client you
also have a /dev/drbdXYZ. You mount it, the data comes via the network
from the peer.

This seems to be the problem then. I don't have any /dev/drbd* devices 
to mount on the "Client" node, but I do have them on the control nodes 
(only the ones with the volume located on that specific node:

This is the same on every node:
root@san3:~# drbdmanage list-assignments
++
| Node| Resource | Vol ID 
|  | State |

||
| castle  | filesrv  |  * 
|  |ok |
| castle  | oldNFS   |  * 
|  |ok |
| san2.websitemanagers.com.au | filesrv  |  * 
|  |ok |
| san2.websitemanagers.com.au | oldNFS   |  * 
|  |ok |

++


However, on node san2:
san2:~# ls -l /dev/drbd*
brw-rw 1 root disk 147,   0 Apr 21 21:46 /dev/drbd0
brw-rw 1 root disk 147,   1 Apr 21 21:46 /dev/drbd1
brw-rw 1 root disk 147, 100 Apr 21 21:45 /dev/drbd100
brw-rw 1 root disk 147, 101 Apr 21 21:45 /dev/drbd101

From your post, I should see that also on san3 (controller node, but 
doesn't have storage for this specififc volime:

root@san3:~# ls -l /dev/drbd*
brw-rw 1 root disk 147, 0 Apr 21 21:41 /dev/drbd0
brw-rw 1 root disk 147, 1 Apr 21 21:41 /dev/drbd1

Instead all I get is the drbd internal volumes...

Can you advise why it might not be getting the clietn volumes

Thanks again for your time and any hists you can provide!
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 Satellite nodes

[Adam Goryachev]

On 22/04/2016 00:20, Roland Kammerer wrote:

On Thu, Apr 21, 2016 at 11:54:05PM +1000, Adam Goryachev wrote:

Hi,

I've setup a few DRBD9 machines as a storage cluster with 4 nodes, plus one
"satellite" node.

I was wondering how I can access the content on the various DRBD volumes
from the rest of the cluster?

That has nothing to do with satellite or not.

If you specified a deployment count while creating the volume
(new-volume -d) the volume gets automatically deployed on a set of "best"
nodes. If you did not, you have to assign (assign-resource) to one or
multiple nodes. Then you get a /dev/drbdXYZ on that nodes and use it as
any drbd resource.

My deployment count is currently set to 2. So there are two storage 
nodes with a copy of the data.


I think I was looking for the DRBD "Client" mode, which appeared to be 
the same as a DRBD Satellite node, with no local storage.


*Satellite nodes* have:

 * No direct access to the control volume, they receive a copy of the
   cluster configuration via TCP/IP.
 * Local storage if it is a normal satellite node
 * No local storage if it is a pure client

Otherwise, what exactly is the point of a Satellite node which is a pure 
client (ie, no local storage)? What does it add to the DRBD structure?


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] DRBD9 Satellite nodes

[Adam Goryachev]

Hi,

I've setup a few DRBD9 machines as a storage cluster with 4 nodes, plus 
one "satellite" node.


drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
| |  State |

||
| castle  |   5403308 |   4067868 
| | ok |
| san2.websitemanagers.com.au |   3777040 |   2441600 
| | ok |
| san3|   1830932 |   1830924 
| | ok |
| san5|   2764172 |   2764164 
| | ok |
| xen6| 0 | 0 
| | satellite node, no storage |

++
castle:~# drbdmanage list-volumes
++
| Name| Vol ID |   Size | Minor 
|| State |

||
| filesrv |  0 | 667572 |   100 | |ok |
| oldNFS  |  0 | 667572 |   101 | |ok |
++


I was wondering how I can access the content on the various DRBD volumes 
from the rest of the cluster?


I've looked at the docs at:
http://www.drbd.org/en/doc/users-guide-90/s-dm-add-node#_adding_a_satellite_node
This describes the types of nodes available, and how to add them, which 
I've done as above, but I'm not sure how to proceed from here or what to 
do next.


Any pointers to additional docs would be greatly appreciated, hope I 
haven't missed it.


Regards,
Adam



___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD Manual errors

[Adam Goryachev]

On 21/04/2016 22:53, Lars Ellenberg wrote:

On Thu, Apr 21, 2016 at 10:01:12PM +1000, Adam Goryachev wrote:

Hi,

I've found another issue with (my system or) the DRBD docs:
http://www.drbd.org/en/doc/users-guide-90/s-check-status


You can get more information by passing the |--verbose| and/or
|--statistics| arguments:

# drbdadm status home --verbose --statistics

Except this happens:
castle:~# drbdadm status oldNFS --verbose --statistics

please use:
# drbdsetup status oldNFS --verbose --statistics

Thanks that solved it.

Is there some reason that there are so many tools used to manage the 
same thing?
There is drbdmanage, drbdadm and drbdsetup Is this something that 
might be improved in future versions?

While we are at it, I would appreciate if you can provide any tips
on how to determine why the volume is "outdated"

hard to guess.
may be normal behaviour, may be a bug.


and how I can change it back to a working state?

usually, a resync triggered by disconnect/reconnect should help.
if not, maybe an explicit "primary" or even "primary --force".
I rebooted one of the nodes, which didn't make any difference (before I 
sent the above message, I was upgrading to 9.0.2).
So I did a drbdadm primary but that didn't work, so added the --force 
which fixed it.


Thank you for your assistance.

Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] DRBD Manual errors

[Adam Goryachev]

Hi,

I've found another issue with (my system or) the DRBD docs:
http://www.drbd.org/en/doc/users-guide-90/s-check-status

You can get more information by passing the |--verbose| and/or 
|--statistics| arguments:


# drbdadm status home --verbose --statistics

Except this happens:
castle:~# drbdadm status oldNFS --verbose --statistics
drbdadm: unrecognized option '--statistics'
try 'drbdadm help'

drbdadm help doesn't mention --statistics

In addition:
castle:~# drbdadm status oldNFS --verbose
drbdsetup status oldNFS
oldNFS role:Secondary
  disk:Outdated blocked:lower
  san2.websitemanagers.com.au role:Secondary
peer-disk:Outdated
and compare to this (note, it is identical output)
castle:~# drbdadm status oldNFS
oldNFS role:Secondary
  disk:Outdated blocked:lower
  san2.websitemanagers.com.au role:Secondary
peer-disk:Outdated

So verbose is not really providing any extra information or detail at 
all.


While we are at it, I would appreciate if you can provide any tips on 
how to determine why the volume is "outdated" and how I can change it 
back to a working state?
I'm not entirely concerned about the content if it is not recoverable, 
but in case this happens later with more important data then it would be 
good to know the right procedure to recovery.


Regards,
Adam
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Stability issue Proxmox4/DRBD9

[Adam Goryachev]

Hi,

Could you please include the commands you are using to "clone a VM from 
one ressource to the other or when I try to move the disk of a VM from a 
ressource to the other"


ie, what are you really doing?

Also, have you checked your kernel logs/syslog ? What was the complete 
contents, especially around the time of the crash.


What is your architecture? You say each server has 2x2TB drives, and a 
software RAID1 (assume mdadm) for the system partition. What have you 
done with the "second" partition. Is it also a RAID1 with mdadm, or is 
it just two plain partitions?

What device did you use as the PV?
Where are you creating LV's ?
What device does your VM use to access it's data?

Hopefully after collecting this information you will see the issue, or 
once you post it, someone here can assist.


Finally, I'm not sure, but your email shows it was sent on the 10th of 
April, but just arrived today, perhaps your system date is wrong?


Regards,
Adam


On 10/04/16 01:54, Jean-Laurent Ivars wrote:

Dear list,

Sorry it’s a long message and my english may not be the best :(




--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] Documentation error/bug

[Adam Goryachev]

Hi,

Just noticed this probably error:
http://www.drbd.org/en/doc/users-guide-90/s-dm-rebalance

# drbdmanage unassign  

I'm assuming that should say:

# drbdmanage unassign  

Perhaps someone could confirm this, and update the docs.

Once my resource has finished syncing, then I'll test it (using old-node not 
new-node)

Regards,
Adam

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Problems with DRBD9/drbdmanage

[Adam Goryachev]

Just for fun, I did the following:
root@san3:/etc/drbd.d# drbd-overview
 0:.drbdctrl/0  Connected(2*) Secondary(2*) UpToDa/UpToDa
 1:.drbdctrl/1  Connected(2*) Secondary(2*) UpToDa/UpToDa
root@san3:/etc/drbd.d# drbdmanage add-volume test3 5GB --deploy 2
Operation completed successfully
Operation completed successfully
root@san3:/etc/drbd.d# drbd-overview
  0:.drbdctrl/0  Connected(2*) Secondary(2*) UpToDa/UpToDa
  1:.drbdctrl/1  Connected(2*) Secondary(2*) UpToDa/UpToDa
101:oldNFS/0 Connected(2*) Secondary(2*) Incons/UpToDa
102:test3/0  Connected(2*) Secondary(2*) UpToDa/Incons

ie, add a volume on the "other" node, and it magically found both new 
volumes and started syncing them both.


So, I'm not sure why the issue happened in the first place, or why 
adding a second volume from the second server fixed it. Any advice would 
be appreciated,


Regards,
Adam

On 7/04/2016 17:52, Adam Goryachev wrote:

Hi,

I'm trying to build a new cluster of servers using debian plus DRBD9 
and drbdmanage.


After a number of attempts, I thought I had everything right, and it's 
all been "ok" for a couple of weeks.


Today, I rebooted both machines (new power being installed for the 
UPS), and then I tried to create a new volume of 700GB.


Here is what I did:
san2:~# drbdmanage add-volume oldNFS 700GB --deploy 2
Operation completed successfully
Operation completed successfully
san2:~# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
|  | State |

||
| san2.websitemanagers.com.au |   3777040 |   3109316 
|  |ok |
| san3|   1830932 |   1830924 
|  |ok |

++
san2:~# drbdmanage list-volumes --show Port
++
| Name   | Vol ID |   Size | Minor | Port 
|  | State |

||
| oldNFS |  0 | 667572 |   101 | 7001 
|  | ok |
| test1  |  0 |   9536 |   100 | 7000 
|  | ok |

++
san2:~# lvs
  LV  VG   Attr   LSize   Pool Origin Data% Meta%  
Move Log Cpy%Sync Convert

  .drbdctrl_0 drbdpool -wi-ao 4.00m
  .drbdctrl_1 drbdpool -wi-ao 4.00m
  oldNFS_00   drbdpool -wi-ao 652.07g
san2:~# dpkg -l | grep drbd
ii  drbd-utils 8.9.6-1   amd64RAID 1 over 
TCP/IP for Linux (user utilities)
ii  python-drbdmanage 0.94-1all  DRBD 
distributed resource management utility

san2:~# cat /proc/drbd
version: 9.0.1-1 (api:2/proto:86-111)
GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e build by 
r...@san2.websitemanagers.com.au, 2016-03-01 00:38:53

Transports (api:14): tcp (1.0.0)

*So far, everything looks good, so I thought to check out the other 
node, and see what is happening there*


root@san3:~# lvs
  LVVGAttr   LSize  Pool 
Origin Data%  Meta%  Move Log Cpy%Sync Convert

  .drbdctrl_0   drbdpool  -wi-ao 4.00m
  .drbdctrl_1   drbdpool  -wi-ao 4.00m
  backup_system_20141006_193935 san1small -wi-a- 8.00g
  swap  san1small -wi-ao 3.72g
  systemsan1small -wi-ao 13.97g

*Hmmm, that's strange, we don't have any new LV **here?*

root@san3:~# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
|  | State |

||
| san2.websitemanagers.com.au |   3777040 |   3109316 
|  |ok |
| san3|   1830932 |   1830924 
|  |ok |

++
root@san3:~# drbdmanage list-volumes --show Port
++
| Name   | Vol ID |  

[DRBD-user] Problems with DRBD9/drbdmanage

[Adam Goryachev]

Hi,

I'm trying to build a new cluster of servers using debian plus DRBD9 and 
drbdmanage.


After a number of attempts, I thought I had everything right, and it's 
all been "ok" for a couple of weeks.


Today, I rebooted both machines (new power being installed for the UPS), 
and then I tried to create a new volume of 700GB.


Here is what I did:
san2:~# drbdmanage add-volume oldNFS 700GB --deploy 2
Operation completed successfully
Operation completed successfully
san2:~# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
|  | State |

||
| san2.websitemanagers.com.au |   3777040 |   3109316 
|  |ok |
| san3|   1830932 |   1830924 
|  |ok |

++
san2:~# drbdmanage list-volumes --show Port
++
| Name   | Vol ID |   Size | Minor | Port 
|  | State |

||
| oldNFS |  0 | 667572 |   101 | 7001 
|  |ok |
| test1  |  0 |   9536 |   100 | 7000 
|  |ok |

++
san2:~# lvs
  LV  VG   Attr   LSize   Pool Origin Data% Meta%  Move 
Log Cpy%Sync Convert

  .drbdctrl_0 drbdpool -wi-ao 4.00m
  .drbdctrl_1 drbdpool -wi-ao 4.00m
  oldNFS_00   drbdpool -wi-ao 652.07g
san2:~# dpkg -l | grep drbd
ii  drbd-utils   8.9.6-1 amd64RAID 1 over 
TCP/IP for Linux (user utilities)
ii  python-drbdmanage0.94-1 all  DRBD 
distributed resource management utility

san2:~# cat /proc/drbd
version: 9.0.1-1 (api:2/proto:86-111)
GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e build by 
r...@san2.websitemanagers.com.au, 2016-03-01 00:38:53

Transports (api:14): tcp (1.0.0)

*So far, everything looks good, so I thought to check out the other 
node, and see what is happening there*


root@san3:~# lvs
  LVVGAttr   LSize  Pool Origin 
Data%  Meta%  Move Log Cpy%Sync Convert

  .drbdctrl_0   drbdpool  -wi-ao 4.00m
  .drbdctrl_1   drbdpool  -wi-ao 4.00m
  backup_system_20141006_193935 san1small -wi-a- 8.00g
  swap  san1small -wi-ao 3.72g
  systemsan1small -wi-ao 13.97g

*Hmmm, that's strange, we don't have any new LV **here?*

root@san3:~# drbdmanage list-nodes
++
| Name| Pool Size | Pool Free 
|  | State |

||
| san2.websitemanagers.com.au |   3777040 |   3109316 
|  |ok |
| san3|   1830932 |   1830924 
|  |ok |

++
root@san3:~# drbdmanage list-volumes --show Port
++
| Name   | Vol ID |   Size | Minor | Port 
|  | State |

||
| oldNFS |  0 | 667572 |   101 | 7001 
|  |ok |
| test1  |  0 |   9536 |   100 | 7000 
|  |ok |

++
root@san3:~# dpkg -l | grep drbd
ii  drbd-utils   8.9.6-1 amd64RAID 1 over 
TCP/IP for Linux (user utilities)
ii  python-drbdmanage0.94-1 all  DRBD 
distributed resource management utility

root@san3:~# cat /proc/drbd
version: 9.0.1-1 (api:2/proto:86-111)
GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e build by root@san1, 
2016-03-01 00:38:33

Re: [DRBD-user] DRBD9 drbdadm complains about fencing being in wrong section

[Adam Goryachev]

On 18/02/2016 22:34, Lars Ellenberg wrote:
Of course, eventually we need to provide a more or less convenient way 
to move 8.4 clusters to 9. And we will. But for now, if you have an 
existing, working, production DRBD 8.4 cluster, I see no reason at all 
to move that to DRBD 9, not in this year anyways. 
Aha, (light bulb moment). I think this perhaps clarifies the correct 
(current) state of things.


Please correct my statements below, as I suspect what I think is flawed...
1) A new production install should use DRBD9 because it is the most 
stable, the best performing, and has the really cool drbdmanage "magic"


2) If currently using drbd 8.4 (in production), there is no need or 
point in upgrading to drbd 9 (yet)


3) If we want to use the multi-peer technology in DRBD, then you can 
upgrade to drbd 9 and continue to use manually managed config files


4) If we want to use the multi-peer technology in DRBD, then you can 
upgrade to drbd 9 and manually convert/migrate to drbdmanage


I'm currently in the situation of 3 or 4, my concerns are that drbd9 is 
not yet production ready/stable, and that documentation for doing 3 or 4 
doesn't seem to be complete. You say there is no tool for 4 because it 
would need to handle every situation, and people do weird stuff (I agree 
with this view BTW, even if it would be nice to eventually get a tool 
that handles 70% of cases), however, it should be possible to at least 
document one example (which is a fairly common scenario) and leave the 
application/extension of this example to the admin.


Can you alleviate my concerns in relation to DRBD9 being stable? I'd 
really like to use the multiple peer technology, and from linbit 
reports, there are also some significant performance gains available.


I don't mind dealing with "rough edges" like difficult migration tools, 
or even lack of documentation, as long as things work reliably once 
configured.


Regards,
Adam

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD9 drbdadm complains about fencing being in wrong section

[Adam Goryachev]

On 18/02/2016 20:03, Roland Kammerer wrote:

On Thu, Feb 18, 2016 at 10:37:58AM +1100, Igor Cicimov wrote:

On Wed, Feb 17, 2016 at 8:18 PM, Roland Kammerer 

Re: [DRBD-user] DRBD9 and multiple sync sources

[Adam Goryachev]

On 12/02/16 12:34, Adam Goryachev wrote:

Hi,

Can anyone advise the steps needed to "prepare" my LV which is 
currently on-disk format v08 so that I can upgrade it to v09 with 
additional sync nodes?


ie,
lvextend -L+xx /dev/vg0/drbdspace
How to calculate xx ?


Wow, after lots of searching, I ask the question, and two minutes later, 
I found half the answer (sorry for that)
So, I now see: 
http://drbd.linbit.com/users-guide-9.0/ch-internals.html#s-metadata

Which shows a formula for calculating the metadata as:
C / 2^18 X 8 X N + 72
Where C is the device size in sectors and N is the number of peers.

However, I would guess some of that space is already reserved, for the 
two peers I already have? Also, the rest of the questions still apply, 
on how will DRBD find the old signature at a different offset to normal, 
etc...



drbdadm -v --max-peers= create-md 
If the size of the device has changed, will drbdadm still "find" the 
old v08 signature that it needs to update? I think the DRBD signature 
is at the end of the device, but now the end of the device has changed


Also, I wasn't able to find much (any) information on how to migrate 
to drbdmanage. It almost seems like it would be a completely manual 
process, managing the old style config files manually, creating the 
new drbdmanage based device, then migrate the data from the old to the 
new, etc... Ideally, there would be some automated solution to import 
configs and devices/etc, and have drbdmanage do the 
"conversion/migration" itself.


Finally, just how "stable/reliable" is v9? How do you use it, what 
issues have you seen? Is it faster/slower or more stable/less stable 
etc than v8.4?


Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au



--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] DRBD9 and multiple sync sources

[Adam Goryachev]

Hi,

I'm preparing to upgrade my san to drbd9, but decided to test on VM's 
first (smart idea).


I've proceeded up to this point in the manual:
http://drbd.linbit.com/users-guide-9.0/s-upgrading-drbd.html#s-upgrade-convert

This talks about upgrading the meta-data:


Now you need to convert the on-disk metadata to the new version; this 
is really easy, it’s just running one command and acknowledging two 
questions.


If you want to change the number of nodes, you should already have 
increased the size of the lower level device, so that there is enough 
space to store the additional bitmaps; in that case, you’d run the 
command below with an additional argument |--max-peers=//|. When 
determining the number of (possible) peers please take setups like the 
Section 2.21, “DRBD client” 
<http://drbd.linbit.com/users-guide-9.0/s-drbd-client.html> into account.


However, when I think, yes, that is a good idea, lets plan for the 
future, and allow more peers, because I might want a 3rd copy of my 
data, and/or I might use the DRBD "client". So I look in Section 2.21 
which shows me how to prepare the config file, but I still need to 
address the above problem "enough space to store the additional bitmaps".


Finally I find section 18.4 which references:


In the quick-sync bitmap, one bit represents a 4-KiB chunk of on-disk 
data. If the bit is cleared, it means that the corresponding block is 
still in sync with the peer node. That implies that the block has not 
been written to since the time of disconnection. Conversely, if the 
bit is set, it means that the block has been modified and needs to be 
re-synchronized whenever the connection becomes available again.


So size of device divided by 4k divided by 8 should equal the size of 
the bitmap in bytes, but I would assume there is some sort of header or 
similar on this?



So, I'm clearly missing something in the documentation, and my google 
fu, plus RTFM isn't working.


Can anyone advise the steps needed to "prepare" my LV which is currently 
on-disk format v08 so that I can upgrade it to v09 with additional sync 
nodes?


ie,
lvextend -L+xx /dev/vg0/drbdspace
How to calculate xx ?

drbdadm -v --max-peers= create-md 

If the size of the device has changed, will drbdadm still "find" the old 
v08 signature that it needs to update? I think the DRBD signature is at 
the end of the device, but now the end of the device has changed


Also, I wasn't able to find much (any) information on how to migrate to 
drbdmanage. It almost seems like it would be a completely manual 
process, managing the old style config files manually, creating the new 
drbdmanage based device, then migrate the data from the old to the new, 
etc... Ideally, there would be some automated solution to import configs 
and devices/etc, and have drbdmanage do the "conversion/migration" itself.


Finally, just how "stable/reliable" is v9? How do you use it, what 
issues have you seen? Is it faster/slower or more stable/less stable etc 
than v8.4?


Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD with CentOS in Production?

[Adam Goryachev]

On 15/08/13 04:54, Digimer wrote:

On 14/08/13 10:58, Christian Völker wrote:

Hi all,

I'm planning to use DRBD in a production environment. I prefer to use
CentOS as base system.

The reason to use drbd is the synchronisation, not the high 
availability.


We'll have two locations connected through a 100Mbit line. On both
locations users will access the data at the same time. So I know I have
to use a cluster aware filesystem.

I'm a little bit unsure about the performance- of course it will slow
down all access to the device which might be secondary. But are there
any tweaks to improve the performance despite of the slow 100 Mbit
connection?

So questions are:
Is CentOS6 with DRBD suitable for production use?
Which filesystem is recommended? GFS? ZFS (experimental?)?

Thanks Greetings

Christian


First, the short answer; Yes, DRBD on CentOS 6 is perfectly stable. 
I've used 8.3.{11~15} on CentOS 6.{0~4} in production without issue. I 
also use GFS2 partitions on all my clusters without issue.


If you want both locations to have simultaneous access to the storage 
/ filesystem, then you need a cluster aware filesystem and you need to 
run DRBD in dual-primary. This, in turn, requires the use of Protocol 
C which says that DRBD will not tell the caller that the write was 
completed until it has hit persistent storage on both nodes. 
Effectively making your storage performance that of the speed/latency 
of your network link. Across a 100 Mbit link, this means that your raw 
write speeds will never exceed ~11~12 MB/sec. The write latency will 
also be the same as the network link's latency + the storage latency.


Performance will not be stellar.

What you're proposing is called a stretch cluster and it's 
notoriously hard to do well.


There is a further complication to your plan though; It will be nearly 
impossible to differentiate a broken link from a failed remote server. 
So your network link becomes a single point of failure... If the link 
breaks, both nodes will block and call a fence against their peer. The 
fence will fail because the link to the fence device is lost, so the 
nodes will remain blocked and your storage will hang (better to hang 
than to risk corruption). The fence actions will remain pending for 
however long it takes to repair the link, and then both will try to 
fence the other at the exact same time. There is a chance that, post 
network repair, both nodes will get fenced and you will have to 
manually boot the nodes back up.


There is yet another concern; Corosync expects low latency networks 
(corosync being the communication and membership layer of the 
cluster). So you will need to allocate time to tweaking the corosync 
timeouts to handle your high-latency network. If there is an 
intermittent blip in your network that exceeds corosync's timeouts, 
the cluster will partition and one or both of the nodes will be 
fenced, as per the issue above.


You said that HA is not your highest concern, so this might be an 
acceptable risk... You have to make that call. The software is stable 
though. Your implementation may not be stable, however.


digimer



Just to jump in on this

If one of those nodes was for a small mini-office, could you configure 
the small office node to always shutdown/disable access/offline, and the 
big office to become a single primary upon link failure? Then on 
recovery, the small office node reconnects, resyncs any updates, and 
then returns to dual primary?


I don't know if this is applicable for Christian, but it is something 
I've considered previously.



Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] r0 ok, r1 PingAck did not arrive in time

[Adam Goryachev]

On 27/06/13 20:42, Chris Tothill wrote:
 Are you using active/active channel bonding? If so I'd suggest trying
 active/passive (or just a single direct connection with no bonding)
 and see if that helps. If in doubt, simplify...

 Incidentally, with gigE my experience is that you don't actually need
 a 'crossover' cable to do direct connection, a normal 'straight' cable
 works just as well. I think this is because gigE uses all 4 pairs
 simultaneously in both directions rather than one pair in each
 direction as 100M does.


AFAIK, the specs for gigabit specify that all devices must support auto
MDI/MDIx (in otherwords, automatically detect whether crossover is
required or not). I also don't believe that gigabit uses all four pairs,
it still only uses 2 pairs, and even 10M full duplex will use all four
wires (2 pairs) simultaneously...

My suggestion would also to avoid any bonding/etc until you have things
working properly.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
Ph: +61 2 8304 a...@websitemanagers.com.au
Fax: +61 2 8304 0001www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Replication problems constants with DRBD 8.3.10

[Adam Goryachev]

On 16/06/13 18:47, Luca Fornasari wrote:
 On Sun, Jun 16, 2013 at 6:44 AM, cesar br...@click.com.py
 mailto:br...@click.com.py wrote:
  

 And about DRBD over a direct connection in mode round robin, can
 you give me
 links or comments about this case? (This is very important for me
 because I
 will lose connection speed if I change of balance-rr to
 active-backup).


 https://www.kernel.org/doc/Documentation/networking/bonding.txt
 balance-rr: This mode is the only mode that will permit a single
   TCP/IP connection to stripe traffic across multiple
   interfaces. It is therefore the only mode that will allow a
   single TCP/IP stream to utilize more than one interface's
   worth of throughput.  This comes at a cost, however: the
   striping generally results in peer systems receiving packets out
   of order, causing TCP/IP's congestion control system to kick
   in, often by retransmitting segments.
 The problem is you have out of order packets and it doesn't help if
 you start to play around with net.ipv4.tcp_reordering sysctl parameter 
 because there will always be a chance to have out of order packets.
 Ordered packets are indeed fundamental to DRBD.
 In DRBD world the bonding driver is used to achieve HA using
 active/backup or 802.3ad. Neither of which will boost your performance
 (802.3ad can improve performance if and only if you have a great
 number of TCP connections but that's not the case with your DRBD
 scenario).

Hi,

It was my understanding that this was only the case if switches were
involved, and especially different switches. When you have 2 (or more
ports) on one machine connected via crossover cables to another server
with an equal number of ports (obviously), and you configure both sides
with RR, then all packets should arrive in order.

Since packet 1 is sent on eth0, and packet 2 on eth1 from server 1
Server 2 will receive packet 1 on eth0 and packet 2 on eth1 in the same
order.

If you had switch1 between the two machines on eth0, and switch2 between
the machines on eth1, then it is very possible the switch will introduce
different amounts of delay, therefore causing out of order packets.

At least, this was my understanding after some recent detailed
reading/research into these matters. If this is still wrong, even for a
set of direct cross over connections, please educate both me and the OP.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] drbd+mysql+innodb

[Adam Goryachev]

On 13/06/13 10:37, Dan Barker wrote:
 ProActive Software

 rsync will not be able to synchronize from a “failed” disk, drbd
 already has done so.

  

 Dan in Atlanta

  

 *From:*drbd-user-boun...@lists.linbit.com
 [mailto:drbd-user-boun...@lists.linbit.com] *On Behalf Of *Robinson, Eric
 *Sent:* Wednesday, June 12, 2013 6:20 PM
 *To:* Dirk Bonenkamp - ProActive
 *Cc:* drbd-user@lists.linbit.com
 *Subject:* Re: [DRBD-user] drbd+mysql+innodb

  

 Hi Dirk – Thanks for the feedback. I do need some clarification,
 though. DRBD replicates disk block changes to a standby volume. If the
 primary node suddenly fails, the cluster manager promotes the standby
 node to primary and starts the MySQL service. Logically, this seems
 exactly the same as simply rsyncing the data to the new server and
 starting the MySQL service. Why would it work with DRBD but not with
 rsync? Thanks for your patience while I explore this.

  

 Note: we have over 500 separate MySQL database instances using MyISAM.
 I am totally not stoked about the idea of using 300% more disk space
 and gobs more memory.


I think the real issue with rsync is that it will start copying the
file, but new writes may come in during the copy process, therefore the
destination copy is not a point in time copy of the source. This is the
same issue with using rsync for disk images of VM's that are running
during the rsync. As explained, this is not an issue for DRBD
replication, because at any instant in time, the replica (destination
copy) is an exact copy of the source at some point in time, even if it
is not the latest point in time.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] drbd+mysql+innodb

[Adam Goryachev]

On 13/06/13 14:29, Robinson, Eric wrote:
 I really appreciate everyone's input but everyone misunderstood the
 question I was really asking. It's my fault for mentioning rsync. I
 fully understand the difference between how rsync works and how drbd
 works. We use them both all the time. But the real question is more high
 level. We have been told that even if you stop the mysql service, you
 still cannot backup an innodb database by simply copying the files from
 one server to another. We have been told that the mysql service will not
 start properly on the new server even if you make a 100% exact copy of
 all the files from the old server. If that is true, I don't understand
 how using drbd can work with innodb tables, since it basically just
 makes a copy of the files to another server. Am I asking the question right?

A fraction of common sense should tell you that this is baloney (well,
almost).

Consider, what does mysql need, in order to function properly?
A) The various binary/executable files and scripts
B) Probably some log files/directories (depending on config)
C) Maybe /run or /var/run etc to create some lock file or similar
D) The DB files

Do they claim that shutting down the machine, removing the drive,
installing into a new machine, and turning on the new machine will also
not work?

What is the difference to the above, compared to using DRBD to sync the
entire drive?

what is the difference is you only sync the DB directories but ensure
matching versions/locations of config files and executables?

MySQL doesn't read any magic number of figure and decide it's suddenly
on a different machine and not work (at least, open source, unlicensed
versions, maybe licensed/paid for versions are different). As long as
the files exist, in the expected locations, and are not corrupt, then it
will work.

Maybe you should be asking this question of the person who claims it
will not work, or at least provide the details of the claim (ie, why) or
a reference to the discussion where this was claimed etc...


Just my 2c... don't believe everything you read on the Internet, apply
your own common sense filter.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] umount costs lots of time in drbd 8.4.3

[Adam Goryachev]

On 06/06/13 22:42, Lars Ellenberg wrote:
 On Tue, May 21, 2013 at 11:52:58PM +0800, Mia Lueng wrote:
 I have 16G RAM in this server.   Using a low dirty configuration may lead
 to a pool I/O performance?
 Maybe this blog post, and the links referenced therein help?
 https://blogs.linbit.com/p/548/umount-takes-time/

 http://lists.linbit.com/pipermail/drbd-user/2012-September/019080.html

   Lars
Is there a simple method to be notified of new blog posts? They seem
fairly infrequent, but quite useful.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Dual-primary setup: disconnecting

[Adam Goryachev]

On 20/03/13 05:14, Stephan Budach wrote:
 Am 19.03.13 17:32, schrieb Lars Ellenberg:
 On Tue, Mar 19, 2013 at 05:11:05PM +0100, Stephan Budach wrote:
 Am 18.03.13 20:05, schrieb Stephan Budach:
 Am 18.03.13 10:07, schrieb Lars Ellenberg:
 On Sun, Mar 17, 2013 at 08:19:26AM +0100, Stephan Budach wrote:
 Hi all,

 I am running DRBD 8.4.3 in a dual-pri setup. DRBD interlink is
 through a 10 GbE Intel Cx520 link, which is piggy-backed between the
 two Supermicro boxes. I have configured two SCST iscsi targets from
 two DRBD volumes, which configured as multipathed targets on three
 Don't.

 Simply: Don't.


 Multipath expects to talk via more than one path to *the same
 target*.

 What your setup does it have multipath talk (via one path each) to
 *two different targets*, that don't know of each other,
 and happen to be able to present the same data most of the time.

 As iSCSI is not just about READs and WRITEs, but, especially in a
 clustered setup, about reservations and other management commands,
 this CAN NOT WORK, and that is NOT drbd's fault.

 So just don't.

 Thank you.


 Lars

 Hi Lars,


 okay, point taken, but what about the issue with drbd
 disconnecting when both primary get under high load? Even without
 iSCSI access this shouldn't happen, should it?


 Thanks,
 Stephan

 Btw, LIO does support SCSI3 persistent reservations, so one could
 swap scst with LIO and build multipathed iSCSI targets with it,
 unless I am mistaken and have misread anything.
 You are absolutely mistaken.
 First, SCST does support SCSI-3 persistent reservations as well.

 But the point is not about wether or not the target supports some SCSI
 command subset. But wether or not it is cluster aware.


 The initiators are assuming to talk to *ONE* and the same target,
 whereas in this setup, it would talk to *TWO* independent targets
 (which happen to be able to present the same data most of the time).

 So, staying in the reservation example,
 one initiator would succeed in checking out reservations on the
 pink target, while an other initiator succeeds in checking out the same
 reservation on the purple target.

 As the targets are not cluster aware, all management commands
 (even just LUN resets) can (and thus will!) cause serious harm.

 Ok, so would it then even make sense to deploy, a dual-primary DRBD
 where the SCST target fails over along with a virtual IP for the
 target then? This would cause a little timeout to the initiators, but
 they should then be able to log into their targets again.

 I guess this may be good enough and even, altough this setup lacks the
 inherent load balancing, but if the target servers are connected
 through 10GbE, I guess I could live with that.

Why not configure as dual primary, but configure the initiators to only
talk to a single target, and only change to the other target after
failure/timeout. I haven't done this yet, but it is my plan to implement
it soon.

It would be nice to hear whether this should work as expected Any
comments?

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] drbd pacemaker scst/srp 2 node active/passive question

[Adam Goryachev]

 - Original Message -
 From: Dan Barker dbar...@visioncomm.net
 To: drbd List (drbd-user@lists.linbit.com) drbd-user@lists.linbit.com
 Sent: Friday, March 1, 2013 4:59:40 AM
 Subject: Re: [DRBD-user] drbd pacemaker scst/srp 2 node active/passive 
 question

 That's easy, I've been doing it for years, going back to ESXi 4.1 at least, 
 maybe even to 4.0. I run ESXi 5.1 now.

 Set up both the servers in ESXi, Configuration, Storage adapters. Use static 
 discovery, because you can list the targets whether they exist or not. When 
 the primary goes down, the secondary will come up (if it's available) on ESXi 
 without intervention.

 In my setup, the .46 drbd is secondary, and invisible to ESXi. .47 is primary 
 and visible to ESXi. I run the following targets (you can do this with the 
 GUI, but I get lazy):

 vmkiscsi-tool -S -a 172.30.0.46 iqn.2012-05.com.visioncomm.DrbdR:Storage03 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.46 iqn.2012-06.com.visioncomm.DrbdR:Storage02 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.46 iqn.2012-08.com.visioncomm.DrbdR:Storage01 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.46 iqn.2012-08.com.visioncomm.DrbdR:Storage00 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.47 iqn.2012-05.com.visioncomm.DrbdR:Storage03 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.47 iqn.2012-06.com.visioncomm.DrbdR:Storage02 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.47 iqn.2012-08.com.visioncomm.DrbdR:Storage01 
 vmhba39
 vmkiscsi-tool -S -a 172.30.0.47 iqn.2012-08.com.visioncomm.DrbdR:Storage00 
 vmhba39

 If both are primary, I see 4 targets, 8 paths. This neverg happens. 
 Usually, I see 4 targets, 4 paths.

 I always do the switchover manually, so you might see slightly different 
 results. My steps are:

  Go primary on the .46 server.

  Start the target (iscsi-target) software on the .46 server.

  Rescan on all ESXi.

  Stop the target software on the .47 server (ESXi fails over to the other 
 path seamlessly at this point).

  Stop drbd on .47 and do whatever maintenance was necessary.

 To reverse:

  The same steps, but you can skip the scan if the ESXi have seen both 
 targets since boot.  One shows up as active and the other shows up as dead, 
 but the VMs don't care.
Question: Given the above, at some point, you have dual primary, and
iscsi-target on both nodes for a short period of time. Is there actually
a problem to run like this all the time? Regardless of which DRBD node
is written, DRBD should ensure it is copied to the other node. Also,
reads should not be relevant since it doesn't matter which DRBD node the
data comes from.

However, I'm not so confident to actually try this, especially if it
will break in some subtle and horrible way by corrupting the data slowly
over a period of 6 months etc...

Thanks,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] external metadata on ssd vs bcache

[Adam Goryachev]

Lionel Sausin l...@numerigraphe.com wrote:

I wouldn't expect anything like the gains of
bcache/flashcache/enhencio.
Normally internal metadata are just as fast, thanks to the write cache 
of your disks and RAID adapter. Those are much faster than SSDs and 
metadata are small enough.
However you may benefit from external metadata when your those caches 
are saturated by writes (high throughput for a long time).
If you do have an SSD and expect big writes, give it a try and please 
tell us if it really makes a difference.

Lionel Sausin.

Le 26/02/2013 23:30, James Harper a écrit :
 Smallish SSD's are dirt cheap these days and I have a spare slot in
my servers so I was thinking of putting one in and running bcache
(layering would look like drbd-raid0-bcache-sd[ab]). It's a bit of
mucking around to set up as bcache isn't in the kernel at this time and
is targeted against a kernel version that isn't what Debian uses, so it
seems that maybe for now just putting the metadata on the SSD would be
a much simpler option.

 I'm currently using internal metadata, does anyone have any comments
on which would be the best way to go?

 Also, is there a procedure for moving metadata from internal to
external?

Just a word of warning for anyone who is considering to use SSD's, I was 
recently bitten by a severe performance degradation by Linux 2.6.26 
(specifically the version in debian stable), where SSD performance was reduced 
to about half that achievable by a single hard drive (yes seriously). There is 
a simple patch to resolve this, or in my case, I upgraded to 3.2 from backports.

Hope this helps someone else quicker than it took me to work it out (using 5 x 
480G SSD's in RAID5 and wondering why performance sucked so badly)



Regards,
Adam

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD Unconfigured state after service switch

[Adam Goryachev]

On 26/02/13 20:59, Simone Del Pinto wrote:
 Hi Lars,

 yes now is more clar for me.

 We made node standby to simulate a package switching ( in a wrong
 way it seems ).
 Let me ask one thing: do ypu know a way to do a cluster functionalty
 test?

 I'm thinking about unplugging lan cable... is that a right test?

 Thanks again for your support.

I'm pretty sure it was netflix recently released a tool which will
randomly kill amazon EC2 instances to simulate failure situations.
Basically, you want ANY, RANDOM item to fail, that could be a disk, a
LAN cable, a switch, a complete machine.

Walk into the server room, and look around, write down a list of
everything you own, every little thing as well as the big things.

Now work down the list, and decide what might happen if this item
failed, if you think you should be protected, test it (not during prime
time, something always unexpected happens).
If you know you are not protected, decide what you need to do to protect
it, then decide if that is worth it (chance of failure, ost of downtime,
cost of repair, time to repair, etc)

Hope this helps in some way

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] Performance - upgrade hardware suggestions?

[Adam Goryachev]

Hi all,

I've been having significant performance issues with my DRBD storage,
and I'm trying to work out what I need to do to fix it. I'm assuming the
purchase of new hardware will be part of the solution, but I don't want
to do that if it isn't going to help.

I'm running DRBD with the secondary disconnected from 7am to 7pm, so
these performance figures are not affected by the network connection to
the secondary, nor the performance of the secondary server itself. The
primary is a 5 x 480G Intel SSD in RAID5 using Linux md (no battery
backed storage).

I have collected data (every 10 secs) into an RRD from
/sys/block/device/stat which includes the reads/writes, and the
backlog and activetime.

In particular, during times of high read/write, I see spikes in the
equivalent backlog and activetime values.

1) Does a high value for backlog imply that a lower layer is slowing
things down?
ie, if my drbd2 device shows a backlog value of 2100 (my max peak in the
past two hours), then that means the underlying md device is too slow?
At the same time drbd2 had a backlog value of 2100, the underlying
drives sd[b-f] had high values of around 70 to 100.

Graphs can be seen at:
http://203.98.89.64/graphs/
You can see the drbd2 and each of the SSD from the RAID5 array for
backlog/activetime

2) Does it follow that if I improve the performance of the underlying
RAID, then DRBD will improve, reducing the backlog, and making the users
happy?

/proc/mdstat:
md1 : active raid5 sdf1[0] sdc1[4] sdb1[5] sdd1[3] sde1[1]
  1863535104 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5]
[U]
  bitmap: 4/4 pages [16KB], 65536KB chunk

There is only a single partition on each drive, and I have left a small
part of each drive empty (which is supposed to help performance because
of the lack of TRIM).

Each SSD has it's scheduler set to deadline (which did seem to help a
little)

DRBD config:
resource storage2 {
protocol A;
device /dev/drbd2 minor 2;
meta-disk internal;
on san1 {
address 172.17.56.1:7802;
disk /dev/md1;
}
on san2 {
address 172.17.56.2:7802;
disk /dev/md0;
}

net {
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
max-buffers 16000;
max-epoch-size 16000;
unplug-watermark 8192;
sndbuf-size 512k;
on-congestion pull-ahead;
}
startup {
wfc-timeout 10;
degr-wfc-timeout 20;
}
syncer {
verify-alg crc32c;
rate 20M;
al-extents 3389;
}
}

Any other comments on this config which might impact on performance?

Basically, I just want to rule out DRBD as being the performance
limiting factor, and focus on underlying hardware, but no point throwing
hardware at it if the problem is actually somewhere else.

Finally, suggestions on specific hardware which could be added to
improve the performance (or software, or config changes) ?

Thanks,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Expanding a cluster

[Adam Goryachev]

On 01/02/13 02:58, Marcelo Pereira wrote:
 Hello Everyone,

 I'm about to perform an upgrade on my servers and I was wondering how
 to do that.

 Here is the scenario:

 Server A has 16x 1Tb hard drives, under RAID-6.
 Server B has 16x 1Tb hard drives, under RAID-6.

 And both are in sync, using DRBD.

 I though about replacing the hard drives for 2Tb units, one by one.

 So, on each run, I would:

   * Remove a 1Tb disk
   * Add a 2Tb disk
   * Wait for it to rebuild the RAID

 After replacing ALL disks, I would expand the RAID unit, on each server.

 However, I was wondering how DRBD would like this procedure.

 I know that, before expanding the RAID, the cluster size, and the
 block numbers would remain the same, as I would be wasting the extra
 space on the newly added drives.

 So, after both servers have all the drives replaces, and the RAID is
 properly rebuild. Would that be a problem to expand it? How would DRBD
 handle it?

 I will appreciate any comment or suggestion here.
DRBD will work perfectly...

You probably need to do the following:
1) Pull one drive and replace (you could do one on each server at the
same time, although better/safer to do one server at a time)
2) Wait for rebuild to complete
3) Repeat for all disks on BOTH servers
4) Resize the RAID array on each server
5) Resize DRBD (see the fantastic online manual for your version of DRBD
for the details)
6) Resize the underlying filesystem or whatever

BTW, depending on your kernel version, and/or RAID (I'm assuming linux
software raid), you might like to query the linux-raid list to see if
you can ADD the new drive, tell md that this new drive is replacing
drive X, this way you avoid degrading the RAID array, hence lose less
performance during the rebuild, and have a lower risk of disk failure
and especially URE (Unrecoverable Read Error) during the rebuilds.

Regards,
Adam


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Expanding a cluster

[Adam Goryachev]

On 01/02/13 04:04, Justin Edmands wrote:
 I'm on the fence about the amount of time it will take to degrade and
 rebuild a RAID6 at 16 drives (x2 systems). 
 
 Anyone against the idea of:
 Backup data friday night through saturday morning
 stop drbd and heartbeat on node2
 replace all drives on node2
 build raid 6 and match setup/sizes from node1
 initialize metadata, etc.
 start drbd and heartbeat
 let it sync
 make node2 primary
 repeat steps for node1

In theory, the set of drives you pulled from the secondary are an extra
backup you could put all those drives back in, and make that set the
primary In some ways this might be a better solution, since you are
then simply doing a single large read on the primary, and a large write
on the secondary no raid rebuilds, except for the initial resync on
the secondary (which you might be able to skip since you know you will
write to every sector very soon when drbd does the sync).

1) Stop DRBD on secondary
2) Pull all drives on secondary
3) Add all drives on secondary and build new RAID6 array
4) Enable DRBD on secondary
5) sync from primary to secondary

Danger of read errors on the primary during this sync, but I would guess
this is better than doing 16 rebuild's

Personally, I would try to set the primary read-only during the process
(if an option) so that the spare set of drives is an exact match to
the primary (ie, they don't get outdated).

Depends on how much downtime can be scheduled

Finally, I think you have a fairly high risk with 16 drives in a single
RAID6, you might consider 2 sets of 8 drives in RAID6, and do a linear
concat of the two sets (or raid0). That allows you to lose any 2 out of
8 drives, instead of only 2 out of 16. Also, chances of URE on just one
of the remaining 14 drives after a 2 drive failure is not a good risk I
would want. Though depends on capacity requirements if you can use
another 2 drives to ensure you don't lose the data.

Just my 0.02c worth

At the end of the day, the direct answer to the original question was
RTFM, it really is a very nice manual, and you didn't tell us what
version of DRBD you use. The rest is really off-topic for this list,
maybe discuss on the linux-raid list if you are interested.

Regards,
Adam

 On Thu, Jan 31, 2013 at 11:20 AM, Adam Goryachev
 mailingli...@websitemanagers.com.au
 mailto:mailingli...@websitemanagers.com.au wrote:
 
 On 01/02/13 02:58, Marcelo Pereira wrote:
 Hello Everyone,

 I'm about to perform an upgrade on my servers and I was wondering
 how to do that.

 Here is the scenario:

 Server A has 16x 1Tb hard drives, under RAID-6.
 Server B has 16x 1Tb hard drives, under RAID-6.

 And both are in sync, using DRBD.

 I though about replacing the hard drives for 2Tb units, one by one.

 So, on each run, I would:

   * Remove a 1Tb disk
   * Add a 2Tb disk
   * Wait for it to rebuild the RAID

 After replacing ALL disks, I would expand the RAID unit, on each
 server.

 However, I was wondering how DRBD would like this procedure.

 I know that, before expanding the RAID, the cluster size, and
 the block numbers would remain the same, as I would be wasting
 the extra space on the newly added drives.

 So, after both servers have all the drives replaces, and the
 RAID is properly rebuild. Would that be a problem to expand it?
 How would DRBD handle it?

 I will appreciate any comment or suggestion here.
 DRBD will work perfectly...
 
 You probably need to do the following:
 1) Pull one drive and replace (you could do one on each server at
 the same time, although better/safer to do one server at a time)
 2) Wait for rebuild to complete
 3) Repeat for all disks on BOTH servers
 4) Resize the RAID array on each server
 5) Resize DRBD (see the fantastic online manual for your version of
 DRBD for the details)
 6) Resize the underlying filesystem or whatever
 
 BTW, depending on your kernel version, and/or RAID (I'm assuming
 linux software raid), you might like to query the linux-raid list to
 see if you can ADD the new drive, tell md that this new drive is
 replacing drive X, this way you avoid degrading the RAID array,
 hence lose less performance during the rebuild, and have a lower
 risk of disk failure and especially URE (Unrecoverable Read Error)
 during the rebuilds.
 
 Regards,
 Adam
 
 
 -- 
 Adam Goryachev
 Website Managers
 www.websitemanagers.com.au http://www.websitemanagers.com.au
 
 
 ___
 drbd-user mailing list
 drbd-user@lists.linbit.com mailto:drbd-user@lists.linbit.com
 http://lists.linbit.com/mailman/listinfo/drbd-user
 
 


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com

Re: [DRBD-user] Fwd: Re: Secondary Performance

[Adam Goryachev]

On 22/12/12 02:02, Prater, James K. wrote:
 Drbd can only go as fast as the slowest backing store unless you use
 more buffer/page cache.  But since you are using software based RAID,
 I would advise against it.  Your setup dictates write-through.
 Yes, the two different RAID sets appear to quite closely matched but
 you would be better off reversing their roles.

I thought I already responded to this email, but maybe not...

I really don't understand how reversing the roles will help? The
secondary (spinning disks) can't keep up with the writes happening, if I
make it the primary, it will have a lot more load to deal with (reads
plus al updates, etc). This would make the entire system much slower.

I'm just not able to see how making the slower system the primary would
be an improvement?

I was of the understanding that it was somewhat normal for the
secondary to be somewhat less capable than the primary ?

Thanks for your effort in explaining this.

Regards,
Adam


 -Original Message- From: drbd-user-boun...@lists.linbit.com
 [mailto:drbd-user-boun...@lists.linbit.com] On Behalf Of Adam
 Goryachev Sent: Friday, December 21, 2012 8:12 AM To:
 drbd-user@lists.linbit.com Subject: Re: [DRBD-user] Fwd: Re:
 Secondary Performance
 
 On 20/12/12 23:43, Felix Frank wrote:
 Thanks, James, you seem to keep forgetting the list CC though ;)
 
  Original Message  Subject: Re: [DRBD-user]
 Secondary Performance Date: Thu, 20 Dec 2012 11:40:27 + From:
 Prater, James K. jpra...@draper.com To: 'f...@mpexnet.de'
 f...@mpexnet.de
 
 What Felix suggested (switching roles) should work.  Personally,  I
  would not use SSDs for that type of deploy.  It is best placed to
  speed up certain processes (swap, scratch pad database location
 used for indexes or anythi
 
 I think your message was truncated, but I was of the understanding
 that SSD's were perfect for random I/O performance, which essentially
 is what a bunch of VM's do.
 
 With the performance I was getting when the primary was using a 2
 disk RAID1, I either needed to move to a 6 or even 8 disk RAID10, or
 else move to SSD's. The linux-raid list seemed to agree that SSD
 would provide the best performance/etc, the trade-off was of course,
 cost. Also the suggestion against consumer grade SSD's (which I did
 ignore). So far, the SSD solution on the primary has produced the
 desired result and is working well, but spending the same cash on the
 secondary isn't an option at the moment, so I'm doing my best to find
 a low-cost solution.
 
 Currently, with the secondary falling behind about 40 or so times
 per day, it could be considered acceptable, but I'm a little
 concerned about reliability. If that's the best I can get, then so be
 it, but if I can reduce that to less than 5 times per day by
 increasing some buffering/value somewhere, then I'd be a lot
 happier.
 
 Regards, Adam
 
 -- Adam Goryachev Website Managers www.websitemanagers.com.au
 
 ___ drbd-user mailing
 list drbd-user@lists.linbit.com 
 http://lists.linbit.com/mailman/listinfo/drbd-user
 


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Fwd: Re: Secondary Performance

[Adam Goryachev]

On 24/12/12 12:03, Prater, James K. wrote:
 Hello Adam,
 
 You issue is that your secondary peer is running 8Mbyte behind and you could 
 not make it primary without dumping the original primary.   
 

Right, but it's only for a couple of seconds ie, it falls behind for
a maximum period of 20seconds, (usually 3 - 5 seconds) about 20 times
per day. So what I want to know, is how can I tune DRBD to buffer an
additional 8M of data instead of falling behind?

ie, I just need an extra 8M (ok, probably 16M would do it), of write
cache on the secondary between DRBD and the drives.

Sure, ultimately, buying another 5 x SSD's would resolve the issue, but
that just isn't likely to happen any time soon...

On 24/12/12 17:49, Christof.r wrote:
 In our production environment this works out as expected and using the 
 cluster this way around (SSD as primary) delivers around 5-10x the 
 performance as vice versa. We did test this with loading mdir mailboxes with 
 20k+ messages in a webmail interface that fetches ALL the messages from the 
 server and then sorts in in the frontend afterwards.
 The loading time of such boxes, while SSD is primary, are by magnitudes 
 fasteri.e. 3-4 seconds vs 20-30 (of course while system was in production 
 and thousands of users online)

Yep, I want the primary to provide better performance (because of the
reads) than the secondary. This shouldn't be a problem, just my
secondary is a bit too slow to keep up Potentially, upgrading to 6 x
2TB drives in RAID10 would resolve the issue, I can do that with
existing hardware, but then I'm wasting 4TB instead of only 2TB, and I
lose other options with the extra drives (like setting up a 3rd DRBD
server to backup the secondary in a remote location, and just letting it
get out of sync every day, and resync each night)...

Thanks for your advise, I do appreciate it.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Secondary Performance

[Adam Goryachev]

On 20/12/12 19:56, Felix Frank wrote:
 On 12/20/2012 08:31 AM, Adam Goryachev wrote:
 The primary has 5 x 480G Intel SSD drives in a RAID5 configuration
 The secondary has 4 x 2TB WD RE4 drives in a RAID10 configuration
 
 Huh. SSD RAID vs. rotaries? That's about as far apart as you can set
 primary and secondary performance-wise. Have you considered switching
 their roles?

Ummm, no, I need the primary to be fast enough for the work-load...

 If I understand this correctly, you're looking for a way to keep your
 maximum performance and trade in the stability of your replication. Are
 you sure the gain is wort the risk (and pain)? Is your iSCSI maxing out
 the I/O on that SSD RAID?

No, the I/O on the SSD is not significant... at least, I don't think it
is
I thought that since the secondary is a write only mirror, it doesn't
have as much seek overhead because it never has to seek for a read. ie,
if 40% of all disk I/O is a read request, then the secondary has 40%
less load compared to the primary. I hoped that avoiding RAID5 (using
RAID10) on the secondary would allow it to be good enough.

 In general, most of those are small amounts of data over small amounts
 of time, so it would be nice to add a little more buffering 'somewhere'
 so that instead of getting out of sync, the secondary just has more data
 un-written to disk in a buffer somewhere.

 Any suggestions or advice would be greatly appreciated.
 
 DRBD Proxy springs to mind, even though that's not a free option
 (monetarily speaking).

Thanks, I am aware of that option, and have already proposed that as a
solution, but spending that kind of money is not an option at the
moment. I found performance improved after tuning some of the options in
drbd, and was either looking for other parameters to tune (or test/etc)
to find the bottleneck or solve the bottleneck (at least a little).
However, I didn't find a specific formula for setting those options, and
I don't think its a great option to tweak options and just hope for the
best...

I think I've mentioned before the following options that I am currently
using:
max-buffers 16000;
max-epoch-size 16000;
unplug-watermark 8192;
sndbuf-size 512k;
on-congestion pull-ahead;


Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Fwd: Re: Secondary Performance

[Adam Goryachev]

On 20/12/12 23:43, Felix Frank wrote:
 Thanks, James, you seem to keep forgetting the list CC though ;)

  Original Message 
 Subject: Re: [DRBD-user] Secondary Performance
 Date: Thu, 20 Dec 2012 11:40:27 +
 From: Prater, James K. jpra...@draper.com
 To: 'f...@mpexnet.de' f...@mpexnet.de

 What Felix suggested (switching roles) should work.  Personally,  I
 would not use SSDs for that type of deploy.  It is best placed to speed
 up certain processes (swap, scratch pad database location used for
 indexes or anythi

I think your message was truncated, but I was of the understanding that
SSD's were perfect for random I/O performance, which essentially is what
a bunch of VM's do.

With the performance I was getting when the primary was using a 2 disk
RAID1, I either needed to move to a 6 or even 8 disk RAID10, or else
move to SSD's. The linux-raid list seemed to agree that SSD would
provide the best performance/etc, the trade-off was of course, cost.
Also the suggestion against consumer grade SSD's (which I did ignore).
So far, the SSD solution on the primary has produced the desired result
and is working well, but spending the same cash on the secondary isn't
an option at the moment, so I'm doing my best to find a low-cost solution.

Currently, with the secondary falling behind about 40 or so times per
day, it could be considered acceptable, but I'm a little concerned
about reliability. If that's the best I can get, then so be it, but if I
can reduce that to less than 5 times per day by increasing some
buffering/value somewhere, then I'd be a lot happier.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] Secondary Performance

[Adam Goryachev]

Hi,

I've been having a problem with performance of DRBD while the secondary
is connected, and I've tried various things (which have helped) but not
resolved the issue. So last night I upgraded to DRBD 8.3.15 and have
been running with that for the past 6 hours or so. I also enabled the
option:
on-congestion pull-ahead;
and changed my protocol to A (because I couldn't enable that option
otherwise).

Since then, I've received the following log messages:
Dec 20 08:16:57 san1 kernel: [1235131.854167] block drbd2:
Congestion-extents threshold reached
Dec 20 08:16:57 san1 kernel: [1235131.854172] block drbd2: conn(
Connected - Ahead ) pdsk( UpToDate - Consistent )
Dec 20 08:17:00 san1 kernel: [1235134.535128] block drbd2: helper
command: /sbin/drbdadm before-resync-source minor-2
Dec 20 08:17:00 san1 kernel: [1235134.536144] block drbd2: helper
command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)
Dec 20 08:17:00 san1 kernel: [1235134.536148] block drbd2: conn( Ahead
- SyncSource ) pdsk( Consistent - Inconsistent )
Dec 20 08:17:00 san1 kernel: [1235134.536151] block drbd2: Began resync
as SyncSource (will sync 8756 KB [2189 bits set]).
Dec 20 08:17:00 san1 kernel: [1235134.543422] block drbd2: updated sync
UUID 75FEFEE50FF154E9:59B7F817FEF13BD6:8E57A8552E6B2153:8E56A8552E6B2153
Dec 20 08:17:01 san1 kernel: [1235135.764082] block drbd2: Resync done
(total 1 sec; paused 0 sec; 8756 K/sec)
Dec 20 08:17:01 san1 kernel: [1235135.764086] block drbd2: updated UUIDs
75FEFEE50FF154E9::59B7F817FEF13BD6:8E57A8552E6B2153
Dec 20 08:17:01 san1 kernel: [1235135.764091] block drbd2: conn(
SyncSource - Connected ) pdsk( Inconsistent - UpToDate )
Dec 20 08:17:01 san1 kernel: [1235135.828036] block drbd2: bitmap WRITE
of 0 pages took 0 jiffies
Dec 20 08:17:01 san1 kernel: [1235135.828038] block drbd2: 0 KB (0 bits)
marked out-of-sync by on disk bit-map.

How might I read this information to determine what the shortfall in
performance is? It would look like there was some blocks (or portions of
blocks) total 8756KB that the secondary fell behind on. This doesn't
seem like a lot to fall behind, and considering that everything was
caught up again in around 4 seconds.

Is there perhaps some additional buffering or tuning I could do within
DRBD (or on the secondary) to allow an additional 16M worth of buffering
available?

Of course, the above is just one instance of falling behind, and if it
happens frequently enough, or by a large enough value, then the only
solution will be to upgrade the hardware in the machine. However, if it
is only falling behind by small amount then I'd rather just increase the
buffering a bit.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] Secondary Performance

[Adam Goryachev]

On 20/12/12 09:59, Prater, James K. wrote:
 What is/are the actual storage device(s)?

The primary has 5 x 480G Intel SSD drives in a RAID5 configuration
The secondary has 4 x 2TB WD RE4 drives in a RAID10 configuration

RAID is using md raid, then drbd, then LVM, then iSCSI.

Neither the primary nor the secondary is using a BBU RAID controller.
All systems are connected by UPS, and the secondary (plus half the
servers) are on a second UPS (and a second circuit).

I don't wish to discard the option of purchasing additional hardware,
but it is unlikely to be possible in the next 6 months, so would like to
'tune' the software side to avoid that need for now. Of course, I also
don't want to undermine the stability of the system either, but some
risks are acceptable. (as always, risk vs cost analysis)...

Since this morning, I've now summarised the below logs to show the
amount of data that was out of sync (in kB), and the time from becoming
out of sync to back in sync (in seconds).

SizeTime
87564
79402
16642
53402
10656   5
122200  25
21168   7
55404
13216   4
87725
48804
41208   10
47776   12
76805
83605
35100   6
44002
34122
82603
12896   4
86803
14952   3
17824   7
37208   8
45843
13512   4
14416   4
42484
13884   4
84725
10804   4
41003
90204
83763
48843
13440   4
67364
97044
60083
25124
60004
15482
56216   10
68044
48724
10996   6
23602
144216  18
42604

In general, most of those are small amounts of data over small amounts
of time, so it would be nice to add a little more buffering 'somewhere'
so that instead of getting out of sync, the secondary just has more data
un-written to disk in a buffer somewhere.

Any suggestions or advice would be greatly appreciated.

Regards,
Adam

 - Original Message -
 From: Adam Goryachev [mailto:mailingli...@websitemanagers.com.au]
 Sent: Wednesday, December 19, 2012 05:01 PM
 To: drbd-user@lists.linbit.com drbd-user@lists.linbit.com
 Subject: [DRBD-user] Secondary Performance
 
 Hi,
 
 I've been having a problem with performance of DRBD while the secondary
 is connected, and I've tried various things (which have helped) but not
 resolved the issue. So last night I upgraded to DRBD 8.3.15 and have
 been running with that for the past 6 hours or so. I also enabled the
 option:
 on-congestion pull-ahead;
 and changed my protocol to A (because I couldn't enable that option
 otherwise).
 
 Since then, I've received the following log messages:
 Dec 20 08:16:57 san1 kernel: [1235131.854167] block drbd2:
 Congestion-extents threshold reached
 Dec 20 08:16:57 san1 kernel: [1235131.854172] block drbd2: conn(
 Connected - Ahead ) pdsk( UpToDate - Consistent )
 Dec 20 08:17:00 san1 kernel: [1235134.535128] block drbd2: helper
 command: /sbin/drbdadm before-resync-source minor-2
 Dec 20 08:17:00 san1 kernel: [1235134.536144] block drbd2: helper
 command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)
 Dec 20 08:17:00 san1 kernel: [1235134.536148] block drbd2: conn( Ahead
 - SyncSource ) pdsk( Consistent - Inconsistent )
 Dec 20 08:17:00 san1 kernel: [1235134.536151] block drbd2: Began resync
 as SyncSource (will sync 8756 KB [2189 bits set]).
 Dec 20 08:17:00 san1 kernel: [1235134.543422] block drbd2: updated sync
 UUID 75FEFEE50FF154E9:59B7F817FEF13BD6:8E57A8552E6B2153:8E56A8552E6B2153
 Dec 20 08:17:01 san1 kernel: [1235135.764082] block drbd2: Resync done
 (total 1 sec; paused 0 sec; 8756 K/sec)
 Dec 20 08:17:01 san1 kernel: [1235135.764086] block drbd2: updated UUIDs
 75FEFEE50FF154E9::59B7F817FEF13BD6:8E57A8552E6B2153
 Dec 20 08:17:01 san1 kernel: [1235135.764091] block drbd2: conn(
 SyncSource - Connected ) pdsk( Inconsistent - UpToDate )
 Dec 20 08:17:01 san1 kernel: [1235135.828036] block drbd2: bitmap WRITE
 of 0 pages took 0 jiffies
 Dec 20 08:17:01 san1 kernel: [1235135.828038] block drbd2: 0 KB (0 bits)
 marked out-of-sync by on disk bit-map.
 
 How might I read this information to determine what the shortfall in
 performance is? It would look like there was some blocks (or portions of
 blocks) total 8756KB that the secondary fell behind on. This doesn't
 seem like a lot to fall behind, and considering that everything was
 caught up again in around 4 seconds.
 
 Is there perhaps some additional buffering or tuning I could do within
 DRBD (or on the secondary) to allow an additional 16M worth of buffering
 available?
 
 Of course, the above is just one instance of falling behind, and if it
 happens frequently enough, or by a large enough value, then the only
 solution will be to upgrade the hardware in the machine. However, if it
 is only falling behind by small amount then I'd rather just increase the
 buffering a bit.
 
 Regards,
 Adam
 


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
___
drbd-user mailing list
drbd-user@lists.linbit.com
http

Re: [DRBD-user] Still experiencing resource spikes

[Adam Goryachev]

On 19/12/12 01:19, Prater, James K. wrote:
 Hello Andreas,

 Thanks for responding, below is a section of my current sysctl.conf file.   
 As you can
 see that I used the alternative parameters that you had sent me.  The 
 resource spikes
 vary from 18-24 hours apart.   The odd thing is that it sometimes occurs when 
 the system
 is just idling i.e. 5-18Mbytes a second.   


 #
 # Kernel tweaks to close the gap between the buffercache and DRBD DIO to 
 remote store.
 #
 vm.swappiness = 10
 vm.zone_reclaim_mode = 1
 vm.dirty_background_ratio = 1
 vm.dirty_ratio = 1 
 vm.dirty_expire_centisecs = 300
 vm.dirty_writeback_centisecs = 60
 vm.vfs_cache_pressure = 40

 The settings forces the delta between data coming into the file server and 
 data being set to the peer to be
 1-3 seconds as opposed to nearly 25seconds.


Shouldn't these two values be different:

vm.dirty_background_ratio = 1
vm.dirty_ratio = 1 

ie, vm.dirty_ratio should be higher than vm.dirty_background_ratio. If
they are both the same value, it means you never get the advantage of
slowly catching up with your writes. This might be intended if you
expect very consistent high write load... However, if you expect short
spikes of high write load, then the vm.dirty_ratio could be higher,
which would allow the system to catch up over the following low load period.

Making them the same means that as soon as you have 1% dirty, you will
stall all writes, and limit them to the speed of the underlying disk
sub-system, which means you lose any advantage of buffering etc...

Just my interpretation of the various meanings...

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] remote buffer cache

[Adam Goryachev]

On 25/10/12 20:24, Lars Ellenberg wrote:
 On Tue, Oct 23, 2012 at 11:03:14AM +0200, Thilo Uttendorfer wrote:
 Hi,

 the man-page of drbd.conf says about protocol B ...completed, if it has 
 reached local disk and remote buffer cache.

 How can I determine the size of the remote buffer cache? Is this cache also 
 limited by the max-buffers option?
 DRBDs max-buffers settings has something to do with it,
 but that does not change between DRBD protocols.

 There are pictures somewhere in those papers linked from the
 documentation page. iirc.

 Anyways, I'll try some ascii art:


Thank you very much for your explanations, is it possible to get a
pointer to the docs for the configuration options available for each
buffer, and what the buffer is for, what impact it has, etc...

I think some of my performance issues are that the secondary is not
allowed to get too far behind because a buffer is filling up. I'd like
to extend that buffer significantly.

ie, I'd like to use protocol B, and I don't mind how far behind the
secondary gets, (latency between primary and secondary is not an issue,
the secondary just can't write to disk as fast as the primary, so is
getting behind for short periods of time).

That is my specific current question, but in reality, from all the docs
I've read on the drbd website, there doesn't seem to be a page which
documents all of the config options, and explains in some detail what
each option does. (or I could just be blind, dense, or both...)

Thanks,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] LCMC display (and other tools) says up to date but DRBD is not

[Adam Goryachev]

On 26/10/12 00:38, Whit Blauvelt wrote:
 I love ASCII art! Really, any diagram is good. Helps disperse the opacity.

 In the interest of understanding how an average administrator can get
 confused here: DRBD on setup offers to duplicate the data on the primary to
 the secondary. That works. So it would seem (to the naive) like it sees the
 data at that point, and has every opportunity to insert itself under the
 data, per your 3rd diagram. In the case where that data was a normal
 filesystem, that's what DRBD does, right? That's the point of the offer to
 duplicate the existing data from primary to secondary? So with a normal
 filesystem, you can have a filesystem with data, add DRBD, and DRBD slips
 itself under the filesystem. With a VM written raw to a logical volume, DRBD
 can see things well enough to handle the initial duplication, but fails to
 slip itself under the VM.

To me, it is a bit like having a filesystem on /dev/sda1, then telling
mdadm to build a RAID1 array from that as /dev/md0 along with /dev/sdb1.
mdadm is smart enough to copy the bits from /dev/sda1 to /dev/sdb1.
However, if you still do a mount /dev/sda1 /mnt, change lots of bits,
and then umount, then mount /dev/md0 /mnt, you will get a confusing, and
inconsistent mess.

It shouldn't be surprising to learn the unix allows you to shoot
yourself in the foot, or the arm, or even the head, and you can do it in
exceptionally interesting and exciting ways if you choose.

In your case, it sounds like you used LVM (block devices) instead of
/dev/sda1, and that you used DRBD instead of mdadm, and you used kvm
instead of mount... but at the end of the day, you told kvm to slip
under DRBD, which is allowed, but it hurts.

I would suggest that at the least, you will need to
1) Stop one VM (at a time)
2) Start the VM on top of DRBD instead of LVM
3) Instruct DRBD to throw away the secondary
4) do a full sync from the primary to the secondary

Repeat for each VM
Internal or External metadata is irrelevant, as long as the VM's are
reading/writing to DRBD and not LVM.

You might have chosen to do it this way:
sda1 - DRBD - LVM - VM

instead of
sda1 - LVM - DRBD - VM


Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD Versions

[Adam Goryachev]

On 24/10/12 23:52, Dan Barker wrote:
 I don't understand the hubbub over compiling this thing. My first DRBD was
 on Linux from scratch, a distribution where everything is done by hand, so
 there was no package manager availaable. I found the install quite simple.
 My most recent upgrade consisted of these easy steps, and took about 2
 minutes. Selecting the kernel parameters for your environment (step 5) might
 take a little more time, but not counting backups, I'd say 10 minutes ought
 to do it.

It's not that I'm scared of compiling, I've used compile from scratch
linux and freebsd in the past, and starting with slackware when packages
were tar files of the source code (well, that's how I remember it, could
be wrong...).

Using pre-packaged debian stable (or your favourite stable distro) means
that:
a) Someone who (hopefully) knows what they are doing is actually
deciding that this source is stable and should work
b) There are a lot of other people also using this same exact code, so
any bugs are more likely to be found by someone else (and fixed before I
notice)
c) There is already a system in place for noticing a serious bug, and
for alerting the admin (me) and for me to install it (regardless of
whether it is drbd, iscsi, or any otherimportant or trivial component).

However, on the flipside, I keep hearing that there are issues that
are not fixed in 8.3.13, but work fine in 8.4.x, so I am questioning
whether it is worth abandoning the advantages (a, b, c) and building my
own, or not...

The biggest issue that I have is performance (lack of it), and it seems
a newer version (whether 8.3 or 8.4) might improve this. Currently I've
been forced to disable drbd sync during business hours because it
causes freezes a few times a day (for approx 10 secs). So I disconnect
the secondary each morning and re-connect each night. I'd really like to
have 100% of the advantages of DRBD, ie, online and uptodate/uptodate
all the time
 There are some prereqs, but they are probably on your system already: make
 gcc libc6 flex linux-headers-`uname -r` libc6-dev libssl-dev.

 1. cd /usr/src/
 2. wget http://oss.linbit.com/drbd/8.4/drbd-8.4.2.tar.gz
 3. tar -xzf drbd-8.4.2.tar.gz
 4. cd drbd-8.4.2/
 5. ./configure --with-km --prefix /usr --sysconfdir /etc --localstatedir
 /var
 6. make clean all
 7. make install  

 I hope seeing it laid out in all its simplicity encourages you to give it a
 try. Heck, fire up a virtual machine or two and experiment. That's the fun
 part of our jobs anyhow.
Which part is the most relevant, kernel module or user tools?
Or must they match?
Does that re-build both components?
At which point does the current version get taken offline (make install
or some point after that)?

From the docs, it looks like you can upgrade the secondary, then switch
roles, and then upgrade the primary In theory with negligible
downtime...

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] DRBD Versions

[Adam Goryachev]

Hi,

I've been using DRBD on a couple of systems for a while, and have just
used the version that came with my distro (debian stable (squeeze))
since I really don't like to maintain compiling and installing from
source, and managing (remembering) to upgrade, recompile, etc each time
there is a new version.

However, more and more, it would seem that my current 8.3.7 (debian
package 2:8.3.7-2.1) is probably missing a lot of bug fixes, but on
checking, debian testing only has 8.3.13, and even debian unstable has
only 8.3.13.

So the question is, should I just bite the bullet and install DRBD from
source?

I notice from http://www.drbd.org/download/packages/ that DRBD is
integrated into the vanilla kernel 2.6.33 or newer.
If I upgrade my debian stable kernel (2.6.32) to a newer version (either
debian testing or debian-backports) 3.2 based kernel, can I then just
download, compile, and install the latest 8.4.x version of DRBD?

Thank you for your suggestions/comments

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user