Re: o2cb/ocfs2 : cluster is offline after reboot

[Sven Hartge]

Roman Serbski  wrote:
> On Tue, Sep 19, 2017 at 11:25 AM, Roman Serbski  
> wrote:
>> On Mon, Sep 18, 2017 at 6:18 PM, Sven Hartge  wrote:
>>> Roman Serbski  wrote:
>>>
>>> Maybe o2cb and ocfs2 need to be ordered after network-online.target?
>>
>> I thought about it too, but according to /etc/init.d/o2cb and
>> /etc/init.d/ocfs2 they seem to start after $network:

> Nailed it!

> update-rc.d -f o2cb remove
> update-rc.d -f ocfs2 remove
> update-rc.d o2cb defaults
> update-rc.d ocfs2 defaults

> did the trick. Perhaps this is a leftover from Jessie...

Please report this as a bug, the above should not be needed after
dist-upgrading.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

Re: o2cb/ocfs2 : cluster is offline after reboot

[Roman Serbski]

On Tue, Sep 19, 2017 at 11:25 AM, Roman Serbski
 wrote:
> On Mon, Sep 18, 2017 at 6:18 PM, Sven Hartge  wrote:
>> Roman Serbski  wrote:
>>
>> Maybe o2cb and ocfs2 need to be ordered after network-online.target?
>
> I thought about it too, but according to /etc/init.d/o2cb and
> /etc/init.d/ocfs2 they seem to start after $network:

Nailed it!

update-rc.d -f o2cb remove
update-rc.d -f ocfs2 remove
update-rc.d o2cb defaults
update-rc.d ocfs2 defaults

did the trick. Perhaps this is a leftover from Jessie...

Before:

$ ls -al /etc/rc*.d/*o2cb
lrwxrwxrwx 1 root root 14 Sep 10 15:08 /etc/rc0.d/K02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 10 15:08 /etc/rc6.d/K02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 10 15:06 /etc/rcS.d/S13o2cb -> ../init.d/o2cb

$ ls -al /etc/rc*.d/*ocfs2
lrwxrwxrwx 1 root root 15 Sep 10 15:08 /etc/rc0.d/K01ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 10 15:08 /etc/rc6.d/K01ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 10 15:06 /etc/rcS.d/S14ocfs2 -> ../init.d/ocfs2

After:

$ ls -al /etc/rc*.d/*o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc0.d/K02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc1.d/K02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc2.d/S02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc3.d/S02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc4.d/S02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc5.d/S02o2cb -> ../init.d/o2cb
lrwxrwxrwx 1 root root 14 Sep 19 12:53 /etc/rc6.d/K02o2cb -> ../init.d/o2cb

$ ls -al /etc/rc*.d/*ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc0.d/K01ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc1.d/K01ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc2.d/S05ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc3.d/S05ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc4.d/S05ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc5.d/S05ocfs2 -> ../init.d/ocfs2
lrwxrwxrwx 1 root root 15 Sep 19 12:55 /etc/rc6.d/K01ocfs2 -> ../init.d/ocfs2

Thanks for pointing out the right direction!

Re: o2cb/ocfs2 : cluster is offline after reboot

[Roman Serbski]

On Mon, Sep 18, 2017 at 6:18 PM, Sven Hartge  wrote:
> Roman Serbski  wrote:
>
>> /etc/fstab
>
>> ###
>> /dev/drbd0   /var/www ocfs2   noauto,noatime   0   0
>> ###
>
>> After the reboot, no /var/www is mounted.
>
> Missing "_netdev"?

Thank you. Tried that -- no difference.

>> The status of both o2cb and ocfs2 services is inactive (dead) with the
>> ocfs2 cluster offline:
>
>> $ service o2cb status
>> o2cb.service - LSB: Load O2CB cluster services at system boot.
>>Loaded: loaded (/etc/init.d/o2cb; generated; vendor preset: enabled)
>>Active: inactive (dead)
>>  Docs: man:systemd-sysv-generator(8)
>
>> $ service ocfs2 status
>> ocfs2.service - LSB: Mount OCFS2 volumes at boot.
>>Loaded: loaded (/etc/init.d/ocfs2; generated; vendor preset: enabled)
>>Active: inactive (dead)
>>  Docs: man:systemd-sysv-generator(8)
>
> Maybe o2cb and ocfs2 need to be ordered after network-online.target?

I thought about it too, but according to /etc/init.d/o2cb and
/etc/init.d/ocfs2 they seem to start after $network:

#!/bin/bash
# init fragment for O2CB.
#
# chkconfig: 2345 24 19
# description: Load O2CB cluster services at system boot.
#
### BEGIN INIT INFO
# Provides: o2cb
# Required-Start: $network $local_fs
# Required-Stop: $network $local_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Load O2CB cluster services at system boot.
# Description: Load O2CB cluster services at system boot.
### END INIT INFO

#! /bin/bash
# Copyright (c) 2005 Oracle
# All rights reserved.
#
# chkconfig: 2345 25 18
# description: Mount OCFS2 volumes at boot.
#
### BEGIN INIT INFO
# Provides: ocfs2
# Required-Start: $local_fs $network o2cb
# Required-Stop: $local_fs $network o2cb
# Should-Start: drbd iscsi multipath-tools
# Should-Stop: drbd iscsi multipath-tools
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Mount OCFS2 volumes at boot.
# Description:  Mount OCFS2 volumes at boot.
### END INIT INFO

> Any output after a failed boot from "journalctl -b -u  o2cb.service" or
> "journalctl -b -u ocfs2.service"?

Nothing there:

$ journalctl -b -u o2cb.service
-- Logs begin at Tue 2017-09-19 10:13:16 CEST, end at Tue 2017-09-19
11:17:01 CEST. --
-- No entries --

$ journalctl -b -u ocfs2.service
-- Logs begin at Tue 2017-09-19 10:13:16 CEST, end at Tue 2017-09-19
11:17:01 CEST. --
-- No entries --

Thanks.

Re: o2cb/ocfs2 : cluster is offline after reboot

[Sven Hartge]

Roman Serbski  wrote:

> /etc/fstab

> ###
> /dev/drbd0   /var/www ocfs2   noauto,noatime   0   0
> ###

> After the reboot, no /var/www is mounted.

Missing "_netdev"?

> The status of both o2cb and ocfs2 services is inactive (dead) with the
> ocfs2 cluster offline:

> $ service o2cb status
> o2cb.service - LSB: Load O2CB cluster services at system boot.
>Loaded: loaded (/etc/init.d/o2cb; generated; vendor preset: enabled)
>Active: inactive (dead)
>  Docs: man:systemd-sysv-generator(8)

> $ service ocfs2 status
> ocfs2.service - LSB: Mount OCFS2 volumes at boot.
>Loaded: loaded (/etc/init.d/ocfs2; generated; vendor preset: enabled)
>Active: inactive (dead)
>  Docs: man:systemd-sysv-generator(8)

Maybe o2cb and ocfs2 need to be ordered after network-online.target?

Any output after a failed boot from "journalctl -b -u  o2cb.service" or
"journalctl -b -u ocfs2.service"?

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

o2cb/ocfs2 : cluster is offline after reboot

[Roman Serbski]

Hi,

Anyone here using ocfs2 cluster with Stretch? I can't get it to be
online during boot time for some reason, hence ocfs2 partition can't
be mounted. This is an upgrade from Jessie, where everything was
working just fine. My setup consists of two nodes.

$ uname -a
Linux QSRV01 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06)
x86_64 GNU/Linux

$ dpkg -l | grep ocfs
ii  ocfs2-tools1.8.4-4
amd64tools for managing OCFS2 cluster filesystems

Pretty much default /etc/default/o2cb:

###
O2CB_ENABLED=true
O2CB_BOOTCLUSTER=web
O2CB_HEARTBEAT_THRESHOLD=31
O2CB_IDLE_TIMEOUT_MS=3
O2CB_KEEPALIVE_DELAY_MS=2000
O2CB_RECONNECT_DELAY_MS=2000
###

and /etc/ocfs2/cluster.conf (with tabs present):

###
cluster:
heartbeat_mode = local
node_count = 2
name = web

node:
number = 1
cluster = web
ip_port = 
ip_address = 192.168.0.4
name = QSRV01

node:
number = 2
cluster = web
ip_port = 
ip_address = 192.168.0.5
name = QSRV02

heartbeat:
cluster = web
region = 
###

/etc/fstab

###
/dev/drbd0   /var/www ocfs2   noauto,noatime   0   0
###

After the reboot, no /var/www is mounted. I don't see any traces of
o2cb/ocfs2 in dmesg except one entry:

Sep 18 14:24:38 QSRV02 kernel: [  145.104344] ocfs2: Registered
cluster interface o2cb

The status of both o2cb and ocfs2 services is inactive (dead) with the
ocfs2 cluster offline:

$ service o2cb status
o2cb.service - LSB: Load O2CB cluster services at system boot.
   Loaded: loaded (/etc/init.d/o2cb; generated; vendor preset: enabled)
   Active: inactive (dead)
 Docs: man:systemd-sysv-generator(8)

$ service ocfs2 status
ocfs2.service - LSB: Mount OCFS2 volumes at boot.
   Loaded: loaded (/etc/init.d/ocfs2; generated; vendor preset: enabled)
   Active: inactive (dead)
 Docs: man:systemd-sysv-generator(8)

$ o2cb cluster-status web
offline

If I execute 'o2cb register-cluster web', the status changes to
online, and I can 'mount /dev/drbd0' without any issues:

$ o2cb cluster-status
Cluster 'web' is online

As a workaround, I added the following into /etc/rc.local, which did
work, and /var/www is mounted fine after the reboot.

###
o2cb register-cluster web
sleep 30
mount /dev/drbd0
exit 0
###

Any hints would be greatly appreciated, but I have a feeling that
/etc/default/o2cb is somehow not processed during boot time, or
O2CB_ENABLED=true (and/or O2CB_BOOTCLUSTER=web) are ignored
altogether.

Thank you.