Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hmm looks like I restarted everything except MDS... So it's the same issue! That's why the MDS kill themselves during the reboot of one of the monitors with MDS in 12.2.2. Thanks Dan! Adrien Le 28/03/2018 à 16:43, Dan van der Ster a écrit : Do you have the startup banners for

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Do you have the startup banners for mds.cccephadm14 and 15? It sure looks like they were running 12.2.2 with the "not writeable with daemon features" error. -- dan On Wed, Mar 28, 2018 at 3:12 PM, adrien.geor...@cc.in2p3.fr wrote: > Hi, > > All Ceph services were in

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hi, All Ceph services were in 12.2.4 version. Adrien Le 28/03/2018 à 14:47, Dan van der Ster a écrit : Hi, Which versions were those MDS's before and after the restarted standby MDS? Cheers, Dan On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hi, Which versions were those MDS's before and after the restarted standby MDS? Cheers, Dan On Wed, Mar 28, 2018 at 11:11 AM, adrien.geor...@cc.in2p3.fr wrote: > Hi, > > I just had the same issue with our 12.2.4 cluster but not during the > upgrade. > One of our 3

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hi, I just had the same issue with our 12.2.4 cluster but not during the upgrade. One of our 3 monitors restarted (the one with a standby MDS) and the 2 others active MDS killed themselves : 2018-03-28 09:36:24.376888 7f910bc0f700  0 mds.cccephadm14 handle_mds_map mdsmap compatset

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Just run into this problem on our production cluster It would have been nice if the release notes of 12.2.4 had been adapted to inform user about this. Best, Martin On Wed, Mar 14, 2018 at 9:53 PM, Gregory Farnum wrote: > On Wed, Mar 14, 2018 at 12:41 PM, Lars

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree wrote: > On 2018-03-14T06:57:08, Patrick Donnelly wrote: > >> Yes. But the real outcome is not "no MDS [is] active" but "some or all >> metadata I/O will pause" -- and there is no avoiding that. During an >>

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 2018-03-14T06:57:08, Patrick Donnelly wrote: > Yes. But the real outcome is not "no MDS [is] active" but "some or all > metadata I/O will pause" -- and there is no avoiding that. During an > MDS upgrade, a standby must take over the MDS being shutdown (and > upgraded).

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Mar 14, 2018 at 5:48 AM, Lars Marowsky-Bree wrote: > On 2018-02-28T02:38:34, Patrick Donnelly wrote: > >> I think it will be necessary to reduce the actives to 1 (max_mds -> 1; >> deactivate other ranks), shutdown standbys, upgrade the single active,

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 03/14/2018 01:48 PM, Lars Marowsky-Bree wrote: > On 2018-02-28T02:38:34, Patrick Donnelly wrote: > >> I think it will be necessary to reduce the actives to 1 (max_mds -> 1; >> deactivate other ranks), shutdown standbys, upgrade the single active, >> then upgrade/start the

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 2018-02-28T02:38:34, Patrick Donnelly wrote: > I think it will be necessary to reduce the actives to 1 (max_mds -> 1; > deactivate other ranks), shutdown standbys, upgrade the single active, > then upgrade/start the standbys. > > Unfortunately this didn't get flagged in

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 11:05 AM, John Spray wrote: > On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote: >> Hi all, >> >> I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and >> OSD's updated fine. >> >> When updating the MDS's (we

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote: > Hi all, > > I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and > OSD's updated fine. > > When updating the MDS's (we have 2 active and 1 standby), I started > with the standby. > > At the moment the

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 11:38 AM, Patrick Donnelly wrote: > On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote: >> (Sorry to spam) >> >> I guess it's related to this fix to the layout v2 feature id: >> https://github.com/ceph/ceph/pull/18782/files

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote: > (Sorry to spam) > > I guess it's related to this fix to the layout v2 feature id: > https://github.com/ceph/ceph/pull/18782/files > > -#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8, > "file layout

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

(Sorry to spam) I guess it's related to this fix to the layout v2 feature id: https://github.com/ceph/ceph/pull/18782/files -#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8, "file layout v2") +#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(9, "file layout v2")

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

More: here is the MDS_FEATURES map for a running 12.2.2 cluster: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} and here it

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Quoting Dan van der Ster (d...@vanderster.com): > Hi all, > > I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and > OSD's updated fine. 12.2.4? Did you mean 12.2.3? Or did I miss something? Gr. stefan -- | BIT BV http://www.bit.nl/Kamer van Koophandel 09090351 | GPG:

[ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hi all, I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and OSD's updated fine. When updating the MDS's (we have 2 active and 1 standby), I started with the standby. At the moment the standby MDS restarted into 12.2.4 [1], both active MDSs (still running 12.2.2) suicided like