Re: [ceph-users] OSD node reinstallation
Set noout, reinstall the OS without going the OSDs (including any journal partitions and maintaining any dmcrypt keys if you have encryption), install ceph, make sure the ceph.conf file is correct,zip start OSDs, unset noout once they're back up and in. All of the data the OSD needs to start is on the OSD itself. On Mon, Oct 29, 2018, 6:52 PM Luiz Gustavo Tonello < gustavo.tone...@gmail.com> wrote: > Hi list, > > I have a situation that I need to reinstall the O.S. of a single node in > my OSD cluster. > This node has 4 OSDs configured, each one has ~4 TB used. > > The way that I'm thinking to proceed is to put OSD down (one each time), > stop the OSD, reinstall the O.S., and finally add the OSDs again. > > But I want to know if there's a way to do this in a more simple process, > maybe put OSD in maintenance (noout), reinstall the O.S. without formatting > my Storage volumes, install CEPH again and enable OSDs again. > > There's a way like these? > > I'm running CEPH Jewel. > > Best, > -- > Luiz Gustavo P Tonello. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] reducing min_size on erasure coded pool may allow recovery ?
min_size should be at least k+1 for EC. There are times to use k for emergencies like you had. I would suggest seeing it back to 3 once your back to healthy. As far as why you needed to reduce min_size, my guess would be that recovery would have happened as long as k copies were up. Were the PG's refusing to backfill or just hang backfilled yet? On Mon, Oct 29, 2018, 9:24 PM Chad W Seys wrote: > Hi all, >Recently our cluster lost a drive and a node (3 drives) at the same > time. Our erasure coded pools are all k2m2, so if all is working > correctly no data is lost. >However, there were 4 PGs that stayed "incomplete" until I finally > took the suggestion in 'ceph health detail' to reduce min_size . (Thanks > for the hint!) I'm not sure what it was (likely 3), but setting it to 2 > caused all PGs to become active (though degraded) and the cluster is on > path to recovering fully. > >In replicated pools, would not ceph create replicas without the need > to reduce min_size? It seems odd to not recover automatically if > possible. Could someone explain what was going on there? > >Also, how to decide what min_size should be? > > Thanks! > Chad. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] reducing min_size on erasure coded pool may allow recovery ?
Hi all, Recently our cluster lost a drive and a node (3 drives) at the same time. Our erasure coded pools are all k2m2, so if all is working correctly no data is lost. However, there were 4 PGs that stayed "incomplete" until I finally took the suggestion in 'ceph health detail' to reduce min_size . (Thanks for the hint!) I'm not sure what it was (likely 3), but setting it to 2 caused all PGs to become active (though degraded) and the cluster is on path to recovering fully. In replicated pools, would not ceph create replicas without the need to reduce min_size? It seems odd to not recover automatically if possible. Could someone explain what was going on there? Also, how to decide what min_size should be? Thanks! Chad. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD node reinstallation
Hi list, I have a situation that I need to reinstall the O.S. of a single node in my OSD cluster. This node has 4 OSDs configured, each one has ~4 TB used. The way that I'm thinking to proceed is to put OSD down (one each time), stop the OSD, reinstall the O.S., and finally add the OSDs again. But I want to know if there's a way to do this in a more simple process, maybe put OSD in maintenance (noout), reinstall the O.S. without formatting my Storage volumes, install CEPH again and enable OSDs again. There's a way like these? I'm running CEPH Jewel. Best, -- Luiz Gustavo P Tonello. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cluster uses substantially more disk space after rebalancing
Is there a way to force OSDs to remove old data? Hi After I recreated one OSD + increased pg count of my erasure-coded (2+1) pool (which was way too low, only 100 for 9 osds) the cluster started to eat additional disk space. First I thought that was caused by the moved PGs using additional space during unfinished backfills. I pinned most of new PGs to old OSDs via `pg-upmap` and indeed it freed some space in the cluster. Then I reduced osd_max_backfills to 1 and started to remove upmap pins in small portions which allowed Ceph to finish backfills for these PGs. HOWEVER, used capacity still grows! It drops after moving each PG, but still grows overall. It has grown +1.3TB yesterday. In the same period of time clients have written only ~200 new objects (~800 MB, there are RBD images only). Why, what's using such big amount of additional space? Graphs from our prometheus are attached. Only ~200 objects were created by RBD clients yesterday, but used raw space increased +1.3 TB. Additional question is why ceph df / rados df tells there is only 16 TB actual data written, but it uses 29.8 TB (now 31 TB) of raw disk space. Shouldn't it be 16 / 2*3 = 24 TB ? ceph df output: [root@sill-01 ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 38 TiB 6.9 TiB 32 TiB 82.03 POOLS: NAME ID USED%USED MAX AVAIL OBJECTS ecpool_hdd 13 16 TiB 93.94 1.0 TiB 7611672 rpool_hdd 15 9.2 MiB 0 515 GiB 92 fs_meta44 20 KiB 0 515 GiB 23 fs_data45 0 B 0 1.0 TiB 0 How to heal it? -- С наилучшими пожеланиями, Виталий Филиппов ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph cluster uses substantially more disk space after rebalancing
Hi After I recreated one OSD + increased pg count of my erasure-coded (2+1) pool (which was way too low, only 100 for 9 osds) the cluster started to eat additional disk space. First I thought that was caused by the moved PGs using additional space during unfinished backfills. I pinned most of new PGs to old OSDs via `pg-upmap` and indeed it freed some space in the cluster. Then I reduced osd_max_backfills to 1 and started to remove upmap pins in small portions which allowed Ceph to finish backfills for these PGs. HOWEVER, used capacity still grows! It drops after moving each PG, but still grows overall. It has grown +1.3TB yesterday. In the same period of time clients have written only ~200 new objects (~800 MB, there are RBD images only). Why, what's using such big amount of additional space? Graphs from our prometheus are attached. Only ~200 objects were created by RBD clients yesterday, but used raw space increased +1.3 TB. Additional question is why ceph df / rados df tells there is only 16 TB actual data written, but it uses 29.8 TB (now 31 TB) of raw disk space. Shouldn't it be 16 / 2*3 = 24 TB ? ceph df output: [root@sill-01 ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 38 TiB 6.9 TiB 32 TiB 82.03 POOLS: NAME ID USED%USED MAX AVAIL OBJECTS ecpool_hdd 13 16 TiB 93.94 1.0 TiB 7611672 rpool_hdd 15 9.2 MiB 0 515 GiB 92 fs_meta44 20 KiB 0 515 GiB 23 fs_data45 0 B 0 1.0 TiB 0 How to heal it? -- With best regards, Vitaliy Filippov___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph-deploy with a specified osd ID
Gents, My cluster had a gap in the OSD sequence numbers at certain point. Basically, because of missing osd auth del/rm" in a previous disk replacement task for osd.17, a new osd.34 was created. It did not really bother me until recently when I tried to replace all smaller disks to bigger disks. Ceph seems also pick up the next available osd sequence number. When I replace osd.18, the disk came up online as osd.17. When I am doing osd.19, it became osd.18. It generated more backfull_wait pgs than sticking to the original osd number. Using ceph-deploy in version 10.2.3, is there a way to specify osd id when doing osd activate? Thank you. Jin. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] librados3
On Mon, Oct 29, 2018 at 7:48 AM Wido den Hollander wrote:
> On 10/29/18 12:42 PM, kefu chai wrote:
> > + ceph-user for more inputs in hope to get more inputs from librados
> > and librbd 's C++ interfaces.
> >
> > On Wed, Oct 24, 2018 at 1:34 AM Jason Dillaman wrote:
> >>
> >> On Tue, Oct 23, 2018 at 11:38 AM kefu chai wrote:
> >>>
> >>> we plan to introduce some non-backward-compatible changes[0] in
> >>> librados in the coming nautilus release. to be specific, these changes
> >>> are not API/ABI compatible with existing librados2. so i think it's
> >>> time to bump up the soversion of librados from 2 to 3. as bumping up
> >>> soversion is a major change in the life cycle of a library, i think we
> >>> will expect following changes:
> >>>
> >>> 0. piggyback more non-backward-compatible changes listed in
> >>> https://pad.ceph.com/p/librados3 in this change, so we can have less
> >>> soversion bump up.
> >>> 1. in nautilus, librados3 and librados3-{dev,devel} will be packaged
> >>> instead, librados2* will be maintained in LTS releases.
> >>
> >> Just note that this is going to have a potential ripple effect on
> >> non-Ceph packages that use librbd1 if they are also (incorrectly)
> >> specifying a dependency on librados2 [1].
> >>
> >>> 2. we will have separated C++ and C API librados after this change. so
> >>> librados3 will only provide the C API of librados, the C++ API will be
> >>> offered by libradospp, (the name may vary if you suggest a better
> >>> one). and they will be versioned and packaged separately, and will not
> >>> depend on each other. a PR is posted to do this, see [1]
> >>
> >> Do we have a good idea and/or list about who uses the librados C++
> >> API? I seem to vaguely recall perhaps one or so external users. If
> >> librgw/RGW and librbd eventually switch over to something similar to
> >> the API being worked on by Adam [2] and if there are no external users
> >> of the C++ API, should we take the time now to kill all the legacy C++
> >> API methods prior to the release of Nautilus?
> >>
>
> I know various users who use librados through phprados (PHP) and the
> rados-java (Java) bindings.
>
> Those bindings will need to be updated as well.
>
> Qemu and libvirt both mainly use librbd, but they use a very small part
> of librados to initiate the connection and set configuration values.
These should all be using the C API which isn't changing. We would
like to take inventory of who uses the librados/librbd C++ APIs
outside of the core Ceph project.
> >>> 3. in order to enable user to install librados2 and librados3 at the
> >>> same time, libceph-common.so will be versioned since nautilus.
> >>> libceph-common.so is an internal shared library used by ceph
> >>> libraries, tools and daemons. librados depends on libceph-common.
> >>> 4. some executables/libraries' dependencies will be updated
> >>> accordingly . for instance, librbd will depend on libradospp.
> >>>
> >>> if this model works fine, i guess we probably could expand it to librbd.
> >>
> >> It would be interesting to get a feel for who (if anyone) actually
> >> uses the librbd C++ interface before heading down that path. AFAIK,
> >> it's probably really only the rbd CLI tool -- and that's basically for
> >> the convenience of C++-style containers. If it's not being used by
> >> anyone else, perhaps our time could be better spent deprecating the
> >> library for external use to focus our development and maintanence
> >> efforts on the C (and wrapped-C) interface.
> >>
>
> Again, don't forget Qemu/KVM and libvirt who both are very heavy users
> of librbd
>
> Wido
>
> >>> any concerns?
> >>>
> >>> BTW, we discussed this topic last year the same time, see
> >>> https://www.spinics.net/lists/ceph-devel/msg38830.html =)
> >>>
> >>> ---
> >>> [0] for instance, https://github.com/ceph/ceph/pull/24498
> >>> [1] https://github.com/ceph/ceph/pull/24616
> >>> [2] a trello card for librados3:
> >>> https://trello.com/c/pmRkawYV/165-librados3-api-update-cleanup
> >>>
> >>> --
> >>> Regards
> >>> Kefu Chai
> >>
> >> [1] https://src.fedoraproject.org/rpms/qemu/blob/f29/f/qemu.spec#_192
> >> [2] https://github.com/ceph/ceph/pull/16715
> >>
> >> --
> >> Jason
> >
> >
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] librados3
On 10/29/18 12:42 PM, kefu chai wrote:
> + ceph-user for more inputs in hope to get more inputs from librados
> and librbd 's C++ interfaces.
>
> On Wed, Oct 24, 2018 at 1:34 AM Jason Dillaman wrote:
>>
>> On Tue, Oct 23, 2018 at 11:38 AM kefu chai wrote:
>>>
>>> we plan to introduce some non-backward-compatible changes[0] in
>>> librados in the coming nautilus release. to be specific, these changes
>>> are not API/ABI compatible with existing librados2. so i think it's
>>> time to bump up the soversion of librados from 2 to 3. as bumping up
>>> soversion is a major change in the life cycle of a library, i think we
>>> will expect following changes:
>>>
>>> 0. piggyback more non-backward-compatible changes listed in
>>> https://pad.ceph.com/p/librados3 in this change, so we can have less
>>> soversion bump up.
>>> 1. in nautilus, librados3 and librados3-{dev,devel} will be packaged
>>> instead, librados2* will be maintained in LTS releases.
>>
>> Just note that this is going to have a potential ripple effect on
>> non-Ceph packages that use librbd1 if they are also (incorrectly)
>> specifying a dependency on librados2 [1].
>>
>>> 2. we will have separated C++ and C API librados after this change. so
>>> librados3 will only provide the C API of librados, the C++ API will be
>>> offered by libradospp, (the name may vary if you suggest a better
>>> one). and they will be versioned and packaged separately, and will not
>>> depend on each other. a PR is posted to do this, see [1]
>>
>> Do we have a good idea and/or list about who uses the librados C++
>> API? I seem to vaguely recall perhaps one or so external users. If
>> librgw/RGW and librbd eventually switch over to something similar to
>> the API being worked on by Adam [2] and if there are no external users
>> of the C++ API, should we take the time now to kill all the legacy C++
>> API methods prior to the release of Nautilus?
>>
I know various users who use librados through phprados (PHP) and the
rados-java (Java) bindings.
Those bindings will need to be updated as well.
Qemu and libvirt both mainly use librbd, but they use a very small part
of librados to initiate the connection and set configuration values.
>>> 3. in order to enable user to install librados2 and librados3 at the
>>> same time, libceph-common.so will be versioned since nautilus.
>>> libceph-common.so is an internal shared library used by ceph
>>> libraries, tools and daemons. librados depends on libceph-common.
>>> 4. some executables/libraries' dependencies will be updated
>>> accordingly . for instance, librbd will depend on libradospp.
>>>
>>> if this model works fine, i guess we probably could expand it to librbd.
>>
>> It would be interesting to get a feel for who (if anyone) actually
>> uses the librbd C++ interface before heading down that path. AFAIK,
>> it's probably really only the rbd CLI tool -- and that's basically for
>> the convenience of C++-style containers. If it's not being used by
>> anyone else, perhaps our time could be better spent deprecating the
>> library for external use to focus our development and maintanence
>> efforts on the C (and wrapped-C) interface.
>>
Again, don't forget Qemu/KVM and libvirt who both are very heavy users
of librbd
Wido
>>> any concerns?
>>>
>>> BTW, we discussed this topic last year the same time, see
>>> https://www.spinics.net/lists/ceph-devel/msg38830.html =)
>>>
>>> ---
>>> [0] for instance, https://github.com/ceph/ceph/pull/24498
>>> [1] https://github.com/ceph/ceph/pull/24616
>>> [2] a trello card for librados3:
>>> https://trello.com/c/pmRkawYV/165-librados3-api-update-cleanup
>>>
>>> --
>>> Regards
>>> Kefu Chai
>>
>> [1] https://src.fedoraproject.org/rpms/qemu/blob/f29/f/qemu.spec#_192
>> [2] https://github.com/ceph/ceph/pull/16715
>>
>> --
>> Jason
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] librados3
+ ceph-user for more inputs in hope to get more inputs from librados
and librbd 's C++ interfaces.
On Wed, Oct 24, 2018 at 1:34 AM Jason Dillaman wrote:
>
> On Tue, Oct 23, 2018 at 11:38 AM kefu chai wrote:
> >
> > we plan to introduce some non-backward-compatible changes[0] in
> > librados in the coming nautilus release. to be specific, these changes
> > are not API/ABI compatible with existing librados2. so i think it's
> > time to bump up the soversion of librados from 2 to 3. as bumping up
> > soversion is a major change in the life cycle of a library, i think we
> > will expect following changes:
> >
> > 0. piggyback more non-backward-compatible changes listed in
> > https://pad.ceph.com/p/librados3 in this change, so we can have less
> > soversion bump up.
> > 1. in nautilus, librados3 and librados3-{dev,devel} will be packaged
> > instead, librados2* will be maintained in LTS releases.
>
> Just note that this is going to have a potential ripple effect on
> non-Ceph packages that use librbd1 if they are also (incorrectly)
> specifying a dependency on librados2 [1].
>
> > 2. we will have separated C++ and C API librados after this change. so
> > librados3 will only provide the C API of librados, the C++ API will be
> > offered by libradospp, (the name may vary if you suggest a better
> > one). and they will be versioned and packaged separately, and will not
> > depend on each other. a PR is posted to do this, see [1]
>
> Do we have a good idea and/or list about who uses the librados C++
> API? I seem to vaguely recall perhaps one or so external users. If
> librgw/RGW and librbd eventually switch over to something similar to
> the API being worked on by Adam [2] and if there are no external users
> of the C++ API, should we take the time now to kill all the legacy C++
> API methods prior to the release of Nautilus?
>
> > 3. in order to enable user to install librados2 and librados3 at the
> > same time, libceph-common.so will be versioned since nautilus.
> > libceph-common.so is an internal shared library used by ceph
> > libraries, tools and daemons. librados depends on libceph-common.
> > 4. some executables/libraries' dependencies will be updated
> > accordingly . for instance, librbd will depend on libradospp.
> >
> > if this model works fine, i guess we probably could expand it to librbd.
>
> It would be interesting to get a feel for who (if anyone) actually
> uses the librbd C++ interface before heading down that path. AFAIK,
> it's probably really only the rbd CLI tool -- and that's basically for
> the convenience of C++-style containers. If it's not being used by
> anyone else, perhaps our time could be better spent deprecating the
> library for external use to focus our development and maintanence
> efforts on the C (and wrapped-C) interface.
>
> > any concerns?
> >
> > BTW, we discussed this topic last year the same time, see
> > https://www.spinics.net/lists/ceph-devel/msg38830.html =)
> >
> > ---
> > [0] for instance, https://github.com/ceph/ceph/pull/24498
> > [1] https://github.com/ceph/ceph/pull/24616
> > [2] a trello card for librados3:
> > https://trello.com/c/pmRkawYV/165-librados3-api-update-cleanup
> >
> > --
> > Regards
> > Kefu Chai
>
> [1] https://src.fedoraproject.org/rpms/qemu/blob/f29/f/qemu.spec#_192
> [2] https://github.com/ceph/ceph/pull/16715
>
> --
> Jason
--
Regards
Kefu Chai
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-mds failure replaying journal
cephfs is recoverable. Just set mds_wipe_sessions to 1. After mds recovers,
set it back to 0 and flush journal (ceph daemon mds.x flush journal)
On Mon, Oct 29, 2018 at 7:13 PM Jon Morby (Fido) wrote:
> I've experimented and whilst the downgrade looks to be working, you end up
> with errors regarding unsupported feature "mimic" amongst others
>
> 2018-10-29 10:51:20.652047 7f6f1b9f5080 -1 ERROR: on disk data includes
> unsupported features: compat={},rocompat={},incompat={10=mimic ondisk layou
>
> so I gave up on that idea
>
> In addition to the cephfs volume (which is basically just mirrors and some
> backups) we have a large rbd deployment using the same ceph cluster, and if
> we lose that we're screwed ... the cephfs volume was more an "experiment"
> to see how viable it would be as an NFS replacement
>
> There's 26TB of data on there, so I'd rather not have to go off and
> redownload it all .. but losing it isn't the end of the world (but it will
> piss off a few friends)
>
> Jon
>
>
> - On 29 Oct, 2018, at 09:54, Zheng Yan wrote:
>
>
>
> On Mon, Oct 29, 2018 at 5:25 PM Jon Morby (Fido) wrote:
>
>> Hi
>>
>> Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure
>> that ceph-deploy doesn't do another major release upgrade without a lot of
>> warnings
>>
>> Either way, I'm currently getting errors that 13.2.1 isn't available /
>> shaman is offline / etc
>>
>> What's the best / recommended way of doing this downgrade across our
>> estate?
>>
>>
> You have already upgraded ceph-mon. I don't know If it can be safely
> downgraded (If I remember right, I corrupted monitor's data when
> downgrading ceph-mon from minic to luminous).
>
>
>>
>>
>> - On 29 Oct, 2018, at 08:19, Yan, Zheng wrote:
>>
>>
>> We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then
>> run 'ceph mds repaired fido_fs:1" .
>> Sorry for the trouble
>> Yan, Zheng
>>
>> On Mon, Oct 29, 2018 at 7:48 AM Jon Morby wrote:
>>
>>>
>>> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a
>>> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9
>>> and not jump a major release without warning)
>>>
>>> Anyway .. as a result, we ended up with an mds journal error and 1
>>> daemon reporting as damaged
>>>
>>> Having got nowhere trying to ask for help on irc, we've followed various
>>> forum posts and disaster recovery guides, we ended up resetting the journal
>>> which left the daemon as no longer “damaged” however we’re now seeing mds
>>> segfault whilst trying to replay
>>>
>>> https://pastebin.com/iSLdvu0b
>>>
>>>
>>>
>>> /build/ceph-13.2.2/src/mds/journal.cc: 1572: FAILED
>>> assert(g_conf->mds_wipe_sessions)
>>>
>>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>>> (stable)
>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x102) [0x7fad637f70f2]
>>> 2: (()+0x3162b7) [0x7fad637f72b7]
>>> 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b)
>>> [0x7a7a6b]
>>> 4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9]
>>> 5: (MDLog::_replay_thread()+0x864) [0x752164]
>>> 6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d]
>>> 7: (()+0x76ba) [0x7fad6305a6ba]
>>> 8: (clone()+0x6d) [0x7fad6288341d]
>>> NOTE: a copy of the executable, or `objdump -rdS ` is
>>> needed to interpret this.
>>>
>>>
>>> full logs
>>>
>>> https://pastebin.com/X5UG9vT2
>>>
>>>
>>> We’ve been unable to access the cephfs file system since all of this
>>> started …. attempts to mount fail with reports that “mds probably not
>>> available”
>>>
>>> Oct 28 23:47:02 mirrors kernel: [115602.911193] ceph: probably no mds
>>> server is up
>>>
>>>
>>> root@mds02:~# ceph -s
>>> cluster:
>>> id: 78d5bf7d-b074-47ab-8d73-bd4d99df98a5
>>> health: HEALTH_WARN
>>> 1 filesystem is degraded
>>> insufficient standby MDS daemons available
>>> too many PGs per OSD (276 > max 250)
>>>
>>> services:
>>> mon: 3 daemons, quorum mon01,mon02,mon03
>>> mgr: mon01(active), standbys: mon02, mon03
>>> mds: fido_fs-2/2/1 up {0=mds01=up:resolve,1=mds02=up:replay(laggy
>>> or crashed)}
>>> osd: 27 osds: 27 up, 27 in
>>>
>>> data:
>>> pools: 15 pools, 3168 pgs
>>> objects: 16.97 M objects, 30 TiB
>>> usage: 71 TiB used, 27 TiB / 98 TiB avail
>>> pgs: 3168 active+clean
>>>
>>> io:
>>> client: 680 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 345 op/s wr
>>>
>>>
>>> Before I just trash the entire fs and give up on ceph, does anyone have
>>> any suggestions as to how we can fix this?
>>>
>>> root@mds02:~# ceph versions
>>> {
>>> "mon": {
>>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>>> mimic (stable)": 3
>>> },
>>> "mgr": {
>>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>>> mimic (stable)": 3
>>> },
>>> "osd": {
>>> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
>>> luminous
Re: [ceph-users] ceph-mds failure replaying journal
I've experimented and whilst the downgrade looks to be working, you end up with
errors regarding unsupported feature "mimic" amongst others
2018-10-29 10:51:20.652047 7f6f1b9f5080 -1 ERROR: on disk data includes
unsupported features: compat={},rocompat={},incompat={10=mimic ondisk layou
so I gave up on that idea
In addition to the cephfs volume (which is basically just mirrors and some
backups) we have a large rbd deployment using the same ceph cluster, and if we
lose that we're screwed ... the cephfs volume was more an "experiment" to see
how viable it would be as an NFS replacement
There's 26TB of data on there, so I'd rather not have to go off and redownload
it all .. but losing it isn't the end of the world (but it will piss off a few
friends)
Jon
- On 29 Oct, 2018, at 09:54, Zheng Yan wrote:
> On Mon, Oct 29, 2018 at 5:25 PM Jon Morby (Fido) < [ mailto:j...@fido.net |
> j...@fido.net ] > wrote:
>> Hi
>> Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure
>> that
>> ceph-deploy doesn't do another major release upgrade without a lot of
>> warnings
>> Either way, I'm currently getting errors that 13.2.1 isn't available /
>> shaman is
>> offline / etc
>> What's the best / recommended way of doing this downgrade across our estate?
> You have already upgraded ceph-mon. I don't know If it can be safely
> downgraded
> (If I remember right, I corrupted monitor's data when downgrading ceph-mon
> from
> minic to luminous).
>> - On 29 Oct, 2018, at 08:19, Yan, Zheng < [ mailto:uker...@gmail.com |
>> uker...@gmail.com ] > wrote:
>>> We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run
>>> 'ceph
>>> mds repaired fido_fs:1" .
>>> Sorry for the trouble
>>> Yan, Zheng
>>> On Mon, Oct 29, 2018 at 7:48 AM Jon Morby < [ mailto:j...@fido.net |
>>> j...@fido.net
>>> ] > wrote:
We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a
ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9
and
not jump a major release without warning)
Anyway .. as a result, we ended up with an mds journal error and 1 daemon
reporting as damaged
Having got nowhere trying to ask for help on irc, we've followed various
forum
posts and disaster recovery guides, we ended up resetting the journal which
left the daemon as no longer “damaged” however we’re now seeing mds
segfault
whilst trying to replay
[ https://pastebin.com/iSLdvu0b | https://pastebin.com/iSLdvu0b ]
/build/ceph-13.2.2/src/mds/ [ http://journal.cc/ | journal.cc ] : 1572:
FAILED
assert(g_conf->mds_wipe_sessions)
ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102)
[0x7fad637f70f2]
2: (()+0x3162b7) [0x7fad637f72b7]
3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b)
[0x7a7a6b]
4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9]
5: (MDLog::_replay_thread()+0x864) [0x752164]
6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d]
7: (()+0x76ba) [0x7fad6305a6ba]
8: (clone()+0x6d) [0x7fad6288341d]
NOTE: a copy of the executable, or `objdump -rdS ` is needed to
interpret this.
full logs
[ https://pastebin.com/X5UG9vT2 | https://pastebin.com/X5UG9vT2 ]
We’ve been unable to access the cephfs file system since all of this
started ….
attempts to mount fail with reports that “mds probably not available”
Oct 28 23:47:02 mirrors kernel: [115602.911193] ceph: probably no mds
server is
up
root@mds02:~# ceph -s
cluster:
id: 78d5bf7d-b074-47ab-8d73-bd4d99df98a5
health: HEALTH_WARN
1 filesystem is degraded
insufficient standby MDS daemons available
too many PGs per OSD (276 > max 250)
services:
mon: 3 daemons, quorum mon01,mon02,mon03
mgr: mon01(active), standbys: mon02, mon03
mds: fido_fs-2/2/1 up {0=mds01=up:resolve,1=mds02=up:replay(laggy or
crashed)}
osd: 27 osds: 27 up, 27 in
data:
pools: 15 pools, 3168 pgs
objects: 16.97 M objects, 30 TiB
usage: 71 TiB used, 27 TiB / 98 TiB avail
pgs: 3168 active+clean
io:
client: 680 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 345 op/s wr
Before I just trash the entire fs and give up on ceph, does anyone have any
suggestions as to how we can fix this?
root@mds02:~# ceph versions
{
"mon": {
"ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
(stable)":
3
},
"mgr": {
"ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
(stable)":
3
},
"osd": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)": 27
},
"mds": {
"ceph version 13.2.2
Re: [ceph-users] ceph-mds failure replaying journal
please try again debug_mds=10 and send log to me Regards Yan, Zheng On Mon, Oct 29, 2018 at 6:30 PM Jon Morby (Fido) wrote: > fyi, downgrading to 13.2.1 doesn't seem to have fixed the issue either :( > > --- end dump of recent events --- > 2018-10-29 10:27:50.440 7feb58b43700 -1 *** Caught signal (Aborted) ** > in thread 7feb58b43700 thread_name:md_log_replay > > ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic > (stable) > 1: (()+0x3ebf40) [0x55deff8e0f40] > 2: (()+0x11390) [0x7feb68246390] > 3: (gsignal()+0x38) [0x7feb67993428] > 4: (abort()+0x16a) [0x7feb6799502a] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x250) [0x7feb689a5630] > 6: (()+0x2e26a7) [0x7feb689a56a7] > 7: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b) > [0x55deff8ccc8b] > 8: (EUpdate::replay(MDSRank*)+0x39) [0x55deff8ce1c9] > 9: (MDLog::_replay_thread()+0x864) [0x55deff876974] > 10: (MDLog::ReplayThread::entry()+0xd) [0x55deff61a95d] > 11: (()+0x76ba) [0x7feb6823c6ba] > 12: (clone()+0x6d) [0x7feb67a6541d] > NOTE: a copy of the executable, or `objdump -rdS ` is needed > to interpret this. > > --- begin dump of recent events --- > 0> 2018-10-29 10:27:50.440 7feb58b43700 -1 *** Caught signal > (Aborted) ** > in thread 7feb58b43700 thread_name:md_log_replay > > ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic > (stable) > 1: (()+0x3ebf40) [0x55deff8e0f40] > 2: (()+0x11390) [0x7feb68246390] > 3: (gsignal()+0x38) [0x7feb67993428] > 4: (abort()+0x16a) [0x7feb6799502a] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x250) [0x7feb689a5630] > 6: (()+0x2e26a7) [0x7feb689a56a7] > 7: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b) > [0x55deff8ccc8b] > 8: (EUpdate::replay(MDSRank*)+0x39) [0x55deff8ce1c9] > 9: (MDLog::_replay_thread()+0x864) [0x55deff876974] > 10: (MDLog::ReplayThread::entry()+0xd) [0x55deff61a95d] > 11: (()+0x76ba) [0x7feb6823c6ba] > 12: (clone()+0x6d) [0x7feb67a6541d] > NOTE: a copy of the executable, or `objdump -rdS ` is needed > to interpret this. > > --- logging levels --- >0/ 5 none >0/ 0 lockdep >0/ 0 context >0/ 0 crush >3/ 3 mds >1/ 5 mds_balancer >1/ 5 mds_locker >1/ 5 mds_log >1/ 5 mds_log_expire >1/ 5 mds_migrator >0/ 0 buffer >0/ 0 timer >0/ 0 filer >0/ 1 striper >0/ 0 objecter >0/ 0 rados >0/ 0 rbd >0/ 5 rbd_mirror >0/ 5 rbd_replay >0/ 0 journaler >0/ 5 objectcacher >0/ 0 client >0/ 0 osd >0/ 0 optracker >0/ 0 objclass >0/ 0 filestore >0/ 0 journal >0/ 0 ms >0/ 0 mon >0/ 0 monc >0/ 0 paxos >0/ 0 tp >0/ 0 auth >1/ 5 crypto >0/ 0 finisher >1/ 1 reserver >0/ 0 heartbeatmap >0/ 0 perfcounter >0/ 0 rgw >1/ 5 rgw_sync >1/10 civetweb >1/ 5 javaclient >0/ 0 asok >0/ 0 throttle >0/ 0 refs >1/ 5 xio >1/ 5 compressor >1/ 5 bluestore >1/ 5 bluefs >1/ 3 bdev >1/ 5 kstore >4/ 5 rocksdb >4/ 5 leveldb >4/ 5 memdb >1/ 5 kinetic >1/ 5 fuse >1/ 5 mgr >1/ 5 mgrc >1/ 5 dpdk >1/ 5 eventtrace > 99/99 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 1 > max_new 1000 > log_file /var/log/ceph/ceph-mds.mds04.log > --- end dump of recent events --- > > > - On 29 Oct, 2018, at 09:25, Jon Morby wrote: > > Hi > > Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure > that ceph-deploy doesn't do another major release upgrade without a lot of > warnings > > Either way, I'm currently getting errors that 13.2.1 isn't available / > shaman is offline / etc > > What's the best / recommended way of doing this downgrade across our > estate? > > > > - On 29 Oct, 2018, at 08:19, Yan, Zheng wrote: > > > We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run > 'ceph mds repaired fido_fs:1" . > Sorry for the trouble > Yan, Zheng > > On Mon, Oct 29, 2018 at 7:48 AM Jon Morby wrote: > >> >> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a >> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9 >> and not jump a major release without warning) >> >> Anyway .. as a result, we ended up with an mds journal error and 1 daemon >> reporting as damaged >> >> Having got nowhere trying to ask for help on irc, we've followed various >> forum posts and disaster recovery guides, we ended up resetting the journal >> which left the daemon as no longer “damaged” however we’re now seeing mds >> segfault whilst trying to replay >> >> https://pastebin.com/iSLdvu0b >> >> >> >> /build/ceph-13.2.2/src/mds/journal.cc: 1572: FAILED >> assert(g_conf->mds_wipe_sessions) >> >> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic >> (stable) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x102)
Re: [ceph-users] ceph-mds failure replaying journal
fyi, downgrading to 13.2.1 doesn't seem to have fixed the issue either :( --- end dump of recent events --- 2018-10-29 10:27:50.440 7feb58b43700 -1 *** Caught signal (Aborted) ** in thread 7feb58b43700 thread_name:md_log_replay ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable) 1: (()+0x3ebf40) [0x55deff8e0f40] 2: (()+0x11390) [0x7feb68246390] 3: (gsignal()+0x38) [0x7feb67993428] 4: (abort()+0x16a) [0x7feb6799502a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7feb689a5630] 6: (()+0x2e26a7) [0x7feb689a56a7] 7: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b) [0x55deff8ccc8b] 8: (EUpdate::replay(MDSRank*)+0x39) [0x55deff8ce1c9] 9: (MDLog::_replay_thread()+0x864) [0x55deff876974] 10: (MDLog::ReplayThread::entry()+0xd) [0x55deff61a95d] 11: (()+0x76ba) [0x7feb6823c6ba] 12: (clone()+0x6d) [0x7feb67a6541d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2018-10-29 10:27:50.440 7feb58b43700 -1 *** Caught signal (Aborted) ** in thread 7feb58b43700 thread_name:md_log_replay ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable) 1: (()+0x3ebf40) [0x55deff8e0f40] 2: (()+0x11390) [0x7feb68246390] 3: (gsignal()+0x38) [0x7feb67993428] 4: (abort()+0x16a) [0x7feb6799502a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7feb689a5630] 6: (()+0x2e26a7) [0x7feb689a56a7] 7: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b) [0x55deff8ccc8b] 8: (EUpdate::replay(MDSRank*)+0x39) [0x55deff8ce1c9] 9: (MDLog::_replay_thread()+0x864) [0x55deff876974] 10: (MDLog::ReplayThread::entry()+0xd) [0x55deff61a95d] 11: (()+0x76ba) [0x7feb6823c6ba] 12: (clone()+0x6d) [0x7feb67a6541d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 3/ 3 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 99/99 (syslog threshold) -1/-1 (stderr threshold) max_recent 1 max_new 1000 log_file /var/log/ceph/ceph-mds.mds04.log --- end dump of recent events --- - On 29 Oct, 2018, at 09:25, Jon Morby wrote: > Hi > Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure that > ceph-deploy doesn't do another major release upgrade without a lot of warnings > Either way, I'm currently getting errors that 13.2.1 isn't available / shaman > is > offline / etc > What's the best / recommended way of doing this downgrade across our estate? > - On 29 Oct, 2018, at 08:19, Yan, Zheng wrote: >> We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run >> 'ceph >> mds repaired fido_fs:1" . >> Sorry for the trouble >> Yan, Zheng >> On Mon, Oct 29, 2018 at 7:48 AM Jon Morby < [ mailto:j...@fido.net | >> j...@fido.net >> ] > wrote: >>> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a >>> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9 and >>> not jump a major release without warning) >>> Anyway .. as a result, we ended up with an mds journal error and 1 daemon >>> reporting as damaged >>> Having got nowhere trying to ask for help on irc, we've followed various >>> forum >>> posts and disaster recovery guides, we ended up resetting the journal which >>> left the daemon as no longer “damaged” however we’re now seeing mds segfault >>> whilst trying to replay >>> [ https://pastebin.com/iSLdvu0b | https://pastebin.com/iSLdvu0b ] >>> /build/ceph-13.2.2/src/mds/ [ http://journal.cc/ | journal.cc ] : 1572: >>> FAILED >>> assert(g_conf->mds_wipe_sessions) >>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic >>> (stable) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x102) >>> [0x7fad637f70f2] >>> 2: (()+0x3162b7) [0x7fad637f72b7] >>> 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b) >>> [0x7a7a6b] >>> 4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9] >>> 5: (MDLog::_replay_thread()+0x864) [0x752164] >>> 6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d] >>> 7: (()+0x76ba) [0x7fad6305a6ba] >>> 8: (clone()+0x6d) [0x7fad6288341d] >>> NOTE:
Re: [ceph-users] ceph-mds failure replaying journal
On Mon, Oct 29, 2018 at 5:25 PM Jon Morby (Fido) wrote:
> Hi
>
> Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure
> that ceph-deploy doesn't do another major release upgrade without a lot of
> warnings
>
> Either way, I'm currently getting errors that 13.2.1 isn't available /
> shaman is offline / etc
>
> What's the best / recommended way of doing this downgrade across our
> estate?
>
>
You have already upgraded ceph-mon. I don't know If it can be safely
downgraded (If I remember right, I corrupted monitor's data when
downgrading ceph-mon from minic to luminous).
>
>
> - On 29 Oct, 2018, at 08:19, Yan, Zheng wrote:
>
>
> We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run
> 'ceph mds repaired fido_fs:1" .
> Sorry for the trouble
> Yan, Zheng
>
> On Mon, Oct 29, 2018 at 7:48 AM Jon Morby wrote:
>
>>
>> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a
>> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9
>> and not jump a major release without warning)
>>
>> Anyway .. as a result, we ended up with an mds journal error and 1 daemon
>> reporting as damaged
>>
>> Having got nowhere trying to ask for help on irc, we've followed various
>> forum posts and disaster recovery guides, we ended up resetting the journal
>> which left the daemon as no longer “damaged” however we’re now seeing mds
>> segfault whilst trying to replay
>>
>> https://pastebin.com/iSLdvu0b
>>
>>
>>
>> /build/ceph-13.2.2/src/mds/journal.cc: 1572: FAILED
>> assert(g_conf->mds_wipe_sessions)
>>
>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>> (stable)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x102) [0x7fad637f70f2]
>> 2: (()+0x3162b7) [0x7fad637f72b7]
>> 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b)
>> [0x7a7a6b]
>> 4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9]
>> 5: (MDLog::_replay_thread()+0x864) [0x752164]
>> 6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d]
>> 7: (()+0x76ba) [0x7fad6305a6ba]
>> 8: (clone()+0x6d) [0x7fad6288341d]
>> NOTE: a copy of the executable, or `objdump -rdS ` is needed
>> to interpret this.
>>
>>
>> full logs
>>
>> https://pastebin.com/X5UG9vT2
>>
>>
>> We’ve been unable to access the cephfs file system since all of this
>> started …. attempts to mount fail with reports that “mds probably not
>> available”
>>
>> Oct 28 23:47:02 mirrors kernel: [115602.911193] ceph: probably no mds
>> server is up
>>
>>
>> root@mds02:~# ceph -s
>> cluster:
>> id: 78d5bf7d-b074-47ab-8d73-bd4d99df98a5
>> health: HEALTH_WARN
>> 1 filesystem is degraded
>> insufficient standby MDS daemons available
>> too many PGs per OSD (276 > max 250)
>>
>> services:
>> mon: 3 daemons, quorum mon01,mon02,mon03
>> mgr: mon01(active), standbys: mon02, mon03
>> mds: fido_fs-2/2/1 up {0=mds01=up:resolve,1=mds02=up:replay(laggy or
>> crashed)}
>> osd: 27 osds: 27 up, 27 in
>>
>> data:
>> pools: 15 pools, 3168 pgs
>> objects: 16.97 M objects, 30 TiB
>> usage: 71 TiB used, 27 TiB / 98 TiB avail
>> pgs: 3168 active+clean
>>
>> io:
>> client: 680 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 345 op/s wr
>>
>>
>> Before I just trash the entire fs and give up on ceph, does anyone have
>> any suggestions as to how we can fix this?
>>
>> root@mds02:~# ceph versions
>> {
>> "mon": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>> mimic (stable)": 3
>> },
>> "mgr": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>> mimic (stable)": 3
>> },
>> "osd": {
>> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
>> luminous (stable)": 27
>> },
>> "mds": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>> mimic (stable)": 2
>> },
>> "overall": {
>> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
>> luminous (stable)": 27,
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
>> mimic (stable)": 8
>> }
>> }
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> --
> --
> Jon Morby
> FidoNet - the internet made simple!
> 10 - 16 Tiller Road, London, E14 8PX
> tel: 0345 004 3050 / fax: 0345 004 3051
>
> Need more rack space?
> Check out our Co-Lo offerings at http://www.fido.net/services/colo/
> 32 amp racks in London and Brighton
> Linx ConneXions available at all Fido sites!
> https://www.fido.net/services/backbone/connexions/
> PGP Key : 26DC B618 DE9E F9CB F8B7 1EFA
> 2A64 BA69 B3B5 AD3A - http://jonmorby.com/B3B5AD3A.asc
>
___
ceph-users
Re: [ceph-users] ceph-mds failure replaying journal
Hi
Ideally we'd like to undo the whole accidental upgrade to 13.x and ensure that
ceph-deploy doesn't do another major release upgrade without a lot of warnings
Either way, I'm currently getting errors that 13.2.1 isn't available / shaman
is offline / etc
What's the best / recommended way of doing this downgrade across our estate?
- On 29 Oct, 2018, at 08:19, Yan, Zheng wrote:
> We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run
> 'ceph
> mds repaired fido_fs:1" .
> Sorry for the trouble
> Yan, Zheng
> On Mon, Oct 29, 2018 at 7:48 AM Jon Morby < [ mailto:j...@fido.net |
> j...@fido.net
> ] > wrote:
>> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a
>> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9 and
>> not jump a major release without warning)
>> Anyway .. as a result, we ended up with an mds journal error and 1 daemon
>> reporting as damaged
>> Having got nowhere trying to ask for help on irc, we've followed various
>> forum
>> posts and disaster recovery guides, we ended up resetting the journal which
>> left the daemon as no longer “damaged” however we’re now seeing mds segfault
>> whilst trying to replay
>> [ https://pastebin.com/iSLdvu0b | https://pastebin.com/iSLdvu0b ]
>> /build/ceph-13.2.2/src/mds/ [ http://journal.cc/ | journal.cc ] : 1572:
>> FAILED
>> assert(g_conf->mds_wipe_sessions)
>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x102)
>> [0x7fad637f70f2]
>> 2: (()+0x3162b7) [0x7fad637f72b7]
>> 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b)
>> [0x7a7a6b]
>> 4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9]
>> 5: (MDLog::_replay_thread()+0x864) [0x752164]
>> 6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d]
>> 7: (()+0x76ba) [0x7fad6305a6ba]
>> 8: (clone()+0x6d) [0x7fad6288341d]
>> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
>> interpret this.
>> full logs
>> [ https://pastebin.com/X5UG9vT2 | https://pastebin.com/X5UG9vT2 ]
>> We’ve been unable to access the cephfs file system since all of this started
>> ….
>> attempts to mount fail with reports that “mds probably not available”
>> Oct 28 23:47:02 mirrors kernel: [115602.911193] ceph: probably no mds server
>> is
>> up
>> root@mds02:~# ceph -s
>> cluster:
>> id: 78d5bf7d-b074-47ab-8d73-bd4d99df98a5
>> health: HEALTH_WARN
>> 1 filesystem is degraded
>> insufficient standby MDS daemons available
>> too many PGs per OSD (276 > max 250)
>> services:
>> mon: 3 daemons, quorum mon01,mon02,mon03
>> mgr: mon01(active), standbys: mon02, mon03
>> mds: fido_fs-2/2/1 up {0=mds01=up:resolve,1=mds02=up:replay(laggy or
>> crashed)}
>> osd: 27 osds: 27 up, 27 in
>> data:
>> pools: 15 pools, 3168 pgs
>> objects: 16.97 M objects, 30 TiB
>> usage: 71 TiB used, 27 TiB / 98 TiB avail
>> pgs: 3168 active+clean
>> io:
>> client: 680 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 345 op/s wr
>> Before I just trash the entire fs and give up on ceph, does anyone have any
>> suggestions as to how we can fix this?
>> root@mds02:~# ceph versions
>> {
>> "mon": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>> (stable)":
>> 3
>> },
>> "mgr": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>> (stable)":
>> 3
>> },
>> "osd": {
>> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
>> (stable)": 27
>> },
>> "mds": {
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>> (stable)":
>> 2
>> },
>> "overall": {
>> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
>> (stable)": 27,
>> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>> (stable)":
>> 8
>> }
>> }
>> ___
>> ceph-users mailing list
>> [ mailto:ceph-users@lists.ceph.com | ceph-users@lists.ceph.com ]
>> [ http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com |
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ]
--
Jon Morby
FidoNet - the internet made simple!
10 - 16 Tiller Road, London, E14 8PX
tel: 0345 004 3050 / fax: 0345 004 3051
Need more rack space?
Check out our Co-Lo offerings at [ http://www.fido.net/services/colo/%20 |
http://www.fido.net/services/colo/ ] 32 amp racks in London and Brighton
Linx ConneXions available at all Fido sites! [
https://www.fido.net/services/backbone/connexions/ |
https://www.fido.net/services/backbone/connexions/ ]
[ http://jonmorby.com/B3B5AD3A.asc | PGP Key ] : 26DC B618 DE9E F9CB F8B7 1EFA
2A64 BA69 B3B5 AD3A - http://jonmorby.com/B3B5AD3A.asc
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-mds failure replaying journal
We backported a wrong patch to 13.2.2. downgrade ceph to 13.2.1, then run
'ceph mds repaired fido_fs:1" .
Sorry for the trouble
Yan, Zheng
On Mon, Oct 29, 2018 at 7:48 AM Jon Morby wrote:
>
> We accidentally found ourselves upgraded from 12.2.8 to 13.2.2 after a
> ceph-deploy install went awry (we were expecting it to upgrade to 12.2.9
> and not jump a major release without warning)
>
> Anyway .. as a result, we ended up with an mds journal error and 1 daemon
> reporting as damaged
>
> Having got nowhere trying to ask for help on irc, we've followed various
> forum posts and disaster recovery guides, we ended up resetting the journal
> which left the daemon as no longer “damaged” however we’re now seeing mds
> segfault whilst trying to replay
>
> https://pastebin.com/iSLdvu0b
>
>
>
> /build/ceph-13.2.2/src/mds/journal.cc: 1572: FAILED
> assert(g_conf->mds_wipe_sessions)
>
> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
> (stable)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x102) [0x7fad637f70f2]
> 2: (()+0x3162b7) [0x7fad637f72b7]
> 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x5f4b)
> [0x7a7a6b]
> 4: (EUpdate::replay(MDSRank*)+0x39) [0x7a8fa9]
> 5: (MDLog::_replay_thread()+0x864) [0x752164]
> 6: (MDLog::ReplayThread::entry()+0xd) [0x4f021d]
> 7: (()+0x76ba) [0x7fad6305a6ba]
> 8: (clone()+0x6d) [0x7fad6288341d]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed
> to interpret this.
>
>
> full logs
>
> https://pastebin.com/X5UG9vT2
>
>
> We’ve been unable to access the cephfs file system since all of this
> started …. attempts to mount fail with reports that “mds probably not
> available”
>
> Oct 28 23:47:02 mirrors kernel: [115602.911193] ceph: probably no mds
> server is up
>
>
> root@mds02:~# ceph -s
> cluster:
> id: 78d5bf7d-b074-47ab-8d73-bd4d99df98a5
> health: HEALTH_WARN
> 1 filesystem is degraded
> insufficient standby MDS daemons available
> too many PGs per OSD (276 > max 250)
>
> services:
> mon: 3 daemons, quorum mon01,mon02,mon03
> mgr: mon01(active), standbys: mon02, mon03
> mds: fido_fs-2/2/1 up {0=mds01=up:resolve,1=mds02=up:replay(laggy or
> crashed)}
> osd: 27 osds: 27 up, 27 in
>
> data:
> pools: 15 pools, 3168 pgs
> objects: 16.97 M objects, 30 TiB
> usage: 71 TiB used, 27 TiB / 98 TiB avail
> pgs: 3168 active+clean
>
> io:
> client: 680 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 345 op/s wr
>
>
> Before I just trash the entire fs and give up on ceph, does anyone have
> any suggestions as to how we can fix this?
>
> root@mds02:~# ceph versions
> {
> "mon": {
> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
> mimic (stable)": 3
> },
> "mgr": {
> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
> mimic (stable)": 3
> },
> "osd": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
> luminous (stable)": 27
> },
> "mds": {
> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
> mimic (stable)": 2
> },
> "overall": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
> luminous (stable)": 27,
> "ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
> mimic (stable)": 8
> }
> }
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
