I thought somebody else was going to contact you about this, but in case it
didn't happen off-list:
This appears to be an embarrassing issue on our end where we alter the disk
state despite not being able to start up all the way, and rely on our users
to read release notes carefully. ;) :/
At this point, you're going to need to manually manipulate the OSDs. It
will involve identifying exactly what the Luminous daemons did; *hopefully*
they have only set new features on disk. If that's so, you can probably use
ceph-dencoder on whatever the feature flags file is and pull out everything
added after hammer.
But I'm not sure if that's the only thing that happened. You may need to
get some consulting from somebody who has experience doing Ceph cluster
recovery.
-Greg
On Thu, Nov 16, 2017 at 7:58 PM Gianfilippo <gia...@gmail.com> wrote:
> Hi all,
> I did a pretty bit mistake doing our upgrade from hammer to luminous,
> skipping the jewel release.
> When I realized and tried to switch back to jewel, it was too late -
> the cluster now won't start, complaining about "The disk uses features
> unsupported by the executable.":
>
> 2017-11-17 01:27:26.190971 7fb446ab58c0 0 ceph version 0.94.10
> (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19638
> 2017-11-17 01:27:26.209600 7fb446ab58c0 0
> filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
> 2017-11-17 01:27:26.277323 7fb446ab58c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is supported and appears to work
> 2017-11-17 01:27:26.277353 7fb446ab58c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2017-11-17 01:27:26.302508 7fb446ab58c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> syncfs(2) syscall fully supported (by glibc and kernel)
> 2017-11-17 01:27:26.302668 7fb446ab58c0 0
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
> disabled by conf
> 2017-11-17 01:27:26.325121 7fb446ab58c0 0
> filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
> mode: checkpoint is not enabled
> 2017-11-17 01:27:26.343360 7fb446ab58c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:26.393876 7fb446ab58c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:26.394746 7fb446ab58c0 -1 osd.2 0 The disk uses
> features unsupported by the executable.
> 2017-11-17 01:27:26.394758 7fb446ab58c0 -1 osd.2 0 ondisk features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
> hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
> attr,16=deletes in missing set}
> 2017-11-17 01:27:26.394780 7fb446ab58c0 -1 osd.2 0 daemon features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
> objects,12=transaction hints,13=pg meta object}
> 2017-11-17 01:27:26.394794 7fb446ab58c0 -1 osd.2 0 Cannot write to disk!
> Missing features: compat={},rocompat={},incompat={14=explicit missing
> set,15=fastinfo pg attr,16=deletes in missing set}
> 2017-11-17 01:27:26.419854 7fb446ab58c0 1 journal close
> /var/lib/ceph/osd/ceph-2/journal
> 2017-11-17 01:27:26.422687 7fb446ab58c0 -1 ESC[0;31m ** ERROR: osd init
> failed: (95) Operation not supportedESC[0m
> 2017-11-17 01:27:26.863514 7fcc5f1428c0 0 ceph version 0.94.10
> (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19731
> 2017-11-17 01:27:26.878617 7fcc5f1428c0 0
> filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
> 2017-11-17 01:27:26.880689 7fcc5f1428c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is supported and appears to work
> 2017-11-17 01:27:26.880703 7fcc5f1428c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2017-11-17 01:27:26.898681 7fcc5f1428c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> syncfs(2) syscall fully supported (by glibc and kernel)
> 2017-11-17 01:27:26.898829 7fcc5f1428c0 0
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
> disabled by conf
> 2017-11-17 01:27:26.906300 7fcc5f1428c0 0
> filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
> mode: checkpoint is not enabled
> 2017-11-17 01:27:26.917013 7fcc5f1428c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:26.925628 7fcc5f1428c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:26.926496 7fcc5f1428c0 -1 osd.2 0 The disk uses
> features unsupported by the executable.
> 2017-11-17 01:27:26.926509 7fcc5f1428c0 -1 osd.2 0 ondisk features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
> hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
> attr,16=deletes in missing set}
> 2017-11-17 01:27:26.926533 7fcc5f1428c0 -1 osd.2 0 daemon features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
> objects,12=transaction hints,13=pg meta object}
> 2017-11-17 01:27:26.926553 7fcc5f1428c0 -1 osd.2 0 Cannot write to disk!
> Missing features: compat={},rocompat={},incompat={14=explicit missing
> set,15=fastinfo pg attr,16=deletes in missing set}
> 2017-11-17 01:27:26.927159 7fcc5f1428c0 1 journal close
> /var/lib/ceph/osd/ceph-2/journal
> 2017-11-17 01:27:26.929073 7fcc5f1428c0 -1 ESC[0;31m ** ERROR: osd init
> failed: (95) Operation not supportedESC[0m
> 2017-11-17 01:27:27.364931 7f16ccdc78c0 0 ceph version 0.94.10
> (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19821
> 2017-11-17 01:27:27.379962 7f16ccdc78c0 0
> filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
> 2017-11-17 01:27:27.381509 7f16ccdc78c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is supported and appears to work
> 2017-11-17 01:27:27.381524 7f16ccdc78c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2017-11-17 01:27:27.397192 7f16ccdc78c0 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> syncfs(2) syscall fully supported (by glibc and kernel)
> 2017-11-17 01:27:27.397324 7f16ccdc78c0 0
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
> disabled by conf
> 2017-11-17 01:27:27.402018 7f16ccdc78c0 0
> filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
> mode: checkpoint is not enabled
> 2017-11-17 01:27:27.412815 7f16ccdc78c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 19: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:27.421621 7f16ccdc78c0 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 19: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 01:27:27.422471 7f16ccdc78c0 -1 osd.2 0 The disk uses
> features unsupported by the executable.
> 2017-11-17 01:27:27.422482 7f16ccdc78c0 -1 osd.2 0 ondisk features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
> hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
> attr,16=deletes in missing set}
> 2017-11-17 01:27:27.422495 7f16ccdc78c0 -1 osd.2 0 daemon features
> compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
> object,3=object
>
> locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
> objects,12=transaction hints,13=pg meta object}
> 2017-11-17 01:27:27.422515 7f16ccdc78c0 -1 osd.2 0 Cannot write to disk!
> Missing features: compat={},rocompat={},incompat={14=explicit missing
> set,15=fastinfo pg attr,16=deletes in missing set}
> 2017-11-17 01:27:27.424247 7f16ccdc78c0 1 journal close
> /var/lib/ceph/osd/ceph-2/journal
> 2017-11-17 01:27:27.426533 7f16ccdc78c0 -1 ESC[0;31m ** ERROR: osd init
> failed: (95) Operation not supportedESC[0m
>
>
>
>
> As the cluster won't start on Jewel, i can't give a ceph osd set
> sortbitwise to complete a successful upgrade to Luminous. So trying to
> start on Luminous i get the following errors from OSD:
>
> 2017-11-17 04:57:37.779635 7f04c28ead00 0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2017-11-17 04:57:37.779682 7f04c28ead00 0 ceph version 12.2.1
> (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
> (unknown), pid 8531
> 2017-11-17 04:57:37.783538 7f04c28ead00 -1 Public network was set, but
> cluster network was not set
> 2017-11-17 04:57:37.783562 7f04c28ead00 -1 Using public network also
> for cluster network
> 2017-11-17 04:57:37.790066 7f04c28ead00 0 pidfile_write: ignore empty
> --pid-file
> 2017-11-17 04:57:37.800558 7f04c28ead00 0 load: jerasure load: lrc
> load: isa
> 2017-11-17 04:57:37.801160 7f04c28ead00 0
> filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
> 2017-11-17 04:57:37.802397 7f04c28ead00 0
> filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
> 2017-11-17 04:57:37.802951 7f04c28ead00 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2017-11-17 04:57:37.802965 7f04c28ead00 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config
> option
> 2017-11-17 04:57:37.802970 7f04c28ead00 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> splice() is disabled via 'filestore splice' config option
> 2017-11-17 04:57:37.817358 7f04c28ead00 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
> syncfs(2) syscall fully supported (by glibc and kernel)
> 2017-11-17 04:57:37.817510 7f04c28ead00 0
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
> disabled by conf
> 2017-11-17 04:57:37.818348 7f04c28ead00 0
> filestore(/var/lib/ceph/osd/ceph-2) start omap initiation
> 2017-11-17 04:57:37.818791 7f04c28ead00 1 leveldb: Recovering log #275311
> 2017-11-17 04:57:37.820333 7f04c28ead00 1 leveldb: Delete type=0 #275311
>
> 2017-11-17 04:57:37.820397 7f04c28ead00 1 leveldb: Delete type=3 #275310
>
> 2017-11-17 04:57:38.555491 7f04c28ead00 0
> filestore(/var/lib/ceph/osd/ceph-2) mount(1758): enabling WRITEAHEAD
> journal mode: checkpoint is not enabled
> 2017-11-17 04:57:38.559310 7f04c28ead00 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 28: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 04:57:38.561591 7f04c28ead00 1 journal _open
> /var/lib/ceph/osd/ceph-2/journal fd 28: 5368709120 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2017-11-17 04:57:38.563244 7f04c28ead00 1
> filestore(/var/lib/ceph/osd/ceph-2) upgrade(1365)
> 2017-11-17 04:57:38.564625 7f04c28ead00 0 <cls>
> /build/ceph-12.2.1/src/cls/hello/cls_hello.cc:296: loading cls_hello
> 2017-11-17 04:57:38.564658 7f04c28ead00 0 _get_class not permitted to
> load lua
> 2017-11-17 04:57:38.567611 7f04c28ead00 0 <cls>
> /build/ceph-12.2.1/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
> 2017-11-17 04:57:38.568554 7f04c28ead00 0 _get_class not permitted to
> load sdk
> 2017-11-17 04:57:38.568774 7f04c28ead00 0 _get_class not permitted to
> load kvs
> 2017-11-17 04:57:38.568793 7f04c28ead00 1 osd.2 0 warning: got an error
> loading one or more classes: (1) Operation not permitted
> 2017-11-17 04:57:38.569061 7f04c28ead00 0 osd.2 1531 crush map has
> features 1107558400, adjusting msgr requires for clients
> 2017-11-17 04:57:38.569077 7f04c28ead00 0 osd.2 1531 crush map has
> features 1107558400 was 8705, adjusting msgr requires for mons
> 2017-11-17 04:57:38.569085 7f04c28ead00 0 osd.2 1531 crush map has
> features 1107558400, adjusting msgr requires for osds
> 2017-11-17 04:57:38.735392 7f04c28ead00 0 osd.2 1531 load_pgs
> 2017-11-17 04:57:38.740951 7f04c28ead00 -1 *** Caught signal (Aborted) **
> in thread 7f04c28ead00 thread_name:ceph-osd
>
> ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
> luminous (stable)
> 1: (()+0xa088b9) [0x5618d594b8b9]
> 2: (()+0x10330) [0x7f04c0e0d330]
> 3: (gsignal()+0x37) [0x7f04bfe2dc37]
> 4: (abort()+0x148) [0x7f04bfe31028]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f04c073c535]
> 6: (()+0x5e6d6) [0x7f04c073a6d6]
> 7: (()+0x5e703) [0x7f04c073a703]
> 8: (()+0x5e922) [0x7f04c073a922]
> 9: (object_stat_sum_t::decode(ceph::buffer::list::iterator&)+0x5cc)
> [0x5618d563ec1c]
> 10:
> (object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x54)
> [0x5618d5659e14]
> 11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x1d5)
> [0x5618d565a455]
> 12: (pg_info_t::decode(ceph::buffer::list::iterator&)+0x12a)
> [0x5618d566059a]
> 13: (PG::read_info(ObjectStore*, spg_t, coll_t const&,
> ceph::buffer::list&, pg_info_t&, PastIntervals&, unsigned char&)+0x231)
> [0x5618d54d1231]
> 14: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x7b)
> [0x5618d54d8c6b]
> 15: (OSD::load_pgs()+0x994) [0x5618d5435e74]
> 16: (OSD::init()+0x2127) [0x5618d544ddc7]
> 17: (main()+0x2ba8) [0x5618d5355798]
> 18: (__libc_start_main()+0xf5) [0x7f04bfe18f45]
> 19: (()+0x4b0826) [0x5618d53f3826]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
>
>
> Thank you for any help, the storage cluster is hosting crucial data.
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
