Hi, My OSD's were continuously crashing in cephx_verify_authorizer() while on Luminous v12.1.0 and v12.1.1, but the crashes stopped once I upgraded to v12.1.2.
Now however, one of my OSDs is continuing to crash. Looking closer, the crash reason is different reason and started with v12.1.1. I've been troubleshooting with the aid of http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/. I'm considering reweight to 0 and then redeploy that OSD from scratch, unless you can do a filesystem repair on bluestore/rocksdb somehow. Please advise. Data follows... Log: roger@osd3:~$ sudo journalctl -u ceph-osd@0 --no-pager ... Aug 02 10:38:47 osd3 systemd[1]: ceph-osd@0.service: Failed with result 'signal'. Aug 02 10:39:07 osd3 systemd[1]: ceph-osd@0.service: Service hold-off time over, scheduling restart. Aug 02 10:39:07 osd3 systemd[1]: Stopped Ceph object storage daemon osd.0. Aug 02 10:39:07 osd3 systemd[1]: Starting Ceph object storage daemon osd.0... Aug 02 10:39:07 osd3 systemd[1]: Started Ceph object storage daemon osd.0. Aug 02 10:39:07 osd3 ceph-osd[7413]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal Aug 02 10:40:48 osd3 ceph-osd[7413]: 2017-08-02 10:40:48.583063 7f5262cc3e00 -1 osd.0 25924 log_to_monitors {default=true} Aug 02 10:43:32 osd3 ceph-osd[7413]: *** Caught signal (Aborted) ** Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700 thread_name:tp_osd_tp Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc) Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964] Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390] Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 4: (KernelDevice::direct_read_unaligned(unsigned long, unsigned long, char*)+0x81) [0x5623f0a7cfc1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 5: (KernelDevice::read_random(unsigned long, unsigned long, char*, bool)+0x4f3) [0x5623f0a7da43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x4fa) [0x5623f0a4d9ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10] Aug 02 10:43:32 osd3 ceph-osd[7413]: 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf] Aug 02 10:43:32 osd3 ceph-osd[7413]: 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3) [0x5623f0e21383] Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6] Aug 02 10:43:32 osd3 ceph-osd[7413]: 11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8) [0x5623f0e13928] Aug 02 10:43:32 osd3 ceph-osd[7413]: 12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c] Aug 02 10:43:32 osd3 ceph-osd[7413]: 13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x5623f0e1c4a7] Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e] Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836] Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 17: (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09] Aug 02 10:43:32 osd3 ceph-osd[7413]: 18: (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182) [0x5623f0ec7ed2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb) [0x5623f0ec8c8b] Aug 02 10:43:32 osd3 ceph-osd[7413]: 20: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a] Aug 02 10:43:32 osd3 ceph-osd[7413]: 21: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x1170) [0x5623f093d250] Aug 02 10:43:32 osd3 ceph-osd[7413]: 22: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea] Aug 02 10:43:32 osd3 ceph-osd[7413]: 23: (PGBackend::objects_list_range(hobject_t const&, hobject_t const&, snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*, std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192) [0x5623f0700ef2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 24: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30] Aug 02 10:43:32 osd3 ceph-osd[7413]: 25: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec] Aug 02 10:43:32 osd3 ceph-osd[7413]: 27: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0] Aug 02 10:43:32 osd3 ceph-osd[7413]: 28: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5623f0ae44e4] Aug 02 10:43:32 osd3 ceph-osd[7413]: 29: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520] Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba] Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd] Aug 02 10:43:32 osd3 ceph-osd[7413]: 2017-08-02 10:43:32.251841 7f524861b700 -1 *** Caught signal (Aborted) ** Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700 thread_name:tp_osd_tp Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc) Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964] Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390] Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 4: (KernelDevice::direct_read_unaligned(unsigned long, unsigned long, char*)+0x81) [0x5623f0a7cfc1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 5: (KernelDevice::read_random(unsigned long, unsigned long, char*, bool)+0x4f3) [0x5623f0a7da43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x4fa) [0x5623f0a4d9ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10] Aug 02 10:43:32 osd3 ceph-osd[7413]: 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf] Aug 02 10:43:32 osd3 ceph-osd[7413]: 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3) [0x5623f0e21383] Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6] Aug 02 10:43:32 osd3 ceph-osd[7413]: 11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8) [0x5623f0e13928] Aug 02 10:43:32 osd3 ceph-osd[7413]: 12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c] Aug 02 10:43:32 osd3 ceph-osd[7413]: 13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x5623f0e1c4a7] Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e] Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836] Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 17: (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09] Aug 02 10:43:32 osd3 ceph-osd[7413]: 18: (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182) [0x5623f0ec7ed2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb) [0x5623f0ec8c8b] Aug 02 10:43:32 osd3 ceph-osd[7413]: 20: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a] Aug 02 10:43:32 osd3 ceph-osd[7413]: 21: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x1170) [0x5623f093d250] Aug 02 10:43:32 osd3 ceph-osd[7413]: 22: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea] Aug 02 10:43:32 osd3 ceph-osd[7413]: 23: (PGBackend::objects_list_range(hobject_t const&, hobject_t const&, snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*, std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192) [0x5623f0700ef2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 24: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30] Aug 02 10:43:32 osd3 ceph-osd[7413]: 25: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec] Aug 02 10:43:32 osd3 ceph-osd[7413]: 27: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0] Aug 02 10:43:32 osd3 ceph-osd[7413]: 28: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5623f0ae44e4] Aug 02 10:43:32 osd3 ceph-osd[7413]: 29: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520] Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba] Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd] Aug 02 10:43:32 osd3 ceph-osd[7413]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aug 02 10:43:32 osd3 ceph-osd[7413]: 0> 2017-08-02 10:43:32.251841 7f524861b700 -1 *** Caught signal (Aborted) ** Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700 thread_name:tp_osd_tp Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc) Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964] Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390] Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 4: (KernelDevice::direct_read_unaligned(unsigned long, unsigned long, char*)+0x81) [0x5623f0a7cfc1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 5: (KernelDevice::read_random(unsigned long, unsigned long, char*, bool)+0x4f3) [0x5623f0a7da43] Aug 02 10:43:32 osd3 ceph-osd[7413]: 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x4fa) [0x5623f0a4d9ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10] Aug 02 10:43:32 osd3 ceph-osd[7413]: 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf] Aug 02 10:43:32 osd3 ceph-osd[7413]: 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3) [0x5623f0e21383] Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6] Aug 02 10:43:32 osd3 ceph-osd[7413]: 11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8) [0x5623f0e13928] Aug 02 10:43:32 osd3 ceph-osd[7413]: 12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c] Aug 02 10:43:32 osd3 ceph-osd[7413]: 13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x5623f0e1c4a7] Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e] Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836] Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1] Aug 02 10:43:32 osd3 ceph-osd[7413]: 17: (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09] Aug 02 10:43:32 osd3 ceph-osd[7413]: 18: (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182) [0x5623f0ec7ed2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb) [0x5623f0ec8c8b] Aug 02 10:43:32 osd3 ceph-osd[7413]: 20: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a] Aug 02 10:43:32 osd3 ceph-osd[7413]: 21: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x1170) [0x5623f093d250] Aug 02 10:43:32 osd3 ceph-osd[7413]: 22: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea] Aug 02 10:43:32 osd3 ceph-osd[7413]: 23: (PGBackend::objects_list_range(hobject_t const&, hobject_t const&, snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*, std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192) [0x5623f0700ef2] Aug 02 10:43:32 osd3 ceph-osd[7413]: 24: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30] Aug 02 10:43:32 osd3 ceph-osd[7413]: 25: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca] Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec] Aug 02 10:43:32 osd3 ceph-osd[7413]: 27: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0] Aug 02 10:43:32 osd3 ceph-osd[7413]: 28: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5623f0ae44e4] Aug 02 10:43:32 osd3 ceph-osd[7413]: 29: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520] Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba] Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd] Aug 02 10:43:32 osd3 ceph-osd[7413]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Main process exited, code=killed, status=6/ABRT Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Unit entered failed state. Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Failed with result 'signal'. Aug 02 10:43:52 osd3 systemd[1]: ceph-osd@0.service: Service hold-off time over, scheduling restart. Aug 02 10:43:52 osd3 systemd[1]: Stopped Ceph object storage daemon osd.0. Aug 02 10:43:52 osd3 systemd[1]: Starting Ceph object storage daemon osd.0... Aug 02 10:43:52 osd3 systemd[1]: Started Ceph object storage daemon osd.0. Aug 02 10:43:52 osd3 ceph-osd[8322]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal roger@desktop:~$ ceph -s cluster: id: eea7b78c-b138-40fc-9f3e-3d77afb770f0 health: HEALTH_WARN 1 osds down 1 host (1 osds) down Degraded data redundancy: 43922/162834 objects degraded (26.973%), 300 pgs unclean, 305 pgs degraded 114 pgs not deep-scrubbed for 86400 155 pgs not scrubbed for 86400 10 slow requests are blocked > 32 sec services: mon: 3 daemons, quorum nuc1,nuc2,nuc3 mgr: nuc3(active), standbys: nuc2, nuc1 osd: 3 osds: 2 up, 3 in rgw: 1 daemon active data: pools: 19 pools, 372 pgs objects: 54278 objects, 71724 MB usage: 122 GB used, 27819 GB / 27941 GB avail pgs: 43922/162834 objects degraded (26.973%) 303 active+undersized+degraded 67 active+clean 2 active+recovery_wait+degraded roger@desktop:~$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 27.28679 root default -5 9.09560 host osd1 3 hdd 9.09560 osd.3 up 1.00000 1.00000 -6 9.09560 host osd2 4 hdd 9.09560 osd.4 up 1.00000 1.00000 -2 9.09560 host osd3 0 hdd 9.09560 osd.0 down 1.00000 1.00000 roger@desktop:~$ ceph mon versions { "ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)": 3 } roger@desktop:~$ ceph osd versions { "ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)": 2 } roger@osd3:~$ sudo ceph daemon osd.0 status { "cluster_fsid": "eea7b78c-b138-40fc-9f3e-3d77afb770f0", "osd_fsid": "bdb31a03-e381-4bf8-82e3-18916c838308", "whoami": 0, "state": "waiting_for_healthy", "oldest_map": 25389, "newest_map": 25938, "num_pgs": 372 } roger@desktop:~$ ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 27941G 27819G 122G 0.44 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS default.rgw.rgw.gc 70 0 0 8807G 0 default.rgw.buckets.non-ec 83 0 0 8807G 43 default.rgw.control 85 0 0 8807G 8 default.rgw.data.root 86 15601 0 8807G 49 default.rgw.gc 87 0 0 8807G 32 default.rgw.lc 88 0 0 8807G 32 default.rgw.log 89 0 0 8807G 144 default.rgw.users.uid 90 3346 0 8807G 14 default.rgw.users.email 91 100 0 8807G 7 default.rgw.users.keys 92 100 0 8807G 7 default.rgw.buckets.index 93 0 0 8807G 39 default.rgw.intent-log 95 0 0 8807G 0 default.rgw.meta 96 0 0 8807G 0 default.rgw.usage 97 0 0 8807G 0 default.rgw.users.swift 98 39 0 8807G 4 default.rgw.buckets.extra 99 0 0 8807G 0 .rgw.root 100 1681 0 8807G 4 default.rgw.reshard 101 0 0 8807G 17 default.rgw.buckets.data 103 71724M 0.40 17614G 53878
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
