Hi,
I had originally asked the question on the pull request here:
https://github.com/ceph/ceph/pull/44131 and was asked to continue the
discussion on this list.
Last night I upgraded my cluster from 16.2.5 (I think) to 16.2.7.
Unfortunately, thinking this was a minor patch, I failed to read the
upgrade instructions so I did not set "mon_mds_skip_sanity = true" before
upgrading and I'm not using cephadm. As a result, the monitor on my first
node crashed after upgrade. I then discovered the recommendation, set the
flag, and the monitor started. Once the monitor was up, I removed the flag
from the config, restarted it, and it crashed again. Thinking maybe the
upgrade wasn't fully complete _or_ I had to upgrade all of my monitors
first, I went ahead and upgraded the rest of the cluster with the flag set
and let it sit overnight.
Today I attempted to remove the flag on one of my monitors and discovered
that it still crashes. Here's my `ceph.conf` just in case that's useful:
```ini
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.11.0.0/16
filestore_xattr_use_omap = true
fsid = redacted
mon_allow_pool_delete = true
mon_host = 10.10.0.1, 10.10.0.2, 10.10.0.3
mon_pg_warn_min_per_osd = 10
osd_journal_size = 5120
osd_pool_default_min_size = 1
public_network = 10.10.0.0/16
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[mds.0]
host = ceph1
mds_standby_for_name = pve
[mds.1]
host = ceph2
mds_standby_for_name = pve
[mds.2]
host = ceph3
mds_standby_for_name = pve
[mds.3]
host = ceph4
mds_standby_for_name = pve
[mds.4]
host = ceph5
mds standby for name = pve
[mon.0]
host = ceph2
mon_mds_skip_sanity = true
[mon.1]
host = ceph3
mon_mds_skip_sanity = true
[mon.2]
host = ceph4
mon_mds_skip_sanity = true
```
I've confirmed that my monitors are on 16.2.7:
```json
root@ceph2:/var/log/ceph# sudo ceph mon versions
{
"ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific
(stable)": 3
}
```
Here's a crash log from one of the monitors:
```
2021-12-23T11:33:16.689-0500 7f08d5146580 0 starting mon.0 rank 0 at
public addrs [v2:10.10.0.1:3300/0,v1:10.10.0.1:6789/0] at bind addrs [v2:
10.10.0.1:3300/0,v1:10.10.0.1:6789/0] mon_data /var/lib/ceph/mon/ceph-0
fsid --redacted--
2021-12-23T11:33:16.689-0500 7f08d5146580 1 mon.0@-1(???) e14 preinit fsid
--redacted--
2021-12-23T11:33:16.689-0500 7f08d5146580 0 mon.0@-1(???).mds e3719 new map
2021-12-23T11:33:16.689-0500 7f08d5146580 0 mon.0@-1(???).mds e3719
print_map
e3719
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses
inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 4
Filesystem 'clusterfs-capacity' (4)
fs_name clusterfs-capacity
epoch 3719
flags 12
created 2021-04-25T20:22:33.434467-0400
<http://voice.google.com/calls?a=nc,%2B14344670400>
modified 2021-12-23T04:00:59.962394-0500
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 265418
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=448184627}
failed
damaged
stopped
data_pools [18]
metadata_pool 19
inline_data disabled
balancer
standby_count_wanted 1
[mds.ceph2{0:448184627} state up:active seq 5103 addr [v2:
10.10.0.1:6800/4098373857,v1:10.10.0.1:6801/4098373857] compat
{c=[1],r=[1],i=[7ff]}]
Filesystem 'clusterfs-performance' (5)
fs_name clusterfs-performance
epoch 3710
flags 12
created 2021-09-07T19:50:05.174359-0400
modified 2021-12-22T21:40:43.813907-0500
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 261954
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=448071169}
failed
damaged
stopped
data_pools [28]
metadata_pool 27
inline_data disabled
balancer
standby_count_wanted 1
[mds.ceph4{0:448071169} state up:active seq 2852 addr [v2:
10.10.0.3:6800/368534096,v1:10.10.0.3:6801/368534096] compat
{c=[1],r=[1],i=[77f]}]
Standby daemons:
[mds.ceph3{-1:448024430} state up:standby seq 1 addr [v2:
10.10.0.2:6800/535990846,v1:10.10.0.2:6801/535990846] compat
{c=[1],r=[1],i=[77f]}]
[mds.ceph5{-1:448169888} state up:standby seq 1 addr [v2:
10.10.0.20:6800/1349962329,v1:10.10.0.20:6801/1349962329] compat
{c=[1],r=[1],i=[77f]}]
[mds.ceph1{-1:448321175} state up:standby seq 1 addr [v2:
10.10.0.32:6800/881362738,v1:10.10.0.32:6801/881362738] compat
{c=[1],r=[1],i=[7ff]}]
2021-12-23T11:33:16.693-0500 7f08d5146580 -1 ./src/mds/FSMap.cc: In
function 'void FSMap::sanity(bool) const' thread 7f08d5146580 time
2021-12-23T11:33:16.692037-0500
./src/mds/FSMap.cc: 868: FAILED
ceph_assert(info.compat.writeable(fs->mds_map.compat))
ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x124) [0x7f08d6022046]
2: /usr/lib/ceph/libceph-common.so.2(+0x2511d1) [0x7f08d60221d1]
3: (FSMap::sanity(bool) const+0x4b6) [0x7f08d65823b6]
4: (MDSMonitor::update_from_paxos(bool*)+0x4aa) [0x55857ba2da6a]
5: (Monitor::refresh_from_paxos(bool*)+0x163) [0x55857b7f8f03]
6: (Monitor::preinit()+0x9af) [0x55857b82510f]
7: main()
8: __libc_start_main()
9: _start()
2021-12-23T11:33:16.697-0500 7f08d5146580 -1 *** Caught signal (Aborted) **
in thread 7f08d5146580 thread_name:ceph-mon
ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific
(stable)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f08d5b02140]
2: gsignal()
3: abort()
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x16e) [0x7f08d6022090]
5: /usr/lib/ceph/libceph-common.so.2(+0x2511d1) [0x7f08d60221d1]
6: (FSMap::sanity(bool) const+0x4b6) [0x7f08d65823b6]
7: (MDSMonitor::update_from_paxos(bool*)+0x4aa) [0x55857ba2da6a]
8: (Monitor::refresh_from_paxos(bool*)+0x163) [0x55857b7f8f03]
9: (Monitor::preinit()+0x9af) [0x55857b82510f]
10: main()
11: __libc_start_main()
12: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- begin dump of recent events ---
-82> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command assert hook 0x55857d76e610
-81> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command abort hook 0x55857d76e610
-80> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command leak_some_memory hook 0x55857d76e610
-79> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perfcounters_dump hook 0x55857d76e610
-78> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command 1 hook 0x55857d76e610
-77> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perf dump hook 0x55857d76e610
-76> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perfcounters_schema hook 0x55857d76e610
-75> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perf histogram dump hook 0x55857d76e610
-74> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command 2 hook 0x55857d76e610
-73> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perf schema hook 0x55857d76e610
-72> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perf histogram schema hook 0x55857d76e610
-71> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command perf reset hook 0x55857d76e610
-70> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config show hook 0x55857d76e610
-69> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config help hook 0x55857d76e610
-68> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config set hook 0x55857d76e610
-67> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config unset hook 0x55857d76e610
-66> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config get hook 0x55857d76e610
-65> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config diff hook 0x55857d76e610
-64> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command config diff get hook 0x55857d76e610
-63> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command injectargs hook 0x55857d76e610
-62> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command log flush hook 0x55857d76e610
-61> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command log dump hook 0x55857d76e610
-60> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command log reopen hook 0x55857d76e610
-59> 2021-12-23T11:33:16.545-0500 7f08d5146580 5 asok(0x55857d820000)
register_command dump_mempools hook 0x55857e418068
-58> 2021-12-23T11:33:16.553-0500 7f08d5146580 0 set uid:gid to
64045:64045 (ceph:ceph)
-57> 2021-12-23T11:33:16.553-0500 7f08d5146580 0 ceph version 16.2.7
(f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable), process
ceph-mon, pid 704498
-56> 2021-12-23T11:33:16.553-0500 7f08d5146580 0 pidfile_write: ignore
empty --pid-file
-55> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
init /var/run/ceph/ceph-mon.0.asok
-54> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
bind_and_listen /var/run/ceph/ceph-mon.0.asok
-53> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
register_command 0 hook 0x55857d76dc80
-52> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
register_command version hook 0x55857d76dc80
-51> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
register_command git_version hook 0x55857d76dc80
-50> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
register_command help hook 0x55857d76e520
-49> 2021-12-23T11:33:16.553-0500 7f08d5146580 5 asok(0x55857d820000)
register_command get_command_descriptions hook 0x55857d76e500
-48> 2021-12-23T11:33:16.553-0500 7f08d3f26700 5 asok(0x55857d820000)
entry start
-47> 2021-12-23T11:33:16.573-0500 7f08d5146580 0 load: jerasure load:
lrc load: isa
-46> 2021-12-23T11:33:16.625-0500 7f08d5146580 1 leveldb: Recovering
log #19956085
-45> 2021-12-23T11:33:16.629-0500 7f08d5146580 1 leveldb: Level-0 table
#19956112: started
-44> 2021-12-23T11:33:16.669-0500 7f08d5146580 1 leveldb: Level-0 table
#19956112: 2636812 bytes OK
-43> 2021-12-23T11:33:16.685-0500 7f08d5146580 1 leveldb: Delete type=0
#19956085
-42> 2021-12-23T11:33:16.685-0500 7f08d5146580 1 leveldb: Delete type=3
#19934065
-41> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding auth protocol: cephx
-40> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding auth protocol: cephx
-39> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding auth protocol: cephx
-38> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-37> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-36> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-35> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-34> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-33> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-32> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-31> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-30> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-29> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-28> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: crc
-27> 2021-12-23T11:33:16.685-0500 7f08d5146580 5
AuthRegistry(0x55857ea22140) adding con mode: secure
-26> 2021-12-23T11:33:16.685-0500 7f08d5146580 2 auth: KeyRing::load:
loaded key file /var/lib/ceph/mon/ceph-0/keyring
-25> 2021-12-23T11:33:16.689-0500 7f08d5146580 0 starting mon.0 rank 0
at public addrs [v2:10.10.0.1:3300/0,v1:10.10.0.1:6789/0] at bind addrs [v2:
10.10.0.1:3300/0,v1:10.10.0.1:6789/0] mon_data /var/lib/ceph/mon/ceph-0
fsid --redacted--
-24> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding auth protocol: cephx
-23> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding auth protocol: cephx
-22> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding auth protocol: cephx
-21> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-20> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-19> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-18> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-17> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-16> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-15> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-14> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-13> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-12> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-11> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: crc
-10> 2021-12-23T11:33:16.689-0500 7f08d5146580 5
AuthRegistry(0x55857ea22a40) adding con mode: secure
-9> 2021-12-23T11:33:16.689-0500 7f08d5146580 2 auth: KeyRing::load:
loaded key file /var/lib/ceph/mon/ceph-0/keyring
-8> 2021-12-23T11:33:16.689-0500 7f08d5146580 5 adding auth protocol:
cephx
-7> 2021-12-23T11:33:16.689-0500 7f08d5146580 5 adding auth protocol:
cephx
-6> 2021-12-23T11:33:16.689-0500 7f08d5146580 10 log_channel(cluster)
update_config to_monitors: true to_syslog: false syslog_facility: daemon
prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-5> 2021-12-23T11:33:16.689-0500 7f08d5146580 10 log_channel(audit)
update_config to_monitors: true to_syslog: false syslog_facility: local0
prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-4> 2021-12-23T11:33:16.689-0500 7f08d5146580 1 mon.0@-1(???) e14
preinit fsid --redacted--
-3> 2021-12-23T11:33:16.689-0500 7f08d5146580 0 mon.0@-1(???).mds
e3719 new map
-2> 2021-12-23T11:33:16.689-0500 7f08d5146580 0 mon.0@-1(???).mds
e3719 print_map
e3719
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses
inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 4
Filesystem 'clusterfs-capacity' (4)
fs_name clusterfs-capacity
epoch 3719
flags 12
created 2021-04-25T20:22:33.434467-0400
modified 2021-12-23T04:00:59.962394-0500
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 265418
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=448184627}
failed
damaged
stopped
data_pools [18]
metadata_pool 19
inline_data disabled
balancer
standby_count_wanted 1
[mds.ceph2{0:448184627} state up:active seq 5103 addr [v2:
10.10.0.1:6800/4098373857,v1:10.10.0.1:6801/4098373857] compat
{c=[1],r=[1],i=[7ff]}]
Filesystem 'clusterfs-performance' (5)
fs_name clusterfs-performance
epoch 3710
flags 12
created 2021-09-07T19:50:05.174359-0400
modified 2021-12-22T21:40:43.813907-0500
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 261954
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=448071169}
failed
damaged
stopped
data_pools [28]
metadata_pool 27
inline_data disabled
balancer
standby_count_wanted 1
[mds.ceph4{0:448071169} state up:active seq 2852 addr [v2:
10.10.0.3:6800/368534096,v1:10.10.0.3:6801/368534096] compat
{c=[1],r=[1],i=[77f]}]
Standby daemons:
[mds.ceph3{-1:448024430} state up:standby seq 1 addr [v2:
10.10.0.2:6800/535990846,v1:10.10.0.2:6801/535990846] compat
{c=[1],r=[1],i=[77f]}]
[mds.ceph5{-1:448169888} state up:standby seq 1 addr [v2:
10.10.0.20:6800/1349962329,v1:10.10.0.20:6801/1349962329] compat
{c=[1],r=[1],i=[77f]}]
[mds.ceph1{-1:448321175} state up:standby seq 1 addr [v2:
10.10.0.32:6800/881362738,v1:10.10.0.32:6801/881362738] compat
{c=[1],r=[1],i=[7ff]}]
-1> 2021-12-23T11:33:16.693-0500 7f08d5146580 -1 ./src/mds/FSMap.cc: In
function 'void FSMap::sanity(bool) const' thread 7f08d5146580 time
2021-12-23T11:33:16.692037-0500
./src/mds/FSMap.cc: 868: FAILED
ceph_assert(info.compat.writeable(fs->mds_map.compat))
ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x124) [0x7f08d6022046]
2: /usr/lib/ceph/libceph-common.so.2(+0x2511d1) [0x7f08d60221d1]
3: (FSMap::sanity(bool) const+0x4b6) [0x7f08d65823b6]
4: (MDSMonitor::update_from_paxos(bool*)+0x4aa) [0x55857ba2da6a]
5: (Monitor::refresh_from_paxos(bool*)+0x163) [0x55857b7f8f03]
6: (Monitor::preinit()+0x9af) [0x55857b82510f]
7: main()
8: __libc_start_main()
9: _start()
0> 2021-12-23T11:33:16.697-0500 7f08d5146580 -1 *** Caught signal
(Aborted) **
in thread 7f08d5146580 thread_name:ceph-mon
ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific
(stable)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f08d5b02140]
2: gsignal()
3: abort()
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x16e) [0x7f08d6022090]
5: /usr/lib/ceph/libceph-common.so.2(+0x2511d1) [0x7f08d60221d1]
6: (FSMap::sanity(bool) const+0x4b6) [0x7f08d65823b6]
7: (MDSMonitor::update_from_paxos(bool*)+0x4aa) [0x55857ba2da6a]
8: (Monitor::refresh_from_paxos(bool*)+0x163) [0x55857b7f8f03]
9: (Monitor::preinit()+0x9af) [0x55857b82510f]
10: main()
11: __libc_start_main()
12: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
139675892344576 / admin_socket
139675911349632 / ceph-mon
max_recent 10000
max_new 10000
log_file /var/log/ceph/ceph-mon.0.log
--- end dump of recent events ---
```
Perhaps this is a different sanity check failing that really should fail
and needs to be fixed in my cluster?
Ilya Kogan
w: github.com/ikogan e: iko...@mythicnet.org
<http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
