I found some logs says osd.91 is down. I think that should be same for osd.9 I am not sure what will cause the OSD process treated by peers as down.
2018-01-30 06:39:33.767747 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424396 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 1 entries (version 424396) 2018-01-30 06:39:37.623969 7f2409f07700 0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec) 2018-01-30 06:39:37.751100 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424397 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 1 entries (version 424397) 2018-01-30 06:39:53.535500 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424398 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424398) 2018-01-30 06:40:09.251953 7f2409f07700 0 mon.ubuntuser8@0(leader).data_health(33) update_stats avail 86% total 806 GB, used 70945 MB, avail 696 GB 2018-01-30 06:40:11.668097 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec (REQUEST_SLOW) 2018-01-30 06:40:11.794779 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424399 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 1 entries (version 424399) 2018-01-30 06:40:18.599837 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.0 10.1.248.1:6871/3727 is reporting failure:1 2018-01-30 06:40:18.599861 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.0 10.1.248.1:6871/3727 2018-01-30 06:40:18.783394 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424400 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424400) 2018-01-30 06:40:19.290525 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.61 10.1.248.3:6842/3728 is reporting failure:1 2018-01-30 06:40:19.290546 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.61 10.1.248.3:6842/3728 2018-01-30 06:40:19.316574 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.39 10.1.248.2:6822/3732 is reporting failure:1 2018-01-30 06:40:19.316592 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.39 10.1.248.2:6822/3732 2018-01-30 06:40:19.361317 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.44 10.1.248.2:6800/3683 is reporting failure:1 2018-01-30 06:40:19.361335 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.44 10.1.248.2:6800/3683 2018-01-30 06:40:19.918065 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424401 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424401) 2018-01-30 06:40:20.075119 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.41 10.1.248.2:6830/3677 is reporting failure:1 2018-01-30 06:40:20.075141 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.41 10.1.248.2:6830/3677 2018-01-30 06:40:20.493001 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.56 10.1.248.3:6800/3713 is reporting failure:1 2018-01-30 06:40:20.493023 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.56 10.1.248.3:6800/3713 2018-01-30 06:40:20.787217 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.48 10.1.248.3:6815/3670 is reporting failure:1 2018-01-30 06:40:20.787238 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.48 10.1.248.3:6815/3670 2018-01-30 06:40:20.982180 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.21 10.1.248.1:6800/3711 is reporting failure:1 2018-01-30 06:40:20.982198 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.21 10.1.248.1:6800/3711 2018-01-30 06:40:20.984336 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424402 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424402) 2018-01-30 06:40:21.152463 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.13 10.1.248.1:6842/3700 is reporting failure:1 2018-01-30 06:40:21.152480 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.13 10.1.248.1:6842/3700 2018-01-30 06:40:22.083900 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.38 10.1.248.2:6876/3764 is reporting failure:1 2018-01-30 06:40:22.083920 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.38 10.1.248.2:6876/3764 2018-01-30 06:40:22.117579 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424403 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424403) 2018-01-30 06:40:22.179713 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.89 10.1.248.3:6817/3679 is reporting failure:1 2018-01-30 06:40:22.179737 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.89 10.1.248.3:6817/3679 2018-01-30 06:40:22.318619 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.69 10.1.248.3:6812/3716 is reporting failure:1 2018-01-30 06:40:22.318644 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.69 10.1.248.3:6812/3716 2018-01-30 06:40:22.698652 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.53 10.1.248.3:6843/3701 is reporting failure:1 2018-01-30 06:40:22.698673 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.53 10.1.248.3:6843/3701 2018-01-30 06:40:22.842870 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.23 10.1.248.1:6831/3663 is reporting failure:1 2018-01-30 06:40:22.842886 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.23 10.1.248.1:6831/3663 2018-01-30 06:40:23.183964 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424404 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 0 entries (version 424404) 2018-01-30 06:40:23.259187 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.25 10.1.248.2:6825/3713 is reporting failure:1 2018-01-30 06:40:23.259203 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.25 10.1.248.2:6825/3713 2018-01-30 06:40:23.281029 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.62 10.1.248.3:6872/3735 is reporting failure:1 2018-01-30 06:40:23.281041 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.62 10.1.248.3:6872/3735 2018-01-30 06:40:23.771737 7f2407702700 1 mon.ubuntuser8@0(leader).osd e3016 prepare_failure osd.90 10.1.248.4:6830/3552 from osd.92 10.1.248.4:6801/3581 is reporting failure:1 2018-01-30 06:40:23.771749 7f2407702700 0 log_channel(cluster) log [DBG] : osd.90 10.1.248.4:6830/3552 reported failed by osd.92 10.1.248.4:6801/3581 2018-01-30 06:40:24.317177 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424405 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 2 entries (version 424405) 2018-01-30 06:40:31.694701 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec (REQUEST_SLOW) 2018-01-30 06:40:31.830914 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424406 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 1 entries (version 424406) 2018-01-30 06:40:37.702118 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec (REQUEST_SLOW) 2018-01-30 06:40:37.845087 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424407 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 1 entries (version 424407) 2018-01-30 06:40:38.730131 7f2409f07700 1 mon.ubuntuser8@0(leader).osd e3016 we have enough reporters to mark osd.90 down 2018-01-30 06:40:38.730162 7f2409f07700 0 log_channel(cluster) log [INF] : osd.90 failed (root=default,host=ubuntuser4) (4 reporters from different host after 34.958423 >= grace 31.094427) 2018-01-30 06:40:38.730952 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN) 2018-01-30 06:40:38.795202 7f2402ef9700 1 mon.ubuntuser8@0(leader).osd e3017 e3017: 96 total, 92 up, 93 in 2018-01-30 06:40:38.830011 7f2402ef9700 0 log_channel(cluster) log [DBG] : osdmap e3017: 96 total, 92 up, 93 in 2018-01-30 06:40:38.830254 7f2407702700 0 mon.ubuntuser8@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 17} v 0) v1 2018-01-30 06:40:38.830299 7f2407702700 0 log_channel(audit) log [DBG] : from='client.164108 10.1.248.8:0/3388257888' entity='mgr.ubuntuser8' cmd=[{"prefix": "osd metadata", "id": 17}]: dispatch 2018-01-30 06:40:38.911919 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424408 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 2 entries (version 424408) 2018-01-30 06:40:39.861824 7f2402ef9700 1 mon.ubuntuser8@0(leader).osd e3018 e3018: 96 total, 92 up, 93 in 2018-01-30 06:40:39.895967 7f2402ef9700 0 log_channel(cluster) log [DBG] : osdmap e3018: 96 total, 92 up, 93 in 2018-01-30 06:40:39.896243 7f2407702700 0 mon.ubuntuser8@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 17} v 0) v1 2018-01-30 06:40:39.896291 7f2407702700 0 log_channel(audit) log [DBG] : from='client.164108 10.1.248.8:0/3388257888' entity='mgr.ubuntuser8' cmd=[{"prefix": "osd metadata", "id": 17}]: dispatch 2018-01-30 06:40:39.911825 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 7 pgs peering, 10 pgs stale (PG_AVAILABILITY) 2018-01-30 06:40:39.911883 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 7 pgs unclean (PG_DEGRADED) 2018-01-30 06:40:40.928452 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424409 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 2 entries (version 424409) 2018-01-30 06:40:43.544073 7f2402ef9700 1 mon.ubuntuser8@0(leader).log v424410 check_sub sending message to client.164108 10.1.248.8:0/3388257888 with 5 entries (version 424410) 2018-01-30 06:40:43.761429 7f2409f07700 0 log_channel(cluster) log [WRN] : Health check update: 5 slow requests are blocked > 32 sec (REQUEST_SLOW) 2018-01-30 16:07 GMT+08:00 blackpiglet J. <blackpigletbr...@gmail.com>: > Guys, > > We had set up a five nodes Ceph cluster. Four are OSD servers and the > other one is MON and MGR. > Recently, during RGW stability test, RGW default > pool: default.rgw.buckets.data is accidentally written full. As a result, > RGW is stuck. We don't know the exact steps to recover, then we deleted all > default RGW pools directly. After RGW restarted, all pools are back. > > Then I found 4 OSD processes are down. I am not whether this is related to > our RGW pool delete operation. > > This a the log I think may be useful. The full version is in the > attachment. > > Any help is appreciated. Thanks in advance. > > > 2018-01-29 20:17:48.896544 7fc111122700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fc0fb912700' had timed out after 15 > 2018-01-29 20:17:52.360069 7fc110183700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fc0fb912700' had timed out after 15 > 2018-01-29 20:17:57.360192 7fc110183700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fc0fb912700' had timed out after 15 > 2018-01-29 20:18:02.360315 7fc110183700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7fc0fb912700' had timed out after 15 > 2018-01-29 20:18:06.029867 7fc0fb912700 1 heartbeat_map reset_timeout > 'OSD::osd_op_tp thread 0x7fc0fb912700' had timed out after 15 > 2018-01-29 20:18:06.030576 7fc0fc914700 1 osd.9 pg_epoch: 2481 pg[6.18( v > 2276'10 (0'0,2276'10] local-lis/les=2479/2480 n=3 ec=704/704 lis/c > 2479/2470 les/c/f 2480/2471/2312 2481/2481/2476) [9,91,51] r=0 lpr=2481 > pi=[2470,2481)/2 luod= > 0'0 crt=2276'10 lcod 2151'9 mlcod 0'0 active] start_peering_interval up > [9,51] -> [9,91,51], acting [9,51] -> [9,91,51], acting_primary 9 -> 9, > up_primary 9 -> 9, role 0 -> 0, features acting 2305244844532236283 > upacting 230524484453223 > 6283 > 2018-01-29 20:18:06.030774 7fc0fc914700 1 osd.9 pg_epoch: 2481 pg[6.18( v > 2276'10 (0'0,2276'10] local-lis/les=2479/2480 n=3 ec=704/704 lis/c > 2479/2470 les/c/f 2480/2471/2312 2481/2481/2476) [9,91,51] r=0 lpr=2481 > pi=[2470,2481)/2 crt=2 > 276'10 lcod 2151'9 mlcod 0'0 unknown] state<Start>: transitioning to > Primary > 2018-01-29 20:18:06.031236 7fc0fc914700 1 osd.9 pg_epoch: 2481 pg[6.1d( v > 2316'568 (0'0,2316'568] local-lis/les=2479/2480 n=2 ec=704/704 lis/c > 2479/2470 les/c/f 2480/2471/2312 2481/2481/2481) [91,79,9] r=2 lpr=2481 > pi=[2470,2481)/2 luo > d=0'0 crt=2316'568 lcod 2312'567 active] start_peering_interval up [79,9] > -> [91,79,9], acting [79,9] -> [91,79,9], acting_primary 79 -> 91, > up_primary 79 -> 91, role 1 -> 2, features acting 2305244844532236283 > upacting 2305244844532236 > 283 > 2018-01-29 20:18:06.031324 7fc0fc914700 1 osd.9 pg_epoch: 2481 pg[6.1d( v > 2316'568 (0'0,2316'568] local-lis/les=2479/2480 n=2 ec=704/704 lis/c > 2479/2470 les/c/f 2480/2471/2312 2481/2481/2481) [91,79,9] r=2 lpr=2481 > pi=[2470,2481)/2 crt > =2316'568 lcod 2312'567 unknown NOTIFY] state<Start>: transitioning to > Stray > 2018-01-29 20:18:06.032167 7fc112124700 0 -- 10.1.248.1:6850/6003656 >> > 10.1.248.4:6825/1003558 conn(0x556089bf4800 :6850 > s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH > pgs=5245 cs=1 l=0).handle_connect_reply connect got RESETSESSION > 2018-01-29 20:18:06.032184 7fc111923700 0 -- 10.1.248.1:6850/6003656 >> > 10.1.248.3:6827/3726 conn(0x5560c0c06000 :-1 > s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH > pgs=6734 cs=1 l=0).handle_connect_reply connect got RESETSESSION > 2018-01-29 20:18:06.053899 7fc103121700 0 log_channel(cluster) log [WRN] > : Monitor daemon marked osd.9 down, but it is still running > 2018-01-29 20:18:06.053917 7fc103121700 0 log_channel(cluster) log [DBG] > : map e2485 wrongly marked me down at e2483 > 2018-01-29 20:18:06.053922 7fc103121700 0 osd.9 2485 _committed_osd_maps > marked down 6 > osd_max_markdown_count 5 in last 600.000000 seconds, > shutting down > 2018-01-29 20:18:06.053927 7fc103121700 1 osd.9 2485 > start_waiting_for_healthy > 2018-01-29 20:18:06.064250 7fc0fc914700 1 osd.9 pg_epoch: 2483 pg[10.53( > empty local-lis/les=2476/2477 n=0 ec=1815/1815 lis/c 2476/2476 les/c/f > 2477/2477/2312 2483/2483/2483) [] r=-1 lpr=2483 pi=[2476,2483)/1 crt=0'0 > active] start_peer > ing_interval up [9] -> [], acting [9] -> [], acting_primary 9 -> -1, > up_primary 9 -> -1, role 0 -> -1, features acting 2305244844532236283 > upacting 2305244844532236283 > 2018-01-29 20:18:06.064258 7fc0fd115700 1 osd.9 pg_epoch: 2483 pg[3.16( v > 2284'372 (0'0,2284'372] local-lis/les=2476/2477 n=2 ec=1758/130 lis/c > 2476/2476 les/c/f 2477/2477/2312 2483/2483/1760) [33,85] r=-1 lpr=2483 > pi=[2476,2483)/1 luo > d=0'0 crt=2284'372 lcod 2284'371 active] start_peering_interval up > [33,9,85] -> [33,85], acting [33,9,85] -> [33,85], acting_primary 33 -> 33, > up_primary 33 -> 33, role 1 -> -1, features acting 2305244844532236283 > upacting 2305244844532 > 236283 > 2018-01-29 20:18:06.064493 7fc0fc914700 1 osd.9 pg_epoch: 2485 pg[10.53( > empty local-lis/les=2476/2477 n=0 ec=1815/1815 lis/c 2476/2476 les/c/f > 2477/2477/2312 2483/2483/2483) [] r=-1 lpr=2483 pi=[2476,2483)/1 crt=0'0 > unknown NOTIFY] st > ate<Start>: transitioning to Stray > 2018-01-29 20:18:06.064545 7fc0fd115700 1 osd.9 pg_epoch: 2485 pg[3.16( v > 2284'372 (0'0,2284'372] local-lis/les=2476/2477 n=2 ec=1758/130 lis/c > 2476/2476 les/c/f 2477/2477/2312 2483/2483/1760) [33,85] r=-1 lpr=2483 > pi=[2476,2483)/1 crt > =2284'372 lcod 2284'371 unknown NOTIFY] state<Start>: transitioning to > Stray > 2018-01-29 20:18:06.064886 7fc0fc914700 1 osd.9 pg_epoch: 2483 pg[10.3f( > empty local-lis/les=2476/2477 n=0 ec=1815/1815 lis/c 2476/2476 les/c/f > 2477/2477/2312 2483/2483/2483) [] r=-1 lpr=2483 pi=[2476,2483)/1 crt=0'0 > active] start_peer > ing_interval up [9] -> [], acting [9] -> [], acting_primary 9 -> -1, > up_primary 9 -> -1, role 0 -> -1, features acting 2305244844532236283 > upacting 2305244844532236283 > 2018-01-29 20:18:06.064972 7fc0fd115700 1 osd.9 pg_epoch: 2483 pg[6.42( v > 2298'367 (0'0,2298'367] local-lis/les=2476/2477 n=1 ec=1753/704 lis/c > 2476/2476 les/c/f 2477/2477/2312 2483/2483/2367) [24,84] r=-1 lpr=2483 > pi=[2476,2483)/1 luo > d=0'0 crt=2298'367 lcod 2298'366 active] start_peering_interval up > [24,84,9] -> [24,84], acting [24,84,9] -> [24,84], acting_primary 24 -> 24, > up_primary 24 -> 24, role 2 -> -1, features acting 2305244844532236283 > upacting 2305244844532 > 236283 > 2018-01-29 20:18:06.065091 7fc0fc914700 1 osd.9 pg_epoch: 2485 pg[10.3f( > empty local-lis/les=2476/2477 n=0 ec=1815/1815 lis/c 2476/2476 les/c/f > 2477/2477/2312 2483/2483/2483) [] r=-1 lpr=2483 pi=[2476,2483)/1 crt=0'0 > unknown NOTIFY] st > ate<Start>: transitioning to Stray > > 2018-01-29 20:18:06.072202 7fc0fc914700 1 osd.9 pg_epoch: 2483 pg[5.3c( v > 2312'24 (0'0,2312'24] local-lis/les=2476/2477 n=1 ec=1748/700 lis/c > 2476/2476 les/c/f 2477/2477/2312 2483/2483/2483) [32,66] r=-1 lpr=2483 > pi=[2476,2483)/1 luod= > 0'0 crt=2312'24 lcod 2312'23 active] start_peering_interval up [9,32,66] > -> [32,66], acting [9,32,66] -> [32,66], acting_primary 9 -> 32, up_primary > 9 -> 32, role 0 -> -1, features acting 2305244844532236283 upacting > 2305244844532236283 > 2018-01-29 20:18:06.072324 7fc0fc914700 1 osd.9 pg_epoch: 2485 pg[5.3c( v > 2312'24 (0'0,2312'24] local-lis/les=2476/2477 n=1 ec=1748/700 lis/c > 2476/2476 les/c/f 2477/2477/2312 2483/2483/2483) [32,66] r=-1 lpr=2483 > pi=[2476,2483)/1 crt=2 > 312'24 lcod 2312'23 unknown NOTIFY] state<Start>: transitioning to Stray > 2018-01-29 20:18:06.072463 7fc103121700 0 osd.9 2485 _committed_osd_maps > shutdown OSD via async signal > 2018-01-29 20:18:06.072570 7fc0f4904700 -1 Fail to open '/proc/0/cmdline' > error = (2) No such file or directory > 2018-01-29 20:18:06.072597 7fc0f4904700 -1 received signal: Interrupt > from PID: 0 task name: <unknown> UID: 0 > 2018-01-29 20:18:06.072603 7fc0f4904700 -1 osd.9 2485 *** Got signal > Interrupt *** > 2018-01-29 20:18:06.072609 7fc0f4904700 0 osd.9 2485 prepare_to_stop > starting shutdown > 2018-01-29 20:18:06.072614 7fc0f4904700 -1 osd.9 2485 shutdown > 2018-01-29 20:18:06.429238 7fc10d936700 0 log_channel(cluster) log [WRN] > : 2 slow requests, 2 included below; oldest blocked for > 55.682037 secs > 2018-01-29 20:18:06.429251 7fc10d936700 0 log_channel(cluster) log [WRN] > : slow request 55.682037 seconds old, received at 2018-01-29 > 20:17:10.747125: pg_notify((query:2482 sent:2482 6.18( v 2276'10 > (0'0,2276'10] local-lis/les=2473/247 > 4 n=3 ec=704/704 lis/c 2473/2470 les/c/f 2474/2471/2312 > 2481/2481/2476))=([2470,2480] intervals=([2473,2475] acting > 51,91),([2479,2480] acting 9,51)) epoch 2482) currently wait for new map > 2018-01-29 20:18:06.429265 7fc10d936700 0 log_channel(cluster) log [WRN] > : slow request 55.130508 seconds old, received at 2018-01-29 > 20:17:11.298655: pg_notify((query:2482 sent:2482 6.18( v 2276'10 > (0'0,2276'10] local-lis/les=2479/248 > 0 n=3 ec=704/704 lis/c 2479/2470 les/c/f 2480/2471/2312 > 2481/2481/2476))=([2470,2480] intervals=([2473,2475] acting > 51,91),([2479,2480] acting 9,51)) epoch 2482) currently wait for new map > 2018-01-29 20:18:08.117736 7fc0f4904700 1 bluestore(/var/lib/ceph/osd/ceph-9) > umount > 2018-01-29 20:18:08.317882 7fc0f4904700 1 stupidalloc shutdown > 2018-01-29 20:18:08.322660 7fc0f4904700 1 freelist shutdown > 2018-01-29 20:18:08.322718 7fc0f4904700 4 rocksdb: > [/build/ceph-12.2.2/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling > all background work > 2018-01-29 20:18:08.339096 7fc0f4904700 4 rocksdb: > [/build/ceph-12.2.2/src/rocksdb/db/db_impl.cc:343] Shutdown complete > 2018-01-29 20:18:08.508354 7fc0f4904700 1 bluefs umount > 2018-01-29 20:18:08.508555 7fc0f4904700 1 stupidalloc shutdown > 2018-01-29 20:18:08.508571 7fc0f4904700 1 stupidalloc shutdown > 2018-01-29 20:18:08.508573 7fc0f4904700 1 stupidalloc shutdown > 2018-01-29 20:18:08.508718 7fc0f4904700 1 bdev(0x5560701c7440 > /dev/nvme0n1p19) close > 2018-01-29 20:18:08.786803 7fc0f4904700 1 bdev(0x5560701c6fc0 > /dev/nvme0n1p20) close > 2018-01-29 20:18:08.874747 7fc0f4904700 1 bdev(0x5560701c7200 > /var/lib/ceph/osd/ceph-9/block) close > 2018-01-29 20:18:09.030750 7fc0f4904700 1 bdev(0x5560701c6d80 > /var/lib/ceph/osd/ceph-9/block) close > > BR, > Bruce J. > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com