Hi all, I upgraded to the pacific version with cephadm. However, all our RGW daemons cannot start anymore. any help is appreciated.
Here are the logs when starting RGW. I set debug_rados and debug_rgw to 20/20 systemd[1]: Started Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. bash[9823]: WARNING: Error loading config file: .dockercfg: $HOME is not defined bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 deferred set uid:gid to 167:167 (ceph:ceph) bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 7 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework: beast bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework conf key: port, val: 7480 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 1 radosgw_Main not setting numa affinity bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f742b700 20 reqs_thread_entry: start bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f6c2a700 10 entry start bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting msgr at bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting objecter bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: setting wanted keys bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: calling monclient init bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 1 librados: init done bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 10 librados: wait_for_osdmap waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: wait_for_osdmap done waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 realm smil 6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch nspace= bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=10 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 nspace= bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=877 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 current period 5528a87b-bca7-4936-b348-fcb8f213df20 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: read oid=converted nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=default.region nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.realm bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa.control bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_names.b7-1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.6c92e1e6-c08b-4f4c-b211-7899893978fa:staging bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zone.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms_names.smil bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got period_config.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroups_names.default bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zone_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroups_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=region_map nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup init ret 0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup name default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 using current period zonegroup default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_names.b7-1 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 Cannot find current period zone using local zone bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 zone b7-1 found bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-source-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-target-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started sync module instance, tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started zone id=ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 (name=b7-1) with tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: create oid=notify.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.215+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.219+0000 7ff80f172440 10 librados: create oid=notify.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: create oid=notify.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: create oid=notify.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: create oid=notify.4 nspace= bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: create oid=notify.5 nspace= bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: create oid=notify.6 nspace= bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: create oid=notify.7 nspace= bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=2 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=3 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=4 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=5 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=6 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 2 all 8 watchers are set, enabling cache bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=0 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-61 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 10 librados: Objecter returned from call r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error) bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 20 remove_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 2 removed watcher, disabling cache bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 20 remove_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush exit ... bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 20 remove_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 1 librados: shutdown bash[9823]: debug 2021-04-04T13:01:05.279+0000 7ff80f172440 -1 Couldn't init storage provider (RADOS) systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac...@rgw.smil.b7-1.gpu006.twfefs.service: Main process exited, code=exited, status=5/NOTINSTALLED systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac...@rgw.smil.b7-1.gpu006.twfefs.service: Failed with result 'exit-code'. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac...@rgw.smil.b7-1.gpu006.twfefs.service: Service hold-off time over, scheduling restart. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac...@rgw.smil.b7-1.gpu006.twfefs.service: Scheduled restart job, restart counter is at 4. systemd[1]: Stopped Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. Weiwen Hu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
