Hi, I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from ceph version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the OSDs, I receive the error message:
2016-06-01 12:17:49.219512 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30 2016-06-01 12:17:49.219605 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying 2016-06-01 12:18:19.219740 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30 2016-06-01 12:18:19.219782 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying 2016-06-01 12:18:49.219869 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30 2016-06-01 12:18:49.219908 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying eventually the OSD fails to start. Oddly, this affects only some of the OSDs on this host. On the mon host, there is a warning: 2016-06-01 11:22:52.171152 osd.177 10.31.0.71:6842/10245 433 : cluster [WRN] failed to encode map e282667 with expected crc 2016-06-01 11:22:55.399487 osd.177 10.31.0.71:6842/10245 434 : cluster [WRN] failed to encode map e282668 with expected crc 2016-06-01 11:22:55.572238 osd.177 10.31.0.71:6842/10245 435 : cluster [WRN] failed to encode map e282668 with expected crc 2016-06-01 11:22:59.571924 osd.177 10.31.0.71:6842/10245 436 : cluster [WRN] failed to encode map e282669 with expected crc 2016-06-01 11:22:59.665925 osd.177 10.31.0.71:6842/10245 437 : cluster [WRN] failed to encode map e282669 with expected crc 2016-06-01 11:22:59.671475 osd.177 10.31.0.71:6842/10245 438 : cluster [WRN] failed to encode map e282669 with expected crc 2016-06-01 11:22:59.780646 osd.177 10.31.0.71:6842/10245 439 : cluster [WRN] failed to encode map e282669 with expected crc 2016-06-01 11:23:00.778250 osd.177 10.31.0.71:6842/10245 440 : cluster [WRN] failed to encode map e282670 with expected crc 2016-06-01 11:23:02.075996 osd.177 10.31.0.71:6842/10245 441 : cluster [WRN] failed to encode map e282671 with expected crc 2016-06-01 11:23:03.110117 osd.177 10.31.0.71:6842/10245 442 : cluster [WRN] failed to encode map e282672 with expected crc 2016-06-01 11:23:03.255597 osd.177 10.31.0.71:6842/10245 443 : cluster [WRN] failed to encode map e282672 with expected crc 2016-06-01 11:23:03.258223 osd.177 10.31.0.71:6842/10245 444 : cluster [WRN] failed to encode map e282672 with expected crc 2016-06-01 11:23:05.287753 osd.177 10.31.0.71:6842/10245 445 : cluster [WRN] failed to encode map e282673 with expected crc How do I clear these up after the upgrade? All of the filesystems on the OSDs are mounted and the keyrings are there...... Thanks, Jeff -- Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdon...@msi.umn.edu 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
