When I run the command ceph -s, I see the following information on the mon log:
2012-07-05 02:44:13.298942 7f7d92b14700 0 can't decode unknown message type 54 MSG_AUTH=17 2012-07-05 02:44:13.301588 7f7d9401b700 1 mon.a@0(leader).paxos(auth active c 412..432) is_readable now=2012-07-05 02:44:13.301590 lease_expire=2012-07-05 02:44:17.566529 has v0 lc 432 2012-07-05 02:44:13.302113 7f7d9401b700 1 mon.a@0(leader).paxos(auth active c 412..432) is_readable now=2012-07-05 02:44:13.302114 lease_expire=2012-07-05 02:44:17.566529 has v0 lc 432 2012-07-05 02:44:13.303072 7f7d92b14700 0 can't decode unknown message type 54 MSG_AUTH=17 2012-07-05 02:44:13.309450 7f7d9401b700 1 mon.a@0(leader).paxos(auth active c 412..432) is_readable now=2012-07-05 02:44:13.309452 lease_expire=2012-07-05 02:44:17.566529 has v0 lc 432 2012-07-05 02:44:13.309845 7f7d9401b700 1 mon.a@0(leader).paxos(auth active c 412..432) is_readable now=2012-07-05 02:44:13.309847 lease_expire=2012-07-05 02:44:17.566529 has v0 lc 432
.... Couldn't find any helpful information regarding "can't decode" error message, unless digging into the codes. Thanks for any hint. Xiaopong On 07/05/2012 02:41 PM, Xiaopong Tran wrote:
Hi, I put up a small cluster with 3 osds, 2 mds, 3 mons, on 3 machines. They were running 0.47.2, and this is a test to do rolling upgrade to 0.48. I shutdown, upgraded the software, then restarted. One node at a time. The first two seemed to be ok. The third one gave me some weird thing. While it was doing the conversion and recovering, the command ceph -s gives things like this: root@china:/tmp# ceph -s 2012-07-05 14:28:41.069470 7fa3c8443780 2 auth: KeyRing::load: loaded key file /etc/ceph/client.admin.keyring 2012-07-05 14:28:41.594229 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.596313 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.598949 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.601158 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.603069 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.605020 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.607436 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.609304 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.611047 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.667980 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.670283 7fa3c030e700 0 monclient: hunting for new mon 2012-07-05 14:28:41.672274 7fa3c030e700 0 monclient: hunting for new mon .... And it never stopped. I was thinking, maybe it just behaved like that during recovery. But after the recovery is done, it still get the same thing: root@china:/tmp# ceph health 2012-07-05 14:28:55.077364 7f8306a0d780 2 auth: KeyRing::load: loaded key file /etc/ceph/client.admin.keyring HEALTH_OK root@china:/tmp# ceph -s 2012-07-05 14:30:49.688017 7feb6338e780 2 auth: KeyRing::load: loaded key file /etc/ceph/client.admin.keyring 2012-07-05 14:30:49.691690 7feb5b259700 0 monclient: hunting for new mon 2012-07-05 14:30:49.694295 7feb5b259700 0 monclient: hunting for new mon 2012-07-05 14:30:49.696487 7feb5b259700 0 monclient: hunting for new mon 2012-07-05 14:30:49.698953 7feb5b259700 0 monclient: hunting for new mon 2012-07-05 14:30:49.700833 7feb5b259700 0 monclient: hunting for new mon .... Upgrading the first two nodes have no such problem. This first two nodes all run osd, mds, and mon. The third only runs osd and mon. The mon log on the 3rd node shows this, not sure if this is helpful: .... 925291 lease_expire=2012-07-05 02:38:14.149966 has v44 lc 44 2012-07-05 02:38:12.572107 7f7d9381a700 1 mon.a@0(leader).paxos(pgmap active c 29531..30031) is_readable now=2012-07-05 02:38:12.572114 lease_expire=2012-07-05 02:38:15.889056 has v0 lc 30031 2012-07-05 02:38:12.572128 7f7d9381a700 1 mon.a@0(leader).paxos(pgmap active c 29531..30031) is_readable now=2012-07-05 02:38:12.572129 lease_expire=2012-07-05 02:38:15.889056 has v0 lc 30031 2012-07-05 02:38:15.120439 7f7d9401b700 1 mon.a@0(leader).paxos(mdsmap active c 1..44) is_readable now=2012-07-05 02:38:15.120446 lease_expire=2012-07-05 02:38:17.149967 has v44 lc 44 2012-07-05 02:38:15.925349 7f7d9401b700 1 mon.a@0(leader).paxos(mdsmap active c 1..44) is_readable now=2012-07-05 02:38:15.925356 lease_expire=2012-07-05 02:38:20.149971 has v44 lc 44 2012-07-05 02:38:17.572181 7f7d9381a700 1 mon.a@0(leader).paxos(pgmap active c 29531..30031) is_readable now=2012-07-05 02:38:17.572189 lease_expire=2012-07-05 02:38:21.889065 has v0 lc 30031 2012-07-05 02:38:17.572204 7f7d9381a700 1 mon.a@0(leader).paxos(pgmap active c 29531..30031) is_readable now=2012-07-05 02:38:17.572205 lease_expire=2012-07-05 02:38:21.889065 has v0 lc 30031 2012-07-05 02:38:19.120463 7f7d9401b700 1 mon.a@0(leader).paxos(mdsmap active c 1..44) is_readable now=2012-07-05 02:38:19.120470 lease_expire=2012-07-05 02:38:23.149973 has v44 lc 44 2012-07-05 02:38:19.925323 7f7d9401b700 1 mon.a@0(leader).paxos(mdsmap active c 1..44) is_readable now=2012-07-05 02:38:19.925330 lease_expire=2012-07-05 02:38:23.149973 has v44 lc 44 Could someone give a hint on this? Thanks Xiaopong
-- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html