On 09/23/2011 03:51 AM, Shawn Moore wrote:
I did go ahead this morning and pull down g066d753 and apply it to the
four nodes. I brought up node174 first like usual, and then tried
node173, but it still refused to join, but with a different message:
.......
Sep 22 08:28:49 update_cluster_info(756) status = 2, epoch = 17, 66, 0
Sep 22 08:28:49 update_cluster_info(759) failed to join sheepdog, 66
Sep 22 08:28:49 leave_cluster(1984) 16
Sep 22 08:28:49 update_cluster_info(761) I am really hurt and gonna
leave cluster.
Sep 22 08:28:49 update_cluster_info(762) Fix yourself and restart me
later, pleaseeeee...Bye.
Sep 22 08:28:49 log_sigsegv(367) sheep logger exits abnormally, pid:24265
I then brought up node157 and it recovered. Then I brought up node156
and it recovered as well. Then I was able to bring up node173.
I noticed some odd things (mostly related to sizes of in use
changing), looking at "collie node info" during the node startups.
I'm not sure if this is normal or not, but below is what I saw. The
timing is from top (oldest) to bottom (most current). Do objects
re-distribute themselves around the cluster during recovery or epoch
changes?
node174 and node157:
[root@node174 ~]# collie node info
Id Size Used Use%
0 382 GB 17 GB 4%
1 394 GB 17 GB 4%
Total 775 GB 34 GB 4%, total virtual VDI Size 100 GB
Then added node156:
[root@node174 ~]# collie node info
Id Size Used Use%
0 365 GB 720 MB 0%
1 376 GB 12 GB 3%
2 380 GB 3.6 GB 0%
failed to read object, 80f5969500000000 Remote node has a new epoch
failed to read a inode header
failed to read object, 80f5969600000000 Remote node has a new epoch
failed to read a inode header
Total 1.1 TB 16 GB 1%, total virtual VDI Size 0.0 MB
[root@node174 ~]# collie node info
Id Size Used Use%
0 365 GB 1008 MB 0%
1 377 GB 12 GB 3%
2 382 GB 5.2 GB 1%
Total 1.1 TB 19 GB 1%, total virtual VDI Size 100 GB
Then after everyone is done with recovery, added node173 back:
[root@node174 ~]# collie node info
Id Size Used Use%
0 374 GB 10 GB 2%
1 377 GB 12 GB 3%
2 399 GB 22 GB 5%
Total 1.1 TB 45 GB 3%, total virtual VDI Size 100 GB
[root@node174 ~]# collie node info
Id Size Used Use%
0 365 GB 496 MB 0%
1 366 GB 1.3 GB 0%
2 377 GB 792 MB 0%
3 394 GB 17 GB 4%
failed to read object, 80f5969400000000 Remote node has a new epoch
failed to read a inode header
Total 1.5 TB 20 GB 1%, total virtual VDI Size 100 GB
[root@node174 sheepdog]# collie node info
Id Size Used Use%
0 386 GB 21 GB 5%
1 381 GB 17 GB 4%
2 397 GB 21 GB 5%
3 394 GB 17 GB 4%
Total 1.5 TB 76 GB 4%, total virtual VDI Size 100 GB
But as far as I can tell, everything is working right now.
Hi Shawn,
Thanks for your testing. Currently, for crash cluster (different
epoch with each nodes), we can only recovery the cluster if we obey some
start-up order. Basically, there are two orders that you would follow:
1 start-up the node which is crashed firstly. then start up others
nodes(which would exit with 'bye' message) until the first one starts to
recovery. from now on, you can safely join others in the cluster.
2 start-up the node which has the highest epoch version first. then
start one other node. now the first node is supposed to recovery. from
now one, you can join others into the cluster.
either one is okay to recover the whole cluster into functional state if
there is no object loss. For now, I think method 2 would allow you
recover the cluster safely. But Kazutaka is cooking a patch to address
object loss in the first method.
BTW, would you please try 'collie cluster info' to check if the outputs
are consistent on each node.
Thanks,
Yuan
--
sheepdog mailing list
[email protected]
http://lists.wpkg.org/mailman/listinfo/sheepdog
