>From: Pavel Kirienko <[email protected]>
>Date: Monday, May 6, 2013 6:29 AM
>To: <[email protected]>
>Subject: Inconsistent cluster membership
>Here are the questions:
>1. Why member-status output on different nodes is inconsistent, and what to do
about it?
Some occurrance has caused the nodes to no longer agree on the ring. If you
run riak-admin ring_status or riak-admin ringready you will likely see
errors. The first node appears to have completely lost its copy of the
ring, and on the next restart created a new one as if it were a brand new
node, i.e. 100% ownership. One possible cause of this is disk corruption or
disk full.
The fix is (after ruling out hardware failure) issue a riak-admin cluster
join command from the detached (first) node to cause it to get a fresh copy
of the ring. It will then engage in hinted handoffs to redistribute any
data that should be stored on other nodes.
>2. Sometimes the first node returns error "{insufficient_vnodes,0,need,2}" on
read queries. This error goes away for few days when the node restarted. I
suspect that this has some relation to the first question.
This indicates that the first node does not have any of the vnodes listed in
a preflist currently running, and its incorrect ring prevents it from being
able to query any other nodes . This could be that it has vnodes that have
stopped or crashed, or it could indicate a bad preflist caused by ongoing
ring corruption.
>3. Node on 192.168.0.3 is desperately trying to join the cluster for at least
10 days, still no luck. (overall data size is about 60GB, network is 100Mbit
Ethernet)
No operations that cause partition ownership to change (join/leave/replace)
can occur until all the nodes in the cluster agree on the ring. Fixing the
first node should rectify this issue.
>Which additional info may help?
>As I said before, things went wrong when the cluster was updated to 1.3.1. So,
maybe the right way to fix these problems is to install 1.3.1 cleanly, then
restore the data from backup?
You should not have lost any data, although you are currently partitioned.
Once you have all of the nodes participating again, hinted handoff and AAE
should restore consistency to its previous levels.
>Thanks in advance!
>Pavel.
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
