OK, I've managed to answer one of my own questions:
QUESTION: The new nodes don't own any of the partitions in the data
directory. What does it do?
ANSWER: The nodes won't start. They crash with:
{"Kernel pid
terminated",application_controller,"{application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['riakjade@10.20.15.11',[{'riakmercury@10.20.11.168',[{{riak_core,vnode_routing},[legacy]}]},{'riakpyrite@10.20.11.169',[{{riak_core,vnode_routing},[legacy]}]},{'riaktopaz@10.20.11.170',[{{riak_core,vnode_routing},[legacy]}]}]],[{file,[111,114,100,100,105,99,116,46,101,114,108]},{line,72}]},{riak_core_capability,renegotiate_capabilities,1,[{file,[115,114,99,47,114,105,97,107,95,99,111,114,101,95,99,97,112,97,98,105,108,105,116,121,46,101,114,108]},{line,415}]},{riak_core_capability,handle_call,3,[{file,[115,114,99,47,114,105,97,107,95,99,111,114,101,95,99,97,112,97,98,105,108,105,116,121,46,101,114,108]},{line,208}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,227}]}]},{gen_server,c
all,[riak_core_capability,{register,{riak_core,vnode_routing},{capability,[proxy,legacy],legacy,{riak_core,legacy_vnode_routing,[{true,legacy},{false,proxy}]}}},infinity]}}}}}}"}
Crash dump was written to: ./log/erl_crash.dump
Kernel pid terminated (application_controller)
({application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['riakjade@10.20.15.11',
I assume that's because none of the nodes mentioned in the ring file are
accessible.
Last night I thought I'd got it working! I used "reip" on all the nodes
and started them up. This time "member-status" reported that all the
nodes were valid. "transfers" showed partition handoff was happening.
Unfortunately, today we noticed that some data was missing from the
cluster. Upon investigation I found that I hadn't copied one of the data
directories correctly so one of the nodes actually started with an empty
data directory! The weird thing is if there's one node in the cluster
that hasn't needed to upgrade its data directory then the other nodes
seem to upgrade OK and we don't get the problem of one node being in
"legacy" state.
Two other things we've tried:
"reip" just the node we're going to start Riak on, start it up and then
replace the other nodes. Riak starts up but the "force-replace" fails with:
13:11:53.540 [error] Forced node replacement failed
exit:{{nodedown,'riakamber@10.20.11.167'},{gen_server,call,[{riak_core_claimant,'riakamber@10.20.11.167'},{stage,'riakmercury@10.20.11.168',{force_replace,'riakjade@10.20.15.11'}},infinity]}}
I'm not surprised because none of the nodes it's trying to talk to exist.
Change the node name in vm.args to the old node and start it up then try
to replace the node with the new name. But we couldn't communicate with
the node because riak-admin uses the current node name to connect and it
wasn't valid.
The next thing we're about to do is install 1.1.1 on the new nodes,
"reip" the ring, start Riak and let it handoff any partitions it needs
to, shut it down, install 1.2.1, start it up and hope for the best. In
theory this "rolling upgrade" should work . . . assuming 1.1.1 "reip"
actually works.
I'm really keen to hear from anyone with any suggestions.
On 08/11/12 21:10, Shane McEwan wrote:
G'day!
Just to add to the list of people asking questions about migrating to
1.2.1 . . .
We're about to migrate our 4 node production Riak database from 1.1.1 to
1.2.1. At the same time we're also migrating from virtual machines to
physical machines. These machines will have new names and IP addresses.
The process of doing rolling upgrades is well documented but I'm unsure
of the correct procedure for moving to an entirely new cluster.
We have the luxury of a maintenance window so we don't need to keep
everything running during the migration. Therefore the current plan is
to stop the current cluster, copy the Riak data directories to the new
machines and start up the new cluster. The hazy part of the process is
how we "reip" the database so it will work in the new cluster.
We've tried using the "riak-admin reip" command but were left with one
of our nodes in "(legacy)" mode according to "riak-admin member-status".
From an earlier E-Mail thread[1] it seems like "reip" is deprecated and
we should be doing a "cluster force replace" instead.
So, would the new procedure be the following?
1. Shutdown old cluster
2. Copy data directory
3. Start new cluster (QUESTION: The new nodes don't own any of the
partitions in the data directory. What does it do?) (QUESTION: The new
nodes won't be part of a cluster yet. Do I need to "join" them before I
can do any of the following commands? Or do I just put all the joins and
force-replace commands into the same plan and commit it all together?)
3. Issue "riak-admin cluster force-replace old-node1 new-node1"
(QUESTION: Do I run this command just on "new-node1" or on all nodes?)
4. Issue "force-replace" commands for the remaining three nodes.
5. Issue a "cluster plan" and "cluster commit" to commit the changes.
6. Cross fingers.
In my mind the "replace" and/or "force-replace" commands are something
we would use it we had a failed node and needed to bring a spare online
to take over. It doesn't feel like something you would do if you don't
already have a cluster in place and are needing to "replace" ALL nodes.
Of course, we want to test this procedure before doing it for real. What
are the risks of doing the above procedure while the old cluster is
still running? While the new nodes are on a segregated network and
shouldn't be able to contact the old nodes what would happen if we did
the above and found the network wasn't as segregated as we originally
thought? Would the new nodes start trying to communicate with the old
nodes before the "force-replace" can take effect? Or, because all the
cluster changes are atomic there won't be any risk of that?
Sorry for all the questions. I'm just trying to get a clear procedure
for moving an entire cluster to new hardware and hopefully this thread
will help other people in the future.
Thanks in advance!
Shane.
[1] http://comments.gmane.org/gmane.comp.db.riak.user/8418
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
