Thanks Guido. Looks like we've upgraded to 1.4.0 completely now and the cluster is back up.
I'm not sure of the exact root cause, but what we were seeing was that too many nodes went down for the ring to be healthy, and then when nodes were restarted they waited for the ring to appear for a while, and then exited (or crashed) automatically. I *think* this is because after the package was upgraded, the Riak service didn't restart after the upgrade -- and our automated system to catch that and restart them wasn't kicking in quickly enough compared to the rate at which the servers were being upgraded. You mention in the release notes that the init scripts have been adjusted - maybe that has something to do with it? Cheers, Toby ----- Original Message ----- From: "Guido Medina" <guido.med...@temetra.com> To: "riak-users" <riak-users@lists.basho.com> Sent: Wednesday, 10 July, 2013 9:05:16 PM Subject: Re: Upgrade path 1.4.0 Release notes: https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md Maybe related to this? Known Issues leveldb 1.3 to 1.4 conversion The first execution of 1.4.0 leveldb using a 1.3.x or 1.2.x dataset will initiate an automatic conversion that could pause the startup of each node by 3 to 7 minutes. The leveldb data in "level #1" is being adjusted such that "level #1" can operate as an overlapped data level instead of as a sorted data level. The conversion is simply the reduction of the number of files in "level #1" to being less than eight via normal compaction of data from "level #1" into "level #2". This is a one time conversion. HTH, Guido. On 10/07/13 11:57, Toby Corkindale wrote: Hi, some of our nodes upgraded to Riak 1.4.0, and are now refusing to start and join the cluster. Is there documentation on the upgrade path from 1.3.2 to 1.4.0? It appears we have accidentally begun this journey, and I don't know if it's easier to go back or forwards now.. PS. It would have been pleasant to receive a warning about the new version on the mailing list! Thanks, Toby _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com