Hi Ben, We've seen similar behavior once a time. The cause was unsynchronized time on the nodes. So I can recommend to check that system time is equal over the nodes (especially on that one "bad" server).
On 13 September 2012 05:15, Ben Frank <b...@airlust.com> wrote: > Hey all, > I'm setting up a new ring on 11 machines with cassandra 1.1.5. All > seems to install fine and startup ok (from tarball) but I'm having an issue > when updating the schema. There's one node that just doesn't want to > receive the schema change. I've tried blowing away my /var/lib/cassandra/* > directories and trying it a few times with the same results. I also tried > the instructions here: > http://wiki.apache.org/cassandra/FAQ#schema_disagreement > > If I update the schema with cassandra-cli I get the following error: > > Waiting for schema agreement... > The schema has not settled in 10 seconds; further migrations are > ill-advised until it does. > Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103, > 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, > 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133], > b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109] > > then if I do a 'describe cluster' I get: > > Cluster Information: > Snitch: org.apache.cassandra.locator.SimpleSnitch > Partitioner: org.apache.cassandra.dht.RandomPartitioner > Schema versions: > 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103, > 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, > 12.19.102.135, 12.19.103.105, 12.19.102.133] > > a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109] > > if I keep running describe cluster, both the schema UUID's(?) above keep > changing every few seconds. Looking at the logs shows the nodes > are continually doing a flush and compact. > > if I start over by blowing away /var/lib/cassandra, and then try the > schema update on the machine which is having trouble receiving > it everything works ok and all nodes are in sync. > > Anyone know what's going on here, or what I should look at to troubleshoot > this? > > -Ben >
