I plan to install 2.0.6 as soon as it will be available in datastax rpm repo. But how to deal with schema inconsistency on such scale? best regards Aleksander
2014-03-11 13:40 GMT+01:00 Duncan Sands <duncan.sa...@gmail.com>: > Hi Aleksander, this may be related to CASSANDRA-6799 and CASSANDRA-6700 (if > it is caused by CASSANDRA-6700 then you are in luck: it is fixed in 2.0.6). > > Best wishes, Duncan. > > > On 11/03/14 13:30, olek.stas...@gmail.com wrote: >> >> Hi All, >> I've faced an issue with cassandra 2.0.5. >> I've 6 node cluster with random partitioner, still using tokens >> instead of vnodes. >> Cause we're changing hardware we decide to migrate cluster to 6 new >> machines and change partitioning options to vnode rather then >> token-based. >> I've followed instruction on site: >> >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html >> and started cassandra on 6 new nodes in new DC. Everything seems to >> work correctly, nodes were seen from all others as up and normal. >> Then i performed nodetool repair -pr on the first of new nodes. >> But process falls into infinite loop, sending/receiving merkle trees >> over and over. It hangs on one very small KS it there were no hope it >> will stop sometime (process was running whole night). >> So I decided to stop the repair and restart cass on this particular >> new node. after restart 'Ive tried repair one more time with another >> small KS, but it also falls into infinite loop. >> So i decided to break the procedure of adding datacenter, remove nodes >> from new DC and start all from scratch. >> After running removenode on all new nodes I've wiped data dir and >> start cassandra on new node once again. During the start messages >> "org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find >> cfId=98bb99a2-42f2-3fcd-af67-208a4faae5fa" >> appears in logs. Google said, that they may mean problems with schema >> versions consistency, so I performed describe cluster in cassandra-cli >> and i get: >> Cluster Information: >> Name: Metadata Cluster >> Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch >> Partitioner: org.apache.cassandra.dht.RandomPartitioner >> Schema versions: >> 76198f8b-663f-3434-8860-251ebc6f50c4: [150.254.164.4] >> >> f48d3512-e299-3508-a29d-0844a0293f3a: [150.254.164.3] >> >> 16ad2e35-1eef-32f0-995c-e2cbd4c18abf: [150.254.164.6] >> >> 72352017-9b0d-3b29-8c55-ed86f30363c5: [150.254.164.1] >> >> 7f1faa84-0821-3311-9232-9407500591cc: [150.254.164.5] >> >> 85cd0ebc-5d33-3bec-a682-8c5880ee2fa1: [150.254.164.2] >> >> So now I have 6 diff schema version for cluster. But how it can >> happened? How can I take my cluster to consistent state? >> What did I wrong during extending cluster, so nodetool falls into infinite >> loop? >> At the first sight data looks ok, I can read from cluster and I'm >> getting expected output. >> best regards >> Aleksander >> >
