Re: EC2 node adding trouble

2011-05-26 Thread Sasha Dolgy
All of the nodes are 0.8.x and no longer 0.7.2 ? It is peculiar. As another person noted, maybe reviewing the results of netstat -an on each node could help. I've had issues in the past with JMX -- thinking I had configured it for IP x.x.x.x and went crazy only to see that it was never

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
Thanks for all your helpful suggestions - I've now got it working. It was down to a combination of things. 1. A missing rule in a security group 2. A missing DNS name for the new node, so its default name was defaulting to localhost 3. Google DNS caching the failed DNS lookup for the full

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
On 24 May 2011, at 23:58, Sameer Farooqui wrote: So, once you know what token each of the 3 nodes should have, shut down the first two nodes, change their tokens and add the correct token to the 3rd node (in the YAML file). I'd like to make sure I've got the right sequence of operations for

Re: EC2 node adding trouble

2011-05-26 Thread Sasha Dolgy
As an aside, you can also use that command to pull meta-data about instances in AWS. I have implemented this to maintain a list of seed nodes. This way, when a new instance is brought online, the default cassandra.yaml is `enhanced` to contain a dynamic list of valid seeds, proper hostname and a

Re: EC2 node adding trouble

2011-05-26 Thread Sasha Dolgy
On Thu, May 26, 2011 at 3:12 PM, Marcus Bointon mar...@synchromedia.co.uk wrote: I'd like to make sure I've got the right sequence of operations for adding a node without downtime. If I'm going from 2 to 3 nodes: 1 Calculate new initial_token values using the python script 2 Change token

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
On 26 May 2011, at 15:21, Sasha Dolgy wrote: Turn the node off, remove the node from the ring using nodetool and removetoken i've found this to be the best problem-free way. Maybe it's better now ... http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html So I'd need to have

Re: EC2 node adding trouble

2011-05-26 Thread aaron morton
This ticket may be just the ticket :) https://issues.apache.org/jira/browse/CASSANDRA-2452 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27 May 2011, at 01:16, Sasha Dolgy wrote: As an aside, you can also use that command to

Re: EC2 node adding trouble

2011-05-26 Thread aaron morton
This is the *most* useful page on the wiki http://wiki.apache.org/cassandra/Operations Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27 May 2011, at 02:06, Marcus Bointon wrote: On 26 May 2011, at 15:21, Sasha

Re: EC2 node adding trouble

2011-05-25 Thread Marcus Bointon
On 24 May 2011, at 23:58, Sameer Farooqui wrote: Even with AutoBootstrap it is recommended that you always specify the InitialToken on the new node because the picking of an initial token will almost certainly result in an unbalanced ring. Right now, I'm afraid that if you simply copied

Re: EC2 node adding trouble

2011-05-25 Thread mcasandra
-incubator-apache-org.3065146.n2.nabble.com/EC2-node-adding-trouble-tp6399102p6403602.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: EC2 node adding trouble

2011-05-25 Thread aaron morton
I've seen discussion of using the EIP but I do not have direct experience. Others may be able to provide more help. Previous discussion

EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
Hi, First time here. I'm having trouble adding a third node to an existing 2-node ring (successfully upgraded from 0.72) running cassandra 0.8rc1 (successfully upgraded from 0.72) on ubuntu on EC2. Evidently the seed node is working as the second node is already talking to it, nodetool lists

Re: EC2 node adding trouble

2011-05-24 Thread Sameer Farooqui
What region and availability zones are the different nodes in? Are you using EC2 Snitch? Did you set up the cluster using the Datastax AMI? - Sameer On Tue, May 24, 2011 at 8:35 AM, Marcus Bointon mar...@synchromedia.co.ukwrote: Hi, First time here. I'm having trouble adding a third node to

Re: EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
On 24 May 2011, at 19:33, Sameer Farooqui wrote: What region and availability zones are the different nodes in? Are you using EC2 Snitch? Did you set up the cluster using the Datastax AMI? The two existing ones are in us-east-1c and us-east-1d, the new one is in us-east-1c, so all same

Re: EC2 node adding trouble

2011-05-24 Thread Sameer Farooqui
Even with AutoBootstrap it is recommended that you always specify the InitialToken on the new node because the picking of an initial token will almost certainly result in an unbalanced ring. Right now, I'm afraid that if you simply copied the YAML file from one of the two nodes to the 3rd node,

Re: EC2 node adding trouble

2011-05-24 Thread aaron morton
Check the listen_address and rpc_address in the yaml file for each node. I think they are normally set to the private and public respectively. This may make your live easier http://www.datastax.com/dev/blog/setting-up-a-cassandra-cluster-with-the-datastax-ami Cheers - Aaron