Re: Cassandra cluster setup issue
Try it on the localhost. A - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 11/03/2013, at 11:29 PM, amulya rattan talk2amu...@gmail.com wrote: Here is a screenshot showing command ran and the fact that cassandra started fine with JMX bound on 0.0.0.0. image.png We have tried this with firewall disabled too 2013/3/12 aaron morton aa...@thelastpickle.com I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0 and 7199 can be seen bound by Cassandra upon netstat. When I do telnet x.x.x.x 7199, that works too. If the process was started with a JMX port property then you are not seeing that error. Can you show the output from node ring? Can you show the command you are using for nodetool ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 8/03/2013, at 11:34 PM, amulya rattan talk2amu...@gmail.com wrote: I just saw that the failed to connect to 'x.x.x.x:7199' : connection refused error for nodetool that I am facing used to exist earlier also, case in point https://issues.apache.org/jira/browse/CASSANDRA-4494 I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0 and 7199 can be seen bound by Cassandra upon netstat. When I do telnet x.x.x.x 7199, that works too. Any pointers please. 2013/3/8 amulya rattan talk2amu...@gmail.com I have started cassandra on three nodes, however opscenter only shows two of them. The missing one is starting properly, with gossip and everything else showing no errors. It also has the other nodes as seeds in cassandra.yaml, so I am amazed why it doesn't show as part of the ring. I have tried restarting it a bunch of times but neither does it throw any error, nor it shows as part of the ring Another strange thing is that nodetool won't start on any of the nodes. It keeps throwing connection refused error. It's not the same usual error one faces that starts with Error connection to remote JMX agent!. It simply says connection refused on x.x.x.x:7199. However, I can see the JMX port is bound by cassandra on 0.0.0.0. I can use cassandra-cli on 9160 and opscenter on , but not jmx port. As last resort, I turned off firewall completely, but to no avail. I never faced any such errors setting cassandra cluster up in the past, so this is completely flabbergasting to me given that the port is bound and firewall is off. Anybody faced anything similar in the past? Or anybody knows how to solve it? All responses appreciated. ~Amulya
Re: Cassandra cluster setup issue
I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0 and 7199 can be seen bound by Cassandra upon netstat. When I do telnet x.x.x.x 7199, that works too. If the process was started with a JMX port property then you are not seeing that error. Can you show the output from node ring? Can you show the command you are using for nodetool ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 8/03/2013, at 11:34 PM, amulya rattan talk2amu...@gmail.com wrote: I just saw that the failed to connect to 'x.x.x.x:7199' : connection refused error for nodetool that I am facing used to exist earlier also, case in point https://issues.apache.org/jira/browse/CASSANDRA-4494 I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0 and 7199 can be seen bound by Cassandra upon netstat. When I do telnet x.x.x.x 7199, that works too. Any pointers please. 2013/3/8 amulya rattan talk2amu...@gmail.com I have started cassandra on three nodes, however opscenter only shows two of them. The missing one is starting properly, with gossip and everything else showing no errors. It also has the other nodes as seeds in cassandra.yaml, so I am amazed why it doesn't show as part of the ring. I have tried restarting it a bunch of times but neither does it throw any error, nor it shows as part of the ring Another strange thing is that nodetool won't start on any of the nodes. It keeps throwing connection refused error. It's not the same usual error one faces that starts with Error connection to remote JMX agent!. It simply says connection refused on x.x.x.x:7199. However, I can see the JMX port is bound by cassandra on 0.0.0.0. I can use cassandra-cli on 9160 and opscenter on , but not jmx port. As last resort, I turned off firewall completely, but to no avail. I never faced any such errors setting cassandra cluster up in the past, so this is completely flabbergasting to me given that the port is bound and firewall is off. Anybody faced anything similar in the past? Or anybody knows how to solve it? All responses appreciated. ~Amulya
Re: Cassandra cluster setup issue
I just saw that the failed to connect to 'x.x.x.x:7199' : connection refused error for nodetool that I am facing used to exist earlier also, case in point https://issues.apache.org/jira/browse/CASSANDRA-4494 I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0 and 7199 can be seen bound by Cassandra upon netstat. When I do telnet x.x.x.x 7199, that works too. Any pointers please. 2013/3/8 amulya rattan talk2amu...@gmail.com I have started cassandra on three nodes, however opscenter only shows two of them. The missing one is starting properly, with gossip and everything else showing no errors. It also has the other nodes as seeds in cassandra.yaml, so I am amazed why it doesn't show as part of the ring. I have tried restarting it a bunch of times but neither does it throw any error, nor it shows as part of the ring Another strange thing is that nodetool won't start on any of the nodes. It keeps throwing connection refused error. It's not the same usual error one faces that starts with Error connection to remote JMX agent!. It simply says connection refused on x.x.x.x:7199. However, I can see the JMX port is bound by cassandra on 0.0.0.0. I can use cassandra-cli on 9160 and opscenter on , but not jmx port. As last resort, I turned off firewall completely, but to no avail. I never faced any such errors setting cassandra cluster up in the past, so this is completely flabbergasting to me given that the port is bound and firewall is off. Anybody faced anything similar in the past? Or anybody knows how to solve it? All responses appreciated. ~Amulya
RE: Cassandra Cluster Setup
I still have one open issue: My cluster has only 3 local nodes with 1 CF (5 rows). The initial start works fine. I can see the replicated data and can do all admin tasks I want, successfully. Now I'm simulating a node failure by stopping Cassandra. I remove (a) commit log (b) content in system keyspace (c) application cf When I bring the node back up I can see the new token assignment, streaming notification (we are talking about 5 rows in 1 cf). Thrift port is not available for ~30 min. Why is that? I would expect it to be back in a matter of maybe 1' (this should be data quantity related). Thanks, Stephan -Original Message- From: Gary Dusbabek [mailto:gdusba...@gmail.com] Sent: Friday, June 04, 2010 11:49 AM To: user@cassandra.apache.org Subject: Re: Cassandra Cluster Setup Great. It looks as if the replication factor is still 1, not 3. This means that each key lives only on one node. By increasing it to 3, data will be replicated across all 3 nodes. Gary. On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter stephan.pfammat...@logmein.com wrote: Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor. Now I can see the sample keys being spread across the 3 nodes: Cassandra-ca holds rows (keys: 1,5,8) Cassandra-az holds rows (keys: 11,55) Cassandra-or hodls rows (keys: 88) I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55. -Original Message- From: Nahor [mailto:nahor.j+gm...@gmail.com] Sent: Thursday, June 03, 2010 8:25 PM To: user@cassandra.apache.org Subject: Re: Cassandra Cluster Setup On 2010-06-03 13:07, Stephan Pfammatter wrote: Cassandra-or [...] SeedsSeedcassandra-or/Seed/Seeds Aside from the replication factor noted by Gary, this should point to your existing node (cassandra-ca) otherwise, how will this node know where existing node is and where to get the data from? Cassandra-az [...] SeedsSeedcassandra-az/Seed/Seeds Same here
Re: Cassandra Cluster Setup
It should take roughly 30 seconds (please clarify--not minutes) to join the cluster. Gary. On Mon, Jun 7, 2010 at 08:23, Stephan Pfammatter stephan.pfammat...@logmein.com wrote: I still have one open issue: My cluster has only 3 local nodes with 1 CF (5 rows). The initial start works fine. I can see the replicated data and can do all admin tasks I want, successfully. Now I'm simulating a node failure by stopping Cassandra. I remove (a) commit log (b) content in system keyspace (c) application cf When I bring the node back up I can see the new token assignment, streaming notification (we are talking about 5 rows in 1 cf). Thrift port is not available for ~30 min. Why is that? I would expect it to be back in a matter of maybe 1' (this should be data quantity related). Thanks, Stephan -Original Message- From: Gary Dusbabek [mailto:gdusba...@gmail.com] Sent: Friday, June 04, 2010 11:49 AM To: user@cassandra.apache.org Subject: Re: Cassandra Cluster Setup Great. It looks as if the replication factor is still 1, not 3. This means that each key lives only on one node. By increasing it to 3, data will be replicated across all 3 nodes. Gary. On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter stephan.pfammat...@logmein.com wrote: Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca and changed the replication factor. Now I can see the sample keys being spread across the 3 nodes: Cassandra-ca holds rows (keys: 1,5,8) Cassandra-az holds rows (keys: 11,55) Cassandra-or hodls rows (keys: 88) I don't see the data being duplicated yet. Am I missing a storage-conf setting? I need to proof that if Cassandra-az fails that I can still provide key 11,55. -Original Message- From: Nahor [mailto:nahor.j+gm...@gmail.com] Sent: Thursday, June 03, 2010 8:25 PM To: user@cassandra.apache.org Subject: Re: Cassandra Cluster Setup On 2010-06-03 13:07, Stephan Pfammatter wrote: Cassandra-or [...] SeedsSeedcassandra-or/Seed/Seeds Aside from the replication factor noted by Gary, this should point to your existing node (cassandra-ca) otherwise, how will this node know where existing node is and where to get the data from? Cassandra-az [...] SeedsSeedcassandra-az/Seed/Seeds Same here
Re: Cassandra Cluster Setup
Your replication factor is only set to 1, which means that each key will only live on a single node. If you do wait for bootstrapping to commence (takes 90s in trunk, I don't recall in 0.6), you should see some keys moving unless your inserts were all into a small range. Perhaps your being impatient. If not, I recommend that you start over, set the replication factor to 3, wait a good while for all the nodes to fully join the cluster, and then make sure the keys you write to are random. Gary. On Thu, Jun 3, 2010 at 13:07, Stephan Pfammatter stephan.pfammat...@logmein.com wrote: I’m having difficulties setting up a 3 way cassandra cluster. Any comments/help would be appreciated. My goal is that all data should be fully replicated amongst the 3 nodes. I want to simulate the failure of one node and proof that the test column family still can be accessed. In a nutshell I did the following 3 steps. The excerpt below shows the changed storage-con.xml: (1) I started Cassandra-ca and worked fine. I added 3 rows to a cf cf_test. (2) Then I started Cassandra-or. It comes up ok but goes into “sleeping 3”…..mode (3) Nodetool streams doesn’t reveal anything. Node: Cassandra-ca AutoBootstrapfalse/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-ca/Seed/Seeds ThriftAddresscassandra-us/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-us/ListenAddress Cassandra-or AutoBootstraptrue/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-or/Seed/Seeds ThriftAddresscassandra-or/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-or/ListenAddress Cassandra-az Cassandra-or AutoBootstraptrue/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-az/Seed/Seeds ThriftAddresscassandra-az/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-az/ListenAddress
Re: Cassandra Cluster Setup
http://wiki.apache.org/cassandra/MultinodeCluster On Thu, Jun 3, 2010 at 1:07 PM, Stephan Pfammatter stephan.pfammat...@logmein.com wrote: I’m having difficulties setting up a 3 way cassandra cluster. Any comments/help would be appreciated. My goal is that all data should be fully replicated amongst the 3 nodes. I want to simulate the failure of one node and proof that the test column family still can be accessed. In a nutshell I did the following 3 steps. The excerpt below shows the changed storage-con.xml: (1) I started Cassandra-ca and worked fine. I added 3 rows to a cf cf_test. (2) Then I started Cassandra-or. It comes up ok but goes into “sleeping 3”…..mode (3) Nodetool streams doesn’t reveal anything. Node: Cassandra-ca AutoBootstrapfalse/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-ca/Seed/Seeds ThriftAddresscassandra-us/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-us/ListenAddress Cassandra-or AutoBootstraptrue/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-or/Seed/Seeds ThriftAddresscassandra-or/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-or/ListenAddress Cassandra-az Cassandra-or AutoBootstraptrue/AutoBootstrap Keyspace Name=ks_test ColumnFamily Name=cf_test CompareWith=UTF8Type / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch /Keyspace SeedsSeedcassandra-az/Seed/Seeds ThriftAddresscassandra-az/ThriftAddress ThriftPort9160/ThriftPort ListenAddresscassandra-az/ListenAddress
