Hi Roshni, I was just about to summarize. I got it running. The problem eventually was the firewall. I didn't have the full control over the server so I couldn't check the iptables rules. The incoming traffic on 7000 port was rejected on the second node. I've just checked one way communication from node2 -> node1 using telnet. It was very helpful to run cassandra in TRACE mode. It displays a lot of logs but was able to compare the behavior with a working ring. I've noticed that the second node was sending a Gossip SYN message but was never getting back the SYN_ACK.
Anyway to everybody that will have a problem like that, check your firewall settings:) Thanks for help Roshni. -- regards, Jakub Glapa On Fri, Aug 3, 2012 at 9:41 AM, Roshni Rajagopal < roshni.rajago...@wal-mart.com> wrote: > Hi Jakub, > > Were you able to resolve the issue? > For a multi data center setup I do believe some steps are different. You > may need to set Networktopology as your replication strategy rather than > simple strategy, and setup a snitch. And mention the rack/dc configurations > in a config file. > You can refer to the steps for a multi data center installation. > > Regards, > Roshni > > > > From: Jakub Glapa <jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Re: Unsuccessful attempt to add a second node to a ring. > > I found a similar thread from March : > http://www.mail-archive.com/user@cassandra.apache.org/msg21007.html > > For me clearing the data and starting from the beginning didn't help. > > It's interesting because on my dev environment I was able to add another > node without any problems. > > The only difference is that the second node now is in a different data > center. (but I'm not using any different settings, SimpleSnitch) > 7000,9160,7199 ports were open between those 2 nodes. > > How else can I check if the communication between those 2 nodes is working? > In the logs I see that: > DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 > OutboundTcpConnection.java (line 206) attempting to connect to > NODE1/node1.ip > > So I assume that the communication is somehow established? > > > -- > regards, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa <jakub.gl...@gmail.com > <mailto:jakub.gl...@gmail.com>> wrote: > yes it's the same > > > > -- > regards, > pozdrawiam, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal < > roshni.rajago...@wal-mart.com<mailto:roshni.rajago...@wal-mart.com>> > wrote: > Ok, sorry it may not be required, > I was thinking of a configuration I had done on my local laptop, where I > had aliased my IP address. > In that case the directories and jmx port needed to be different. > > Cluster name is same right? > > > From: Jakub Glapa <jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com > ><mailto:jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com>>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org > ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > Subject: Re: Unsuccessful attempt to add a second node to a ring. > > Hi Roshni, > no they are the same, my changes in cassandra.yaml were only in the > listen_address, rpc_address, seeds and initial_token field. > The rest is exactly the same as on node1. > > That's how the file looks on node2: > > > > cluster_name: 'Test Cluster' > initial_token: 85070591730234615865843651857942052864 > hinted_handoff_enabled: true > hinted_handoff_throttle_delay_in_ms: 1 > authenticator: org.apache.cassandra.auth.AllowAllAuthenticator > authority: org.apache.cassandra.auth.AllowAllAuthority > partitioner: org.apache.cassandra.dht.RandomPartitioner > data_file_directories: > - /data/servers/cassandra_sbe_edtool/cassandra_data/data > commitlog_directory: > /data/servers/cassandra_sbe_edtool/cassandra_data/commitlog > saved_caches_directory: > /data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches > commitlog_sync: periodic > commitlog_sync_period_in_ms: 10000 > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "NODE1" > flush_largest_memtables_at: 0.75 > reduce_cache_sizes_at: 0.85 > reduce_cache_capacity_to: 0.6 > concurrent_reads: 32 > concurrent_writes: 32 > memtable_flush_queue_size: 4 > sliced_buffer_size_in_kb: 64 > storage_port: 7000 > ssl_storage_port: 7001 > listen_address: NODE2 > rpc_address: NODE2 > rpc_port: 9160 > rpc_keepalive: true > rpc_server_type: sync > thrift_framed_transport_size_in_mb: 15 > thrift_max_message_length_in_mb: 16 > incremental_backups: false > snapshot_before_compaction: false > column_index_size_in_kb: 64 > in_memory_compaction_limit_in_mb: 64 > multithreaded_compaction: false > compaction_throughput_mb_per_sec: 16 > compaction_preheat_key_cache: true > rpc_timeout_in_ms: 10000 > endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch > dynamic_snitch_update_interval_in_ms: 100 > dynamic_snitch_reset_interval_in_ms: 600000 > dynamic_snitch_badness_threshold: 0.1 > request_scheduler: org.apache.cassandra.scheduler.NoScheduler > index_interval: 128 > encryption_options: > internode_encryption: none > keystore: conf/.keystore > keystore_password: cassandra > truststore: conf/.truststore > truststore_password: cassandra > > > > > -- > regards, > pozdrawiam, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal < > roshni.rajago...@wal-mart.com<mailto:roshni.rajago...@wal-mart.com > ><mailto:roshni.rajago...@wal-mart.com<mailto: > roshni.rajago...@wal-mart.com>>> wrote: > Jakub, > > Have you set the > Data, commitlog, saved cache directories to different ones in each yaml > file for each node? > > Regards, > Roshni > > > From: Jakub Glapa <jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com > ><mailto:jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com>><mailto: > jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com><mailto: > jakub.gl...@gmail.com<mailto:jakub.gl...@gmail.com>>>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org > ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org > >><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org > ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>> > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>> > Subject: Unsuccessful attempt to add a second node to a ring. > > Hi Everybody! > > I'm trying to add a second node to an already operating one node cluster. > > Some specs: > - cassandra 1.0.7 > - both nodes have a routable listen_address and rpc_address. > - Ports are open: (from node2) telnet node1 7000 is successful > - Seeds parameter on node2 points to node 1. > > [node1] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node1.ip datacenter1 rack1 Up Normal 74.33 KB > 100.00% 0 > > - initial token on node2 was specified > > I see something like that in the logs on node2: > > DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) > collectTimeOrderedData > INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) > JOINING: waiting for ring and schema information > DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 > OutboundTcpConnection.java (line 206) attempting to connect to > NODE1/node1.ip > DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java > (line 86) Disseminating load info ... > INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) > JOINING: schema complete, ready to bootstrap > DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... > got ring + schema info > INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) > JOINING: getting bootstrap token > DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token > manually specified as 85070591730234615865843651857942052864 > DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying > mutation of row 4c > > > but it doesn't join the ring: > > [node2] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node2.ip datacenter1 rack1 Up Normal 13.49 KB 100.00% > 85070591730234615865843651857942052864 > > > > I'm attaching the full log from node2 startup in debug mode. > > > > PS. > When I didn't specified the initial token on node2 I ended up with > exception like that: > "Exception encountered during startup: No other nodes seen! Unable to > bootstrap.If you intended to start a single-node cluster, you should make > sure your broadcast_address (or listen_address) is listed as a seed. > Otherwise, you need to determine why the seed being contacted has no > knowledge of the rest of the cluster. Usually, this can be solved by > giving all nodes the same seed list." > > > I'm not sure how to proceed now. I found a couple of posts with problems > like that but they weren't very useful. > > -- > regards, > Jakub Glapa > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** >
