[
https://issues.apache.org/jira/browse/CASSANDRA-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277058#comment-17277058
]
Berenguer Blasi commented on CASSANDRA-16408:
---------------------------------------------
I've been hacking EC2Snitch around to make it work locally in case it was sthg
snitch related. I haven't been able to repro so [~aholmber] I think we should
try to repro on AWS directly next
{noformat}
Datacenter: us-east-1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 127.0.0.4 68.55 KiB 1 ?
f0289aac-f844-42ee-9b88-ab31182bfd86 us-east-1a1
UN 127.0.0.3 73.6 KiB 1 ?
38eca3b8-d123-4aaa-8dd0-faaa7ac7e734 us-east-1a0
UN 127.0.0.2 68.55 KiB 1 ?
704ef741-dfcd-452a-b68e-2742a61eee09 us-east-1a2
UN 127.0.0.1 68.58 KiB 1 ?
d678157e-1bc8-409c-b2b3-9ba357bd6db0 us-east-1a1
{noformat}
> Unable to bootstrap/join new nodes to existing 4.0 cluster
> ----------------------------------------------------------
>
> Key: CASSANDRA-16408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16408
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership, Consistency/Bootstrap and
> Decommission
> Reporter: J Hickey
> Assignee: Berenguer Blasi
> Priority: Normal
> Fix For: 4.0-beta
>
>
> Trying to add a new node to an existing 4.0 cluster gets stuck in
> bootstrap/joining permanently with no clear error.
> Version: 4.0-beta4 (issue also seen in 4.0-beta3, and NOT seen in 3.11.x) and
> Java 8 (Open JDK 1.8.0_275)
> Topology: 3 rack single DC using EC2Snitch, 1 seed node per rack
> Relevant cassandra.yaml settings:
> {code}
> auto_bootstrap: true (implicit)
> seeds contains the same 3 nodes on all nodes
> num_tokens: 16
> allocate_tokens_for_local_replication_factor: 3
> server_encryption_options.internode_encryption: all
> server_encryption_options.enabled: true
> server_encryption_options.optional: false
> server_encryption_options.require_client_auth: true
> client_encryption_options.enabled: true
> client_encryption_options.optional: false
> client_encryption_options.require_client_auth: true
> {code}
> Scenario:
> * Bring up the 3 seed nodes to create a new cluster.
> * Add a user keyspace: create keyspace test with replication = \{ 'class':
> 'NetworkTopologyStrategy', 'us-east-1-dc': 3 }; and insert some test data.
> * Wait at least 10 minutes after the initial 3 seed nodes come up (nodes will
> join if they are brought up at the same time as the seeds, but not if they
> are brought up later).
> * Start cassandra on a fourth node.
> Cassandra begins to bootstrap but does not ever finish (I have left this
> running overnight) and does not exit nor log any errors. Nodetool status from
> any node shows new node as UJ. Nodetool netstats from new node shows
> receiving file from test keyspace at 100% received. Logs show bootstrap
> starting and streaming starting, but then nothing/no errors.
> Worth noting here that I have also tried this with
> allocate_tokens_for_local_replication_factor disabled and still have this
> issue. I have also tried this without any user keyspace/data, just completely
> empty cluster and still have this issue. The only way I seem to be able to
> bring up a decently sized cluster on 4.0 is to disable
> allocate_tokens_for_local_replication_factor (to avoid collisions as
> mentioned in other issues) and bring up all nodes at about the same time, or
> use auto_bootstrap: false. I have no issue adding a new node in a similar
> fashion to a 3.11.x cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
