[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
[ https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143199#comment-16143199 ] Jeff Jirsa commented on CASSANDRA-13378: Resolving as 'not a problem', since the problem was an error in your dc name. > DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us > -- > > Key: CASSANDRA-13378 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13378 > Project: Cassandra > Issue Type: Bug > Components: Configuration > Environment: 5 node cluster , > • Server Model: Dell R730xd > • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores > • Memory: 192 GB > • OS Drive: 2 * 120 GB > • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD > • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card > OS - OEL 2.6.32-642.15.1.el6.x86_64 > java version "1.8.0_91" > Java(TM) SE Runtime Environment (build 1.8.0_91-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) > 6 nodes in other DC are identical >Reporter: Kevin Rivait >Priority: Blocker > Fix For: 3.0.9 > > Attachments: cassandra-rackdc.properties_from_dc1_node, > cassandra-rackdc.properties_from_dc2_node, cassandra.yaml, > DS-add-a-dc-C3.0.rtf > > > i have replicated the issue on my personal cluster using VMs. > we have many keyspaces and users developing on the Dev Cluster we are trying > to stretch. > With my current VMs (10 total) i can create a 2 DC cluster dc1 and dc2. > i rebuild all nodes - clear data dirs and restart dc1. > i also clear data dirs on dc2 nodes, i donot restart yet, > now i have single dc1 5 nodes cluster. > snitch - GossipingPropertyFileSnitch > i create keyspaces on DC1 > after i alter keyspaces with replication dc1 3 dc2 0i can no longer > query tables - not enough replicas available for query at consistency ONE. > same error with CQL using consistency local_one > continue with procedure, startup dc2 nodes, > alter replication on keyspaces to dc1 3 dc2 2 > from dc2 nodes , nodetool rebuild -- dc1 fails > i am attaching detailed steps with errors and cassandra.yaml -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
[ https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962976#comment-15962976 ] Jeff Jirsa commented on CASSANDRA-13378: In that case, CASSANDRA-12681 will eventually make this problem a nonissue, though for compatibility reasons it won't be available until 4.0 > DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us > -- > > Key: CASSANDRA-13378 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13378 > Project: Cassandra > Issue Type: Bug > Components: Configuration > Environment: 5 node cluster , > • Server Model: Dell R730xd > • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores > • Memory: 192 GB > • OS Drive: 2 * 120 GB > • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD > • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card > OS - OEL 2.6.32-642.15.1.el6.x86_64 > java version "1.8.0_91" > Java(TM) SE Runtime Environment (build 1.8.0_91-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) > 6 nodes in other DC are identical >Reporter: Kevin Rivait >Priority: Blocker > Fix For: 3.0.9 > > Attachments: cassandra-rackdc.properties_from_dc1_node, > cassandra-rackdc.properties_from_dc2_node, cassandra.yaml, > DS-add-a-dc-C3.0.rtf > > > i have replicated the issue on my personal cluster using VMs. > we have many keyspaces and users developing on the Dev Cluster we are trying > to stretch. > With my current VMs (10 total) i can create a 2 DC cluster dc1 and dc2. > i rebuild all nodes - clear data dirs and restart dc1. > i also clear data dirs on dc2 nodes, i donot restart yet, > now i have single dc1 5 nodes cluster. > snitch - GossipingPropertyFileSnitch > i create keyspaces on DC1 > after i alter keyspaces with replication dc1 3 dc2 0i can no longer > query tables - not enough replicas available for query at consistency ONE. > same error with CQL using consistency local_one > continue with procedure, startup dc2 nodes, > alter replication on keyspaces to dc1 3 dc2 2 > from dc2 nodes , nodetool rebuild -- dc1 fails > i am attaching detailed steps with errors and cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
[ https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962952#comment-15962952 ] Kevin Rivait commented on CASSANDRA-13378: -- Datastax has updated their procedure ( 1 small tweak). https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html We updated our scripts to match and in that process, realized we had a problem with our DC name on the alter keyspace stmts. There was an extra space in front of the DC name. Looks like the original procedure would have also worked. We have run through the corrected procedure on EPG and have successfully stretched an existing cluster. I will reschedule the activity for our current DEV physical cluster. > DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us > -- > > Key: CASSANDRA-13378 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13378 > Project: Cassandra > Issue Type: Bug > Components: Configuration > Environment: 5 node cluster , > • Server Model: Dell R730xd > • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores > • Memory: 192 GB > • OS Drive: 2 * 120 GB > • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD > • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card > OS - OEL 2.6.32-642.15.1.el6.x86_64 > java version "1.8.0_91" > Java(TM) SE Runtime Environment (build 1.8.0_91-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) > 6 nodes in other DC are identical >Reporter: Kevin Rivait >Priority: Blocker > Fix For: 3.0.9 > > Attachments: cassandra-rackdc.properties_from_dc1_node, > cassandra-rackdc.properties_from_dc2_node, cassandra.yaml, > DS-add-a-dc-C3.0.rtf > > > i have replicated the issue on my personal cluster using VMs. > we have many keyspaces and users developing on the Dev Cluster we are trying > to stretch. > With my current VMs (10 total) i can create a 2 DC cluster dc1 and dc2. > i rebuild all nodes - clear data dirs and restart dc1. > i also clear data dirs on dc2 nodes, i donot restart yet, > now i have single dc1 5 nodes cluster. > snitch - GossipingPropertyFileSnitch > i create keyspaces on DC1 > after i alter keyspaces with replication dc1 3 dc2 0i can no longer > query tables - not enough replicas available for query at consistency ONE. > same error with CQL using consistency local_one > continue with procedure, startup dc2 nodes, > alter replication on keyspaces to dc1 3 dc2 2 > from dc2 nodes , nodetool rebuild -- dc1 fails > i am attaching detailed steps with errors and cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
[ https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941880#comment-15941880 ] Kevin Rivait commented on CASSANDRA-13378: -- nodetool status and attaching the cassandra-rackdc.properties file, one from each dc [cassandra@epgidvlgds1014 ~]$ nodetool status Datacenter: dc1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns (effective) Host ID Rack UN 10.126.57.95 87.11 KB 128 0.0% ee272ba5-db65-45d3-8f1d-0a7f68d78cbf rack1 UN 10.126.57.93 116.77 KB 128 0.0% c75bfdad-10b9-4b50-99bc-630520615c33 rack1 UN 10.126.56.205 111.8 KB 128 0.0% 7e8b33c0-c63c-478b-8945-dbbcf6d7971a rack1 UN 10.126.57.71 85.25 KB 128 0.0% 2d71cadf-8f32-45fa-aa3e-20b9043253ce rack1 UN 10.126.57.64 116.71 KB 128 0.0% 480832d8-5871-4e26-b2f5-159b3727c016 rack1 Datacenter: dc2 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns (effective) Host ID Rack UN 10.126.58.116 111.94 KB 128 40.2% 1af0b1f0-a1a1-4ccd-b9fc-ac73eb50d97b rack1 UN 10.126.58.117 167.53 KB 128 42.8% a5e50250-9657-40bf-8aed-c0166fea7083 rack1 UN 10.126.58.114 90.27 KB 128 41.7% 30d136ae-9b53-43cc-8fc5-12b048054d12 rack1 UN 10.126.58.115 88 KB 128 36.3% 85da4085-078b-4c84-ae4d-fd2769c71068 rack1 UN 10.126.58.113 94.18 KB 128 39.0% 41763b36-a669-43ea-be2e-93124a425c4b rack1 > DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us > -- > > Key: CASSANDRA-13378 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13378 > Project: Cassandra > Issue Type: Bug > Components: Configuration > Environment: 5 node cluster , > • Server Model: Dell R730xd > • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores > • Memory: 192 GB > • OS Drive: 2 * 120 GB > • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD > • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card > OS - OEL 2.6.32-642.15.1.el6.x86_64 > java version "1.8.0_91" > Java(TM) SE Runtime Environment (build 1.8.0_91-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) > 6 nodes in other DC are identical >Reporter: Kevin Rivait >Priority: Blocker > Fix For: 3.0.9 > > Attachments: cassandra.yaml, DS-add-a-dc-C3.0.rtf > > > i have replicated the issue on my personal cluster using VMs. > we have many keyspaces and users developing on the Dev Cluster we are trying > to stretch. > With my current VMs (10 total) i can create a 2 DC cluster dc1 and dc2. > i rebuild all nodes - clear data dirs and restart dc1. > i also clear data dirs on dc2 nodes, i donot restart yet, > now i have single dc1 5 nodes cluster. > snitch - GossipingPropertyFileSnitch > i create keyspaces on DC1 > after i alter keyspaces with replication dc1 3 dc2 0i can no longer > query tables - not enough replicas available for query at consistency ONE. > same error with CQL using consistency local_one > continue with procedure, startup dc2 nodes, > alter replication on keyspaces to dc1 3 dc2 2 > from dc2 nodes , nodetool rebuild -- dc1 fails > i am attaching detailed steps with errors and cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
[
https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941800#comment-15941800
]
Jeff Jirsa commented on CASSANDRA-13378:
Can you include the output of {{nodetool status}} and your gossiping property
file snitch config?
> DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us
> --
>
> Key: CASSANDRA-13378
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13378
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
> Environment: 5 node cluster ,
> • Server Model: Dell R730xd
> • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores
> • Memory: 192 GB
> • OS Drive: 2 * 120 GB
> • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD
> • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card
> OS - OEL 2.6.32-642.15.1.el6.x86_64
> java version "1.8.0_91"
> Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
> 6 nodes in other DC are identical
>Reporter: Kevin Rivait
>Priority: Blocker
> Fix For: 3.0.9
>
> Attachments: cassandra.yaml, DS-add-a-dc-C3.0.rtf
>
>
> i have replicated the issue on my personal cluster using VMs.
> we have many keyspaces and users developing on the Dev Cluster we are trying
> to stretch.
> With my current VMs (10 total) i can create a 2 DC cluster dc1 and dc2.
> i rebuild all nodes - clear data dirs and restart dc1.
> i also clear data dirs on dc2 nodes, i donot restart yet,
> now i have single dc1 5 nodes cluster.
> snitch - GossipingPropertyFileSnitch
> i create keyspaces on DC1
> after i alter keyspaces with replication dc1 3 dc2 0i can no longer
> query tables - not enough replicas available for query at consistency ONE.
> same error with CQL using consistency local_one
> continue with procedure, startup dc2 nodes,
> alter replication on keyspaces to dc1 3 dc2 2
> from dc2 nodes , nodetool rebuild -- dc1 fails
> i am attaching detailed steps with errors and cassandra.yaml
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
