how to change a write's and a read's consistency level separately in cqlsh?

[Nimbus Lin]

To Sir Oleksandr :
   Thank you very much for your careful teaching,  at the begging, I copied 
system_auth keyspace and tables' sql grammar
  and misunderstood the HA function of cassandra, now I know cassandra'ha as 
hadoop or greenplum.
  And I will check the 3rd answer on Jconsole latter.
  On the 2nd question, would you like to tell me how to change a write's 
and a read's consistency level separately in cqlsh?
Otherwise, how the document's R+W>Replicator to realize to guarantee a strong 
consistency write and read?


Thank you!

Sincerely
Nimbuslin(Lin JiaXin)
Mobile: 0086 180 5986 1565
Mail: jiaxin...@live.com



From: Oleksandr Shulgin 
Sent: Monday, June 17, 2019 7:19 AM
To: User
Subject: Re: How can I check cassandra cluster has a real working function of 
high availability?

On Sat, Jun 15, 2019 at 4:31 PM Nimbus Lin 
mailto:jiaxin...@live.com>> wrote:
Dear cassandra's pioneers:
I am a 5 years' newbie,  it is until now that I have time to use cassandra. 
but I cann't check cassandra's high availabily when I stop a seed node or none 
seed DN as CGE or Greenplum.
Would someone can tell me how to check the cassandra's high availability? 
even I change the consistency level from one to local_one, the cqlsh's select 
is always return an error of NoHostAvailable.

 By the way, would you like to answer me other two questions:
2nd question: although cassandrfa's consistency is a per-operation setting, 
isn't there a whole all operations' consistency setting method?
3rd question: how can I can cassandra cluster's running variables as mysql's 
show global variables? such as hidden variable of  auto_bootstrap?

Hi,

For the purpose of serving client requests, all nodes are equal -- seed or not. 
 So it shouldn't matter which node you are stopping (or making it unavailable 
for the rest of the cluster using other means).

In order to test it with cqlsh you should ensure that the replication factors 
of the keyspace you're testing with is sufficient.  Given the NoHostAvailable 
exception that you are experiencing at consistency level ONE (or LOCAL_ONE), I 
can guess that you are testing with a keyspace with replication factor 1 and 
the node which is unavailable happen to be responsible for the particular 
partition.

For your second question: it depends on a client (or "client driver") you are 
using.  In cqlsh you can set consistency level that will be applied for all 
subsequent queries using the "CONSISTENCY ..." command.  I think that the Java 
driver does have an option to set the default consistency level, as well as has 
an option to set consistency level per query.  Most likely this is also true 
for Python and other drivers.

And for the third question: I'm not aware of a CQL or nodetool command that 
would fulfill the need.  Most likely it is possible to learn (and update) most 
of the configuration parameters using JMX, e.g. with JConsole: 
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsMonitoring.html#opsMonitoringJconsole

Cheers,
--
Alex


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: How can I check cassandra cluster has a real working function of high availability?

[Nimbus Lin]

To Sir Oleksandr :




Thank you!

Sincerely
Nimbuslin(Lin JiaXin)
Mobile: 0086 180 5986 1565
Mail: jiaxin...@live.com



From: Oleksandr Shulgin 
Sent: Monday, June 17, 2019 7:19 AM
To: User
Subject: Re: How can I check cassandra cluster has a real working function of 
high availability?

On Sat, Jun 15, 2019 at 4:31 PM Nimbus Lin 
mailto:jiaxin...@live.com>> wrote:
Dear cassandra's pioneers:
I am a 5 years' newbie,  it is until now that I have time to use cassandra. 
but I cann't check cassandra's high availabily when I stop a seed node or none 
seed DN as CGE or Greenplum.
Would someone can tell me how to check the cassandra's high availability? 
even I change the consistency level from one to local_one, the cqlsh's select 
is always return an error of NoHostAvailable.

 By the way, would you like to answer me other two questions:
2nd question: although cassandrfa's consistency is a per-operation setting, 
isn't there a whole all operations' consistency setting method?
3rd question: how can I can cassandra cluster's running variables as mysql's 
show global variables? such as hidden variable of  auto_bootstrap?

Hi,

For the purpose of serving client requests, all nodes are equal -- seed or not. 
 So it shouldn't matter which node you are stopping (or making it unavailable 
for the rest of the cluster using other means).

In order to test it with cqlsh you should ensure that the replication factors 
of the keyspace you're testing with is sufficient.  Given the NoHostAvailable 
exception that you are experiencing at consistency level ONE (or LOCAL_ONE), I 
can guess that you are testing with a keyspace with replication factor 1 and 
the node which is unavailable happen to be responsible for the particular 
partition.

For your second question: it depends on a client (or "client driver") you are 
using.  In cqlsh you can set consistency level that will be applied for all 
subsequent queries using the "CONSISTENCY ..." command.  I think that the Java 
driver does have an option to set the default consistency level, as well as has 
an option to set consistency level per query.  Most likely this is also true 
for Python and other drivers.

And for the third question: I'm not aware of a CQL or nodetool command that 
would fulfill the need.  Most likely it is possible to learn (and update) most 
of the configuration parameters using JMX, e.g. with JConsole: 
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsMonitoring.html#opsMonitoringJconsole

Cheers,
--
Alex


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

[Jeff Jirsa]

If you’re at RF= 3 and read/write at quorum, you’ll have full visibility of all 
data if you switch to RF=4 and continue reading at quorum because quorum if 4 
is 3, so you’re guaranteed to overlap with at least one of the two nodes that 
got all earlier writes

Going from 3 to 4 to 5 requires a repair after 4.


> On Jun 28, 2019, at 8:32 PM, Oleksandr Shulgin  
> wrote:
> 
>> On Fri, Jun 28, 2019 at 11:29 PM Jeff Jirsa  wrote:
> 
>>  you often have to run repair after each increment - going from 3 -> 5 means 
>> 3 -> 4, repair, 4 -> 5 - just going 3 -> 5 will violate consistency 
>> guarantees, and is technically unsafe.
> 
> Jeff,
> 
> How going from 3 -> 4 is *not violating* consistency guarantees already?  Are 
> you assuming quorum writes and reads and a perfectly repaired keyspace?
> 
> Regards,
> --
> Alex
> 

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

[Oleksandr Shulgin]

On Fri, Jun 28, 2019 at 11:29 PM Jeff Jirsa  wrote:

>  you often have to run repair after each increment - going from 3 -> 5
> means 3 -> 4, repair, 4 -> 5 - just going 3 -> 5 will violate consistency
> guarantees, and is technically unsafe.
>

Jeff,

How going from 3 -> 4 is *not violating* consistency guarantees already?
Are you assuming quorum writes and reads and a perfectly repaired keyspace?

Regards,
--
Alex

Re: Securing cluster communication

[Oleksandr Shulgin]

On Fri, Jun 28, 2019 at 3:57 PM Marc Richter  wrote:

>
> How is this dealt with in Cassandra? Is setting up firewalls the only
> way to allow only some nodes to connect to the ports 7000/7001?
>

Hi,

You can set

server_encryption_options:
internode_encryption: all
...

and distribute the same key/trust-store on each node of the same cluster.

Cheers,
--
Alex

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

[Jon Haddad]

Yep - not to mention the increased complexity and overhead of going from
ONE to QUORUM, or the increased cost of QUORUM in RF=5 vs RF=3.

If you're in a cloud provider, I've found you're almost always better off
adding a new DC with a higher RF, assuming you're on NTS like Jeff
mentioned.

On Fri, Jun 28, 2019 at 2:29 PM Jeff Jirsa  wrote:

> For just changing RF:
>
> You only need to repair the full token range - how you do that is up to
> you. Running `repair -pr -full` on each node will do that. Running `repair
> -full` will do it multiple times, so it's more work, but technically
> correct.The caveat that few people actually appreciate about changing
> replication factors (# of copies per DC) is that you often have to run
> repair after each increment - going from 3 -> 5 means 3 -> 4, repair, 4 ->
> 5 - just going 3 -> 5 will violate consistency guarantees, and is
> technically unsafe.
>
> For changing replication strategy:
>
> Changing replication strategy is nontrivial - going from Simple to NTS is
> often easy to do in a truly eventual consistency use case, but becomes much
> harder if you're:
> - using multiple DCs or
> - vnodes + racks or
> - if you must do it without violating consistency.
>
> It turns out if you're not using multiple DCs or racks, then
> simplestrategy is fine. But if you are using multiple DCs/racks, then
> changing is very very hard. So usually by the time you're asking how to do
> this, you're in a very bad position.
>
> Do you have simple strategy and multiple DCs?
> Are you using vnodes and racks?
>
> I'd be incredibly skeptical about any blog that tried to give concrete
> steps on how to do this - the steps are probably right 80% of the time, but
> horribly wrong 20% of the time, especially if there's not a paragraph or
> two about racks along the way.
>
>
>
>
>
> On Fri, Jun 28, 2019 at 7:52 AM Fd Habash  wrote:
>
>> Hi all …
>>
>>
>>
>> The datastax & apache docs are clear: run ‘nodetool repair’ after you
>> alter a keyspace to change its RF or RS.
>>
>>
>>
>> However, the details are all over the place as what type of repair and on
>> what nodes it needs to run. None of the above doc authorities are clear and
>> what you find on the internet is quite contradictory.
>>
>>
>>
>> For example, this IBM doc
>> 
>> suggest to run both the ‘alter keyspace’ and repair on EACH node affected
>> or on ‘each node you need to change the RF on’.  Others
>> ,
>> suggest to run ‘repair -pr’.
>>
>>
>>
>> On a cluster of 1 DC and three racks, this is how I understand it ….
>>
>>1. Run the ‘alter keyspace’ on a SINGLE node.
>>2. As for repairing the altered keyspac, I assume there are two
>>options …
>>   1. Run ‘repair -full [key_space]’ on all nodes in all racks
>>   2. Run ‘repair -pr -full [keyspace] on all nodes in all racks
>>
>>
>>
>> Sounds correct?
>>
>>
>>
>> 
>> Thank you
>>
>>
>>
>

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

[Jeff Jirsa]

For just changing RF:

You only need to repair the full token range - how you do that is up to
you. Running `repair -pr -full` on each node will do that. Running `repair
-full` will do it multiple times, so it's more work, but technically
correct.The caveat that few people actually appreciate about changing
replication factors (# of copies per DC) is that you often have to run
repair after each increment - going from 3 -> 5 means 3 -> 4, repair, 4 ->
5 - just going 3 -> 5 will violate consistency guarantees, and is
technically unsafe.

For changing replication strategy:

Changing replication strategy is nontrivial - going from Simple to NTS is
often easy to do in a truly eventual consistency use case, but becomes much
harder if you're:
- using multiple DCs or
- vnodes + racks or
- if you must do it without violating consistency.

It turns out if you're not using multiple DCs or racks, then simplestrategy
is fine. But if you are using multiple DCs/racks, then changing is very
very hard. So usually by the time you're asking how to do this, you're in a
very bad position.

Do you have simple strategy and multiple DCs?
Are you using vnodes and racks?

I'd be incredibly skeptical about any blog that tried to give concrete
steps on how to do this - the steps are probably right 80% of the time, but
horribly wrong 20% of the time, especially if there's not a paragraph or
two about racks along the way.





On Fri, Jun 28, 2019 at 7:52 AM Fd Habash  wrote:

> Hi all …
>
>
>
> The datastax & apache docs are clear: run ‘nodetool repair’ after you
> alter a keyspace to change its RF or RS.
>
>
>
> However, the details are all over the place as what type of repair and on
> what nodes it needs to run. None of the above doc authorities are clear and
> what you find on the internet is quite contradictory.
>
>
>
> For example, this IBM doc
> 
> suggest to run both the ‘alter keyspace’ and repair on EACH node affected
> or on ‘each node you need to change the RF on’.  Others
> ,
> suggest to run ‘repair -pr’.
>
>
>
> On a cluster of 1 DC and three racks, this is how I understand it ….
>
>1. Run the ‘alter keyspace’ on a SINGLE node.
>2. As for repairing the altered keyspac, I assume there are two
>options …
>   1. Run ‘repair -full [key_space]’ on all nodes in all racks
>   2. Run ‘repair -pr -full [keyspace] on all nodes in all racks
>
>
>
> Sounds correct?
>
>
>
> 
> Thank you
>
>
>

RE: [EXTERNAL] Re: Bursts of Thrift threads make cluster unresponsive

[Durity, Sean R]

This sounds like a bad query or large partition. If a large partition is 
requested on multiple nodes (because of consistency level), it will pressure 
all those replica nodes. Then, as the cluster tries to adjust the rest of the 
load, the other nodes can get overwhelmed, too.

Look at cfstats to see if you have some large partitions. You may also see them 
as warnings in the system.log when they are getting compacted.

Also check for any ALLOW FILTERING queries in the code (or slow query stats, if 
you have them)

Sean


From: Dmitry Simonov 
Sent: Thursday, June 27, 2019 5:22 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Bursts of Thrift threads make cluster unresponsive

> Is there an order in which the events you described happened, or is the order 
> with which you presented them the order you notice things going wrong?

At first, threads count (Thrift) start increasing.
After 2 or 3 minutes they consume all CPU cores.
After that, simultaneously: message drops occur, read latency increases, active 
read tasks are noticed.

пт, 28 июн. 2019 г. в 01:40, Avinash Mandava 
mailto:avin...@vorstella.com>>:
Yeah i skimmed too fast, don't add more work if CPU is pegged, and if using 
thrift protocol NTR would not have values.

Is there an order in which the events you described happened, or is the order 
with which you presented them the order you notice things going wrong?

On Thu, Jun 27, 2019 at 1:29 PM Dmitry Simonov 
mailto:dimmobor...@gmail.com>> wrote:
Thanks for your reply!

> Have you tried increasing concurrent reads until you see more activity in 
> disk?
When problem occurs, freshly created 1.2k - 2k Thrift threads consume all CPU 
on all cores.
Does increasing concurrent reads may help in this situation?

> org.apache.cassandra.metrics.type=ThreadPools.path=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count
This metric is 0 at all cluster nodes.

пт, 28 июн. 2019 г. в 00:34, Avinash Mandava 
mailto:avin...@vorstella.com>>:
Have you tried increasing concurrent reads until you see more activity in disk? 
If you've always got 32 active reads and high pending reads it could just be 
dropping the reads because the queues are saturated. Could be artificially 
bottlenecking at the C* process level.

Also what does this metric show over time:

org.apache.cassandra.metrics.type=ThreadPools.path=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count



On Thu, Jun 27, 2019 at 1:52 AM Dmitry Simonov 
mailto:dimmobor...@gmail.com>> wrote:
Hello!

We've met several times the following problem.

Cassandra cluster (5 nodes) becomes unresponsive for ~30 minutes:
- all CPUs have 100% load (normally we have LA 5 on 16-cores machine)
- cassandra's threads count raises from 300 to 1300 - 2000,most of them are 
Thrift threads in java.net.SocketInputStream.socketRead0(Native Method) method, 
count of other threads doesn't increase
- some Read messages are dropped
- read latency (p99.9) increases to 20-30 seconds
- there are up to 32 active Read Tasks, up to 3k - 6k pending Read Tasks

Problem starts synchronously on all nodes of cluster.
I cannot tie this problem with increased load from clients ("read rate" does't 
increase during the problem).
Also looks like there is no problem with disks (I/O latencies are OK).

Could anybody please give some advice in further troubleshooting?

--
Best Regards,
Dmitry Simonov


--
www.vorstella.com
408 691 8402


--
Best Regards,
Dmitry Simonov


--
www.vorstella.com
408 691 8402


--
Best Regards,
Dmitry Simonov



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or 

Running Node Repair After Changing RF or Replication Strategy for a Keyspace

[Fd Habash]

Hi all …

The datastax & apache docs are clear: run ‘nodetool repair’ after you alter a 
keyspace to change its RF or RS.

However, the details are all over the place as what type of repair and on what 
nodes it needs to run. None of the above doc authorities are clear and what you 
find on the internet is quite contradictory.

For example, this IBM doc suggest to run both the ‘alter keyspace’ and repair 
on EACH node affected or on ‘each node you need to change the RF on’.  Others, 
suggest to run ‘repair -pr’. 

On a cluster of 1 DC and three racks, this is how I understand it ….
1. Run the ‘alter keyspace’ on a SINGLE node. 
2. As for repairing the altered keyspac, I assume there are two options …
a. Run ‘repair -full [key_space]’ on all nodes in all racks
b. Run ‘repair -pr -full [keyspace] on all nodes in all racks

Sounds correct? 


Thank you

Re: Securing cluster communication

[Hannu Kröger]

I would start checking this page: 
http://cassandra.apache.org/doc/latest/operating/security.html

Then move to this:
https://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html

Cheers,
Hannu

> Marc Richter  kirjoitti 28.6.2019 kello 16.55:
> 
> Hi everyone,
> 
> I'm completely new to Cassandra DB, so please do not roast me for asking 
> obvious stuff.
> 
> I managed to setup one Cassandra node and enter some data to it, 
> successfully. Next, I installed a second node, which connects to that first 
> one via port 7000 and sync all that data from it. This worked fine as well.
> 
> But doing so, it leaves me puzzled a bit because of the security aspect of 
> this: Neither did I need to authenticate to the seeding (first) node, nor did 
> I find a resource which describes how to secure that cluster communication by 
> implementing some kind of authentication, which prevents everyone on the same 
> net to connect to the nodes.
> 
> How is this dealt with in Cassandra? Is setting up firewalls the only way to 
> allow only some nodes to connect to the ports 7000/7001?
> 
> BR,
> Marc
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

Securing cluster communication

[Marc Richter]

Hi everyone,

I'm completely new to Cassandra DB, so please do not roast me for asking 
obvious stuff.


I managed to setup one Cassandra node and enter some data to it, 
successfully. Next, I installed a second node, which connects to that 
first one via port 7000 and sync all that data from it. This worked fine 
as well.


But doing so, it leaves me puzzled a bit because of the security aspect 
of this: Neither did I need to authenticate to the seeding (first) node, 
nor did I find a resource which describes how to secure that cluster 
communication by implementing some kind of authentication, which 
prevents everyone on the same net to connect to the nodes.


How is this dealt with in Cassandra? Is setting up firewalls the only 
way to allow only some nodes to connect to the ports 7000/7001?


BR,
Marc

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Restore from EBS onto different cluster

[Oleksandr Shulgin]

On Fri, Jun 28, 2019 at 8:37 AM Ayub M  wrote:

> Hello, I have a cluster with 3 nodes - say cluster1 on AWS EC2 instances.
> The cluster is up and running, took snapshot of the keyspaces volume.
>
> Now I want to restore few tables/keyspaces from the snapshot volumes, so I
> created another cluster say cluster2 and attached the snapshot volumes on
> to the new cluster's ec2 nodes. Cluster2 is not starting bcz the system
> keyspace in the snapshot taken was having cluster name as cluster1 and the
> cluster on which it is being restored is cluster2. How do I do a restore in
> this case? I do not want to do any modifications to the existing cluster.
>

Hi,

I would try to use the same cluster name just to be able to restore it and
ensure that nodes of cluster2 cannot talk to cluster1 by the means of
setting up Security Groups, for example.

Also when I do restore do I need to think about the token ranges of the old
> and new cluster's mapping?
>

Absolutely.  For a successful restore you must ensure that you restore a
snapshot from a rack (Avalability Zone, if you're using the EC2Snitch) into
the same rack in the new cluster.

Regards,
--
Alex

Re: Restore from EBS onto different cluster

[Rhys Campbell]

Sstableloader is probably your best option

Ayub M  schrieb am Fr., 28. Juni 2019, 08:37:

> Hello, I have a cluster with 3 nodes - say cluster1 on AWS EC2 instances.
> The cluster is up and running, took snapshot of the keyspaces volume.
>
> Now I want to restore few tables/keyspaces from the snapshot volumes, so I
> created another cluster say cluster2 and attached the snapshot volumes on
> to the new cluster's ec2 nodes. Cluster2 is not starting bcz the system
> keyspace in the snapshot taken was having cluster name as cluster1 and the
> cluster on which it is being restored is cluster2. How do I do a restore in
> this case? I do not want to do any modifications to the existing cluster.
>
> Also when I do restore do I need to think about the token ranges of the
> old and new cluster's mapping?
>
> Regards,
> Ayub
>

Restore from EBS onto different cluster

[Ayub M]

Hello, I have a cluster with 3 nodes - say cluster1 on AWS EC2 instances.
The cluster is up and running, took snapshot of the keyspaces volume.

Now I want to restore few tables/keyspaces from the snapshot volumes, so I
created another cluster say cluster2 and attached the snapshot volumes on
to the new cluster's ec2 nodes. Cluster2 is not starting bcz the system
keyspace in the snapshot taken was having cluster name as cluster1 and the
cluster on which it is being restored is cluster2. How do I do a restore in
this case? I do not want to do any modifications to the existing cluster.

Also when I do restore do I need to think about the token ranges of the old
and new cluster's mapping?

Regards,
Ayub

Re: Ec2MultiRegionSnitch difficulties (3.11.2)

[Oleksandr Shulgin]

On Fri, Jun 28, 2019 at 3:14 AM Voytek Jarnot 
wrote:

> Curious if anyone could shed some light on this. Trying to set up a
> 4-node, one DC (for now, same region, same AZ, same VPC, etc) cluster in
> AWS.
>
> All nodes have the following config (everything else basically standard):
> cassandra.yaml:
>   listen_address: NODE?_PRIVATE_IP
>   seeds: "NODE1_ELASTIC_IP"
>   endpoint_snitch: Ec2MultiRegionSnitch
> cassandra-rackdc.properties:
>   empty except prefer_local=true
>
> I've tried setting
>   broadcast_address: NODE?_ELASTIC_IP
> But that didn't help - and it seems redundant, as it appears that that's
> what the Ec2MultiRegionSnitch does anyway.
>

Hi,

It is quite confusing, as I remember hitting this issue before.  You need
to set the following:

broadcast_rpc_address: NODE?_ELASTIC_IP

Even though, listen_address and broadcast_address should be set by the EC2
snitch automatically, the above parameter is not set automatically and this
is the one that the nodes are using to talk to each other.

Cheers,
--
Alex