Re: ssl certificate hot reloading test - cassandra 4.1

[Abe Ratnofsky]

Not to derail from the original conversation too far, but wanted to agree that 
maximum connection establishment time on native transport would be useful. That 
would provide a maximum duration before an updated client keystore is used for 
connections, which can be used to safely roll out client keystore updates.

For example, if the maximum connection establishment time is 12 hours, then you 
can update the keystore on a canary client, wait 24 hours, confirm that 
connectivity is maintained, then upgrade keystores across the rest of the fleet.

With unbounded connection establishment, reconnection isn't tested as often and 
issues can hide behind long-lived connections.

> On Apr 15, 2024, at 5:14 PM, Jeff Jirsa  wrote:
> 
> It seems like if folks really want the life of a connection to be finite 
> (either client/server or server/server), adding in an option to quietly drain 
> and recycle a connection on some period isn’t that difficult.
> 
> That type of requirement shows up in a number of environments, usually on 
> interactive logins (cqlsh, login, walk away, the connection needs to become 
> invalid in a short and finite period of time), but adding it to internode 
> could also be done, and may help in some weird situations (if you changed 
> certs because you believe a key/cert is compromised, having the connection 
> remain active is decidedly inconvenient, so maybe it does make sense to add 
> an expiration timer/condition on each connection).
> 
> 
> 
>> On Apr 15, 2024, at 12:28 PM, Dinesh Joshi  wrote:
>> 
>> In addition to what Andy mentioned, I want to point out that for the vast 
>> majority of use-cases, we would like to _avoid_ interruptions when a 
>> certificate is updated so it is by design. If you're dealing with a 
>> situation where you want to ensure that the connections are cycled, you can 
>> follow Andy's advice. It will require automation outside the database that 
>> you might already have. If there is demand, we can consider adding a feature 
>> to slowly cycle the connections so the old SSL context is not used anymore.
>> 
>> One more thing you should bear in mind is that Cassandra will not load the 
>> new SSL context if it cannot successfully initialize it. This is again by 
>> design to prevent an outage when the updated truststore is corrupted or 
>> could not be read in some way.
>> 
>> thanks,
>> Dinesh
>> 
>> On Mon, Apr 15, 2024 at 9:45 AM Tolbert, Andy > > wrote:
>>> I should mention, when toggling disablebinary/enablebinary between
>>> instances, you will probably want to give some time between doing this
>>> so connections can reestablish, and you will want to verify that the
>>> connections can actually reestablish.  You also need to be mindful of
>>> this being disruptive to inflight queries (if your client is
>>> configured for retries it will probably be fine).  Semantically to
>>> your applications it should look a lot like a rolling cluster bounce.
>>> 
>>> Thanks,
>>> Andy
>>> 
>>> On Mon, Apr 15, 2024 at 11:39 AM pabbireddy avinash
>>> mailto:pabbireddyavin...@gmail.com>> wrote:
>>> >
>>> > Thanks Andy for your reply . We will test the scenario you mentioned.
>>> >
>>> > Regards
>>> > Avinash
>>> >
>>> > On Mon, Apr 15, 2024 at 11:28 AM, Tolbert, Andy >> > > wrote:
>>> >>
>>> >> Hi Avinash,
>>> >>
>>> >> As far as I understand it, if the underlying keystore/trustore(s)
>>> >> Cassandra is configured for is updated, this *will not* provoke
>>> >> Cassandra to interrupt existing connections, it's just that the new
>>> >> stores will be used for future TLS initialization.
>>> >>
>>> >> Via: 
>>> >> https://cassandra.apache.org/doc/4.1/cassandra/operating/security.html#ssl-certificate-hot-reloading
>>> >>
>>> >> > When the files are updated, Cassandra will reload them and use them 
>>> >> > for subsequent connections
>>> >>
>>> >> I suppose one could do a rolling disablebinary/enablebinary (if it's
>>> >> only client connections) after you roll out a keystore/truststore
>>> >> change as a way of enforcing the existing connections to reestablish.
>>> >>
>>> >> Thanks,
>>> >> Andy
>>> >>
>>> >>
>>> >> On Mon, Apr 15, 2024 at 11:11 AM pabbireddy avinash
>>> >> mailto:pabbireddyavin...@gmail.com>> wrote:
>>> >> >
>>> >> > Dear Community,
>>> >> >
>>> >> > I hope this email finds you well. I am currently testing SSL 
>>> >> > certificate hot reloading on a Cassandra cluster running version 4.1 
>>> >> > and encountered a situation that requires your expertise.
>>> >> >
>>> >> > Here's a summary of the process and issue:
>>> >> >
>>> >> > Reloading Process: We reloaded certificates signed by our in-house 
>>> >> > certificate authority into the cluster, which was initially running 
>>> >> > with self-signed certificates. The reload was done node by node.
>>> >> >
>>> >> > Truststore and Keystore: The truststore and keystore passwords are the 
>>> >> > same across the cluster.
>>> >> >
>>> >> > 

Re: How the write path finds the N nodes to write to?

[Abe Ratnofsky]

> if the replication factor is 3 it just picks the other two nodes following 
> the ring clockwise.

The coordinator for a given mutation is not necessarily a replica (depending on 
whether token-aware routing is used by the client) so it may have to forward to 
RF remote nodes, then wait for the required number of acknowledgements by the 
query's consistency level.

For figuring out the replicas that a write should be forwarded to, see 
StorageProxy.performWrite: 
https://github.com/apache/cassandra/blob/999d263a5ddb9bb33981c39ede3125f199dd61ce/src/java/org/apache/cassandra/service/StorageProxy.java#L1359

Then there ReplicaPlan gets built based on the replication strategy for the 
keyspace, which includes the configuration of full / transient replicas:
https://github.com/apache/cassandra/blob/999d263a5ddb9bb33981c39ede3125f199dd61ce/src/java/org/apache/cassandra/locator/ReplicaPlans.java#L351

WriteResponseHandler is also used on the coordinator path when awaiting 
responses from replicas, to determine when to acknowledge a write back to a 
client.

> On Aug 30, 2023, at 3:46 PM, Gabriel Giussi  wrote:
> 
> I know cassandra uses consistent hashing for choosing the node where a key 
> should go to, and if I understand correctly from this image 
> https://cassandra.apache.org/doc/latest/cassandra/_images/ring.svg 
> if the replication factor is 3 it just picks the other two nodes following 
> the ring clockwise.
> I would like to know if someone can point me to where that is implemented, 
> because I want to implement something similar for the finagle http client, 
> the finagle library already 
> has some implementation of partitioning using consistent hashing, but it 
> doesn't support replication so a key only belongs to a single node, see 
> https://github.com/twitter/util/blob/develop/util-hashing/src/main/scala/com/twitter/hashing/ConsistentHashingDistributor.scala
>  .
> 
> 
> Thanks.

Re: Cassandra p95 latencies

[Abe Ratnofsky]

40ms is definitely higher than expected. Have you run your queries with TRACING enabled to see where the latency is coming from?https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshTracing.html40ms is also a fairly specific duration: https://eklitzke.org/the-caveats-of-tcp-nodelay> On Linux this can cause up to a 40 ms delay when acking packets—AbeOn Aug 10, 2023, at 17:13, MyWorld  wrote:Hi,P95 should not be a problem if rightly designed. Levelled compaction strategy further reduces this, however it consume some resources. For read, caching is also helpful. Can you check your cpu iowait as it could be the reason for delay Regards,AshishOn Fri, 11 Aug, 2023, 04:58 Shaurya Gupta,  wrote:Hi communityWhat is the expected P95 latency for Cassandra Read and Write queries executed with Local_Quorum over a table with 3 replicas ? The queries are done using the partition + clustering key and row size in bytes is not too much, maybe 1-2 KB maximum.Assuming CPU is not a crunch ?We observe those to be 40 ms P95 Reads and same for Writes. This looks very high as compared to what we expected. We are using Cassandra 4.0.Any documentation / numbers will be helpful.Thanks-- Shaurya Gupta

Re: nodetool suddenly failing with "Access denied!"

[Abe Ratnofsky]

Hey Mitch,

The security upgrade schedule that your colleague is working on may well be 
relevant. Is your entire cluster on 3.11.6 or are the failing hosts possibly on 
a newer version?

Abe

> On Feb 26, 2023, at 10:38, Mitch Gitman  wrote:
> 
> 
> We're running Cassandra 3.11.6 on AWS EC2 instances. These clusters have been 
> running for a few years.
> 
> We're suddenly noticing now that on one of our clusters the nodetool command 
> is failing on certain nodes but not on others.
> 
> The failure:
> nodetool: Failed to connect to '...:7199' - SecurityException: 'Access 
> denied! Invalid access level for requested MBeanServer operation.'.
> 
> I suspect that this stems from some colleague I'm not in coordination with 
> recently doing some security upgrades, but that's a bit of an academic matter 
> for now.
> 
> I've compared the jmxremote.access and jvm.options files on a host where 
> nodetool is not working vs. a host where nodetool is working, and no 
> meaningful differences.
> 
> Any ideas? The interesting aspect of this problem is that it is occurring on 
> some nodes in the one cluster but not others.
> 
> I'll update on this thread if I find any solutions on my end.