I'm using cassandra java driver to access a small cassandra cluster * The cluster have 3 nodes in DC1 and 3 nodes in DC2 * The keyspace is originally created in DC1 only with RF=2 * The client had good read latency about 40 ms of 99 percentile under 100 requests/sec (measured at the client side) * Then keyspace is updated with 2-DC and RF=3 for each DC * After the repair started (DBA started it, I don't exactly the command), the client's read latency reached to 2 secs. * The metric ClientRequest.read.latency.99percentile is still about 4ms * There were two nodes having 3MB/sec outgoing streaming.
I'm using Cassandra 2.1.8 and the read consistency is LOCAL_ONE. Can you point me some metrics to see what's the bottleneck? Thanks