Never mind. I found the root cause. This has nothing to do with Cassandra and repair. Some web services called by the client caused the problem.
On Fri, Aug 19, 2016 at 11:53 AM, Benyi Wang <bewang.t...@gmail.com> wrote: > I'm using cassandra java driver to access a small cassandra cluster > > * The cluster have 3 nodes in DC1 and 3 nodes in DC2 > * The keyspace is originally created in DC1 only with RF=2 > * The client had good read latency about 40 ms of 99 percentile under 100 > requests/sec (measured at the client side) > * Then keyspace is updated with 2-DC and RF=3 for each DC > * After the repair started (DBA started it, I don't exactly the command), > the client's read latency reached to 2 secs. > * The metric ClientRequest.read.latency.99percentile is still about 4ms > * There were two nodes having 3MB/sec outgoing streaming. > > I'm using Cassandra 2.1.8 and the read consistency is LOCAL_ONE. > > Can you point me some metrics to see what's the bottleneck? > > Thanks >
