Hi,
we're running a 8 node cassandra-0.7.6 cluster, with avg. throughput of 5k
reads/s and almost as much writes/s. The client API is pelops 1.1-0.7.x.
Latencies in the CFs (RecentReadLatencyHistogramMicros) look fine with 99th
percentile at 61ms. However, on the client side, p99 latency is at 1.1s
(seconds!) and we only have 91% below 60ms! So there is a big difference
between the numbers shown in the CF latencies and what the client actually
experiences.
The cluster is not very balanced currently, but nothing indicates a latency of
1.1 seconds. I also see high write latency in the clients, with about 4% taking
50+ ms. Whereas in the RecentWriteLatencyHistogramMicros, 99,9% of the
latencies are below 1ms.
I'm not sure where the additional latency is gained. Is it possible, the
request spends some time in a queue before being processed? If so, is there a
way to optimize that? I already increased core pool size for the ReadStage,
which didn't improve things.
Any other ideas?
Thanks!
Marcel
<hr style="border-color:blue">
<p>chors GmbH
<br><hr style="border-color:blue">
<p>specialists in digital and direct marketing solutions<br>
Haid-und-Neu-Straße 7<br>
76131 Karlsruhe, Germany<br>
www.chors.com</p>
<p>Managing Directors: Dr. Volker Hatz, Markus Plattner<br>Amtsgericht
Montabaur, HRB 15029</p>
<p style="font-size:9px">This e-mail is for the intended recipient only and may
contain confidential or privileged information. If you have received this
e-mail by mistake, please contact us immediately and completely delete it (and
any attachments) and do not forward it or inform any other person of its
contents. If you send us messages by e-mail, we take this as your authorization
to correspond with you by e-mail. E-mail transmission cannot be guaranteed to
be secure or error-free as information could be intercepted, amended,
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
Neither chors GmbH nor the sender accept liability for any errors or omissions
in the content of this message which arise as a result of its e-mail
transmission. Please note that all e-mail communications to and from chors GmbH
may be monitored.</p>
