There could be many reasons for this if it is intermittent. CPU usage + I/O wait status. As read are I/O intensive, your IOPS requirement should be met that time load. Heap issue if CPU is busy for GC only. Network health could be the reason. So better to look system health during that time when it comes.
--------------------------------------------------------------------------------------------------------------------- Atul Saroha *Lead Software Engineer* *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369 Plot # 362, ASF Centre - Tower A, Udyog Vihar, Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA On Tue, Aug 30, 2016 at 5:10 PM, Joseph Tech <jaalex.t...@gmail.com> wrote: > Hi Patrick, > > The nodetool status shows all nodes up and normal now. From OpsCenter > "Event Log" , there are some nodes reported as being down/up etc. during > the timeframe of timeout, but these are Search workload nodes from the > remote (non-local) DC. The RF is 3 and there are 9 nodes per DC. > > Thanks, > Joseph > > On Mon, Aug 29, 2016 at 11:07 PM, Patrick McFadin <pmcfa...@gmail.com> > wrote: > >> You aren't achieving quorum on your reads as the error is explains. That >> means you either have some nodes down or your topology is not matching up. >> The fact you are using LOCAL_QUORUM might point to a datacenter mis-match >> on node count + RF. >> >> What does your nodetool status look like? >> >> Patrick >> >> On Mon, Aug 29, 2016 at 10:14 AM, Joseph Tech <jaalex.t...@gmail.com> >> wrote: >> >>> Hi, >>> >>> We recently started getting intermittent timeouts on primary key queries >>> (select * from table where key=<key>) >>> >>> The error is : com.datastax.driver.core.exceptions.ReadTimeoutException: >>> Cassandra timeout during read query at consistency LOCAL_QUORUM (2 >>> responses were required but only 1 replica >>> a responded) >>> >>> The same query would work fine when tried directly from cqlsh. There are >>> no indications in system.log for the table in question, though there were >>> compactions in progress for tables in another keyspace which is more >>> frequently accessed. >>> >>> My understanding is that the chances of primary key queries timing out >>> is very minimal. Please share the possible reasons / ways to debug this >>> issue. >>> >>> We are using Cassandra 2.1 (DSE 4.8.7). >>> >>> Thanks, >>> Joseph >>> >>> >>> >>> >> >
