RE: Help with sudden spike in read requests
If it’s a legacy write table why does it write 10% of the time? Maybe it’s the design of the big legacy table you mentioned. It could be so many things. Is it the same time of day? Same days of the week or month? Are there analytics run at that time? What are you using for monitoring and how did you find out it was happening? Is this a DSE cluster or OSS Cassandra cluster? Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 10:48 AM To: user@cassandra.apache.org Subject: Re: Help with sudden spike in read requests We migrated one of the application from on-Prem to aws; the queries are very light, more like registration info; Queries from the new app is via pk of data type, “text”, no cc (this table has about 200 rows; however the legacy table (more like reference table) has several million rows, about 800 sstables per node, using lcs (9:1, read-write ratio) Subroto On Feb 1, 2019, at 10:33 AM, Kenneth Brotman wrote: Do you have that many queries? You could just review them and your data model to see if there was an error of some kind. How long has it been happening? What changed since it started happening? Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 10:13 AM To: user@cassandra.apache.org Subject: Re: Help with sudden spike in read requests Vnode is 256 C*: 3.0.15 on m4.4xlarge gp2 vol There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers If this event is triggered by some bad query/queries, what is the best way to trap it? Subroto On Feb 1, 2019, at 8:55 AM, Kenneth Brotman wrote: If you had a query that went across the partitions and especially if you had vNodes set high, that would do it. Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 8:45 AM To: User cassandra.apache.org <http://cassandraapache.org> Subject: Help with sudden spike in read requests In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day. I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event? Thanks! Subroto
Re: Help with sudden spike in read requests
We migrated one of the application from on-Prem to aws; the queries are very light, more like registration info; Queries from the new app is via pk of data type, “text”, no cc (this table has about 200 rows; however the legacy table (more like reference table) has several million rows, about 800 sstables per node, using lcs (9:1, read-write ratio) Subroto > On Feb 1, 2019, at 10:33 AM, Kenneth Brotman > wrote: > > Do you have that many queries? You could just review them and your data > model to see if there was an error of some kind. How long has it been > happening? What changed since it started happening? > > Kenneth Brotman > > From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] > Sent: Friday, February 01, 2019 10:13 AM > To: user@cassandra.apache.org > Subject: Re: Help with sudden spike in read requests > > Vnode is 256 > C*: 3.0.15 on m4.4xlarge gp2 vol > > There are 2 more DCs on bare metal (raid 10 and older machines) attached to > this cluster and we have not seen this behavior on on-prem servers > > If this event is triggered by some bad query/queries, what is the best way to > trap it? > > Subroto > > On Feb 1, 2019, at 8:55 AM, Kenneth Brotman > wrote: > > If you had a query that went across the partitions and especially if you had > vNodes set high, that would do it. > > Kenneth Brotman > > From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] > Sent: Friday, February 01, 2019 8:45 AM > To: User cassandra.apache.org > Subject: Help with sudden spike in read requests > > In our production cluster, we observed sudden spike (over 160 MB/s) in read > requests on *all* Cassandra nodes for a very short period (less than a min); > this event happens few times a day. > > I am not able to get to the bottom of this issue, nothing interesting in > system.log or from app level; repair was not running > > Does anyone have any thoughts on what could have triggered this event? Under > what condition C* (if it is tied to c*) will trigger this type of event? > > Thanks! > > Subroto
RE: Help with sudden spike in read requests
Do you have that many queries? You could just review them and your data model to see if there was an error of some kind. How long has it been happening? What changed since it started happening? Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 10:13 AM To: user@cassandra.apache.org Subject: Re: Help with sudden spike in read requests Vnode is 256 C*: 3.0.15 on m4.4xlarge gp2 vol There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers If this event is triggered by some bad query/queries, what is the best way to trap it? Subroto On Feb 1, 2019, at 8:55 AM, Kenneth Brotman wrote: If you had a query that went across the partitions and especially if you had vNodes set high, that would do it. Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 8:45 AM To: User cassandra.apache.org <http://cassandraapache.org> Subject: Help with sudden spike in read requests In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day. I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event? Thanks! Subroto
Re: Help with sudden spike in read requests
Vnode is 256 C*: 3.0.15 on m4.4xlarge gp2 vol There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers If this event is triggered by some bad query/queries, what is the best way to trap it? Subroto > On Feb 1, 2019, at 8:55 AM, Kenneth Brotman > wrote: > > If you had a query that went across the partitions and especially if you had > vNodes set high, that would do it. > > Kenneth Brotman > > From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] > Sent: Friday, February 01, 2019 8:45 AM > To: User cassandra.apache.org > Subject: Help with sudden spike in read requests > > In our production cluster, we observed sudden spike (over 160 MB/s) in read > requests on *all* Cassandra nodes for a very short period (less than a min); > this event happens few times a day. > > I am not able to get to the bottom of this issue, nothing interesting in > system.log or from app level; repair was not running > > Does anyone have any thoughts on what could have triggered this event? Under > what condition C* (if it is tied to c*) will trigger this type of event? > > Thanks! > > Subroto
RE: Help with sudden spike in read requests
If you had a query that went across the partitions and especially if you had vNodes set high, that would do it. Kenneth Brotman From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID] Sent: Friday, February 01, 2019 8:45 AM To: User cassandra.apache.org Subject: Help with sudden spike in read requests In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day. I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event? Thanks! Subroto
