Re: Understanding Blocked and All Time Blocked columns in tpstats

2018-03-23 Thread John Sanda
We do small inserts. For a modest size environment we do about 90,000 inserts every 30 seconds. For a larger environment, we could be doing 300,000 or more inserts every 30 seconds. In earlier versions of the project, each insert was a separate request as each insert targets a different partition.

Re: Understanding Blocked and All Time Blocked columns in tpstats

2018-03-23 Thread Chris Lohfink
Increasing queue would increase the number of requests waiting. It could make GCs worse if the requests are like large INSERTs, but for a lot of super tiny queries it helps to increase queue size (to a point). Might want to look into what and how queries are being made, since there are possibly

Re: Understanding Blocked and All Time Blocked columns in tpstats

2018-03-23 Thread John Sanda
Thanks for the explanation. In the past when I have run into problems related to CASSANDRA-11363, I have increased the queue size via the cassandra.max_queued_native_transport_requests system property. If I find that the queue is frequently at capacity, would that be an indicator that the node is

Re: Understanding Blocked and All Time Blocked columns in tpstats

2018-03-23 Thread Chris Lohfink
It blocks the caller attempting to add the task until theres room in queue, applying back pressure. It does not reject it. It mimics the behavior from pre-SEP DebuggableThreadPoolExecutor's RejectionExecutionHandler that the other thread pools use (exception on sampling/trace which just throw

Understanding Blocked and All Time Blocked columns in tpstats

2018-03-22 Thread John Sanda
I have been doing some work on a cluster that is impacted by https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through the ticket prompted me to take a closer look at org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14 code. I am a little confused about the Blocked