Hi Team,
We are struggling with a problem related to cassandra counts, after backup
and restore of the cluster. Aaron Morton has suggested to send this to user
list, so some one of the list will be able to help me.
We are have a rest api to talk to cassandra and one of our query which
fetches count is creating problems for us.
We have done backup and restore and copied all the data to new cluster. We
have done nodetool refresh on the tables, and did the nodetool repair as
well.
However, one of our key API call is returning inconsistent results. The
result count is 0 in the first call and giving the actual values for later
calls. The query frequency is bit high and failure rate has also raised
considerably.
1) The count query has partition keys in it. Didnt see any read timeout or
any errors from api logs.
2) This is how our code of creating session looks.
val poolingOptions = new PoolingOptions
poolingOptions
.setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
.setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
.setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
.setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
val builtCluster = clusterBuilder.withCredentials(username, password)
.withPoolingOptions(poolingOptions)
.build()
val cassandraSession = builtCluster.get.connect()
val preparedStatement =
cassandraSession.prepare(statement).setConsistencyLevel(ConsistencyLevel.QUORUM)
cassandraSession.execute(preparedStatement.bind(args :_*))
Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
date_column_of_clustering_key>=?
3) Cluster configuration:
6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
machine is equipped with 16 Cores and 64 GB Ram.
Replication factor is 3, and write consistency is ONE and read
consistency is QUORUM.
4) cassandra is never down on any machine
5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
6) nodetool tpstats shows no read failures, and no other failures.
7) Do not see any other issues from system.log of cassandra. We just see
few warnings as below.
Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 -
88 prepared statements discarded in the last minute because cache limit
reached (32 MB)
The first api call returns 0 and the api calls later gives right values.
Please let me know, if any other details needed.
Could you please have a look at this issue once and kindly give me your
inputs? This issue literally broke the confidence on Cassandra from our
business team.
Your inputs will be really helpful.
Thank You,
Regards,
Srini
