Hi,
*On short:*
I use cassandra 3.0.9 in a cluster of 6 nodes.
1. I create a keyspace called test:
CREATE KEYSPACE business WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
2. I create table called test:
CREATE TABLE test.test (
test_id bigint,
test_value text
PRIMARY KEY (test_id)
)
3. I insert test_id=23 and test_value=some very large string/html (like
406088 chars utf8).
4. I query for test_id=35 and I get timeout (even with clqsh
--request-timeout=3600).......
5. If I run the above on an existing cassandra cluster with cassa 2.0 the
select returns instantly....The Java heap size is 8GB and in JMX I see max
4GB used of these 8 GB in the new cluster....
*Detailed:*
The above was just a test. The real scenario is:
I migrated some tables from an old cassa (2.0) cluster with 9 nodes into
another with 6 nodes and with cassa 3.0.9 and there was a lot of
problems....
I have a table like this:
CREATE TABLE table (
id text,
ts text,
score decimal,
type text,
values text,
PRIMARY KEY (id, ts)
) WITH CLUSTERING ORDER BY (ts DESC)
and the following query (which returns instantly):
SELECT * FROM keyspace.table WHERE id='someId' AND ts IN
('2017-10-15','2017-10-16','2017-10-17','2017-10-18','2017-10-19','2017-10-20','2017-10-21','2017-10-22','2017-10-23','2017-10-24','2017-10-25','2017-10-26','2017-10-27','2017-10-28','2017-10-29','2017-10-30','2017-10-31','2017-11-01','2017-11-02','2017-11-03','2017-11-04','2017-11-05','2017-11-06');
*If I add another day in the IN clause, the response never comes (even
after 10 minutes!!!):*
SELECT * FROM keyspace.table WHERE id='someId' AND ts IN
('2017-10-15','2017-10-16','2017-10-17','2017-10-18','2017-10-19','2017-10-20','2017-10-21','2017-10-22','2017-10-23','2017-10-24','2017-10-25','2017-10-26','2017-10-27','2017-10-28','2017-10-29','2017-10-30','2017-10-31','2017-11-01','2017-11-02','2017-11-03','2017-11-04','2017-11-05','2017-11-06',
*'2017-11-07'*);
*The 'values' column may have large json data. *
I managed to trace one of the timeouts by looking into system_trace
keyspace. Please look into the attached image and see the last process took
10 minutes!!!
I think there is some size limit somewhere because in* the IN clause *if I
have 23 params it works(under 1 second), but with more(1+) it fails. The
rows are the same size (same json size on all). In node2 of those 6 it
works with 24 params. In node1 and node3 no. The other nodes I haven't
checked yet.
I saw no concluding logs except this one from cassa's debug.log (in the
moment of the timeout or very close to that):
*DEBUG [Thrift:2608] 2017-11-15 13:48:05,611 ReadCallback.java:126 - Timed
out; received 0 of 1 responses*
I think this problem has the same root cause as the one from the test
(large html text) and it is related to some memory limit by code somewhere.
Thank you,
Alex.
[image: screenshot.png]
