We are exploring cassandra's limit by creating a lot of keyspaces with moderate number of column families (roughly 40 - 50) per keyspace and we have a problem after we reach certain amount of keyspaces, that cqlsh starts to time out when connecting to cassandra.
This is our cassandra setup. We have one cassandra running locally and we assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and use default memtable_heap_space_in_mb, which i believe is 2GB. The cassandra version is 2.1.9. So after we create more than 250 keyspaces, cqlsh starts to times out when connecting to cassandra in most case. (Sometimes it still can connect to cassandra). And from cassandra log, we can see it takes roughly 3 seconds to do gc when there is an incoming connection. And the gc is the only difference between the timeout connection and the successful connection. So we suspect this Stop-The-World GC might block the connection until it times out. This is the log that i think is relevant. INFO 20160524-060930.028882 :: Initializing sandbox_20160524_t06_09_18.table1 INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 Eden Space: 222298112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old Gen: 3034119696 -> 2290099032; We suspect the issue might relate to the reported bug as well: https://issues.apache.org/jira/browse/CASSANDRA-9291 but not really sure about it. Sorry for the setup, so my question is 1) is the connection timeout related to gc or the tomestone in system.schame_keyspaces table? 2) how can we fix this problem? 3) I did some tests by dropping a bunch of keyspaces after timing out and it seems to fix the problem as i never got another time out. Is this the only way to fix it? Thanks a lot for your help. -- come on
