so i guess i have to 1) increase the heap size or 2) reduce the number of keyspaces/column families.
Thanks for you confirmation. On Tue, May 24, 2016 at 10:08 AM, Eric Stevens <migh...@gmail.com> wrote: > Large numbers of tables is generally recommended against. Each table has > a fixed on-heap memory overhead, and by your description it sounds like you > might have as many as 12,000 total tables when you start running into > trouble. > > With such a small heap to begin with, you've probably used up most of the > available heap just with managing the tables. This is supported by the big > STW pauses you're observing in Cassandra's logs. > > On Tue, May 24, 2016 at 11:04 AM Justin Lin <linjianfeng...@gmail.com> > wrote: > >> We are exploring cassandra's limit by creating a lot of keyspaces with >> moderate number of column families (roughly 40 - 50) per keyspace and we >> have a problem after we reach certain amount of keyspaces, that cqlsh >> starts to time out when connecting to cassandra. >> >> This is our cassandra setup. We have one cassandra running locally and we >> assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and >> use default memtable_heap_space_in_mb, which i believe is 2GB. The >> cassandra version is 2.1.9. >> >> So after we create more than 250 keyspaces, cqlsh starts to times out >> when connecting to cassandra in most case. (Sometimes it still can connect >> to cassandra). And from cassandra log, we can see it takes roughly 3 >> seconds to do gc when there is an incoming connection. And the gc is the >> only difference between the timeout connection and the successful >> connection. So we suspect this Stop-The-World GC might block the connection >> until it times out. This is the log that i think is relevant. >> >> INFO 20160524-060930.028882 :: Initializing >> sandbox_20160524_t06_09_18.table1 >> >> INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 Eden >> Space: 222298112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; >> >> INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old >> Gen: 3034119696 -> 2290099032; >> >> We suspect the issue might relate to the reported bug as well: >> https://issues.apache.org/jira/browse/CASSANDRA-9291 >> but not really sure about it. >> >> Sorry for the setup, so my question is >> 1) is the connection timeout related to gc or the tomestone in >> system.schame_keyspaces table? >> 2) how can we fix this problem? >> 3) I did some tests by dropping a bunch of keyspaces after timing out and >> it seems to fix the problem as i never got another time out. Is this the >> only way to fix it? >> >> Thanks a lot for your help. >> >> -- >> come on >> > -- come on
