If I were you, I'd do both. If you're trying to build a multi-tenanted system, it's probably a better idea to include tenant ID as the partition key of every cross-tenant table. You can easily run Cassandra with a 4 gig heap, but I'd never plan on doing so for a production use except for very small databases.
On Tue, May 24, 2016 at 11:54 AM Justin Lin <linjianfeng...@gmail.com> wrote: > so i guess i have to 1) increase the heap size or 2) reduce the number of > keyspaces/column families. > > Thanks for you confirmation. > > On Tue, May 24, 2016 at 10:08 AM, Eric Stevens <migh...@gmail.com> wrote: > >> Large numbers of tables is generally recommended against. Each table has >> a fixed on-heap memory overhead, and by your description it sounds like you >> might have as many as 12,000 total tables when you start running into >> trouble. >> >> With such a small heap to begin with, you've probably used up most of the >> available heap just with managing the tables. This is supported by the big >> STW pauses you're observing in Cassandra's logs. >> >> On Tue, May 24, 2016 at 11:04 AM Justin Lin <linjianfeng...@gmail.com> >> wrote: >> >>> We are exploring cassandra's limit by creating a lot of keyspaces with >>> moderate number of column families (roughly 40 - 50) per keyspace and we >>> have a problem after we reach certain amount of keyspaces, that cqlsh >>> starts to time out when connecting to cassandra. >>> >>> This is our cassandra setup. We have one cassandra running locally and >>> we assign 4GB memory to jvm, set the memtable_allocation_type to be onheap >>> and use default memtable_heap_space_in_mb, which i believe is 2GB. The >>> cassandra version is 2.1.9. >>> >>> So after we create more than 250 keyspaces, cqlsh starts to times out >>> when connecting to cassandra in most case. (Sometimes it still can connect >>> to cassandra). And from cassandra log, we can see it takes roughly 3 >>> seconds to do gc when there is an incoming connection. And the gc is the >>> only difference between the timeout connection and the successful >>> connection. So we suspect this Stop-The-World GC might block the connection >>> until it times out. This is the log that i think is relevant. >>> >>> INFO 20160524-060930.028882 :: Initializing >>> sandbox_20160524_t06_09_18.table1 >>> >>> INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 >>> Eden Space: 222298112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; >>> >>> INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old >>> Gen: 3034119696 -> 2290099032; >>> >>> We suspect the issue might relate to the reported bug as well: >>> https://issues.apache.org/jira/browse/CASSANDRA-9291 >>> but not really sure about it. >>> >>> Sorry for the setup, so my question is >>> 1) is the connection timeout related to gc or the tomestone in >>> system.schame_keyspaces table? >>> 2) how can we fix this problem? >>> 3) I did some tests by dropping a bunch of keyspaces after timing out >>> and it seems to fix the problem as i never got another time out. Is this >>> the only way to fix it? >>> >>> Thanks a lot for your help. >>> >>> -- >>> come on >>> >> > > > -- > come on >
