Hi, On 11/12/2013 05:29 AM, Aaron Morton wrote: >>> Are you doing large slices or do could you have a lot of tombstones >>> on the rows ? >> don't really know - how can I monitor that? > For tombstones, do you do a lot of deletes ? > Also in v2.0.2 cfstats has this > > Average live cells per slice (last five minutes): 0.0 > Average tombstones per slice (last five minutes): 0.0 > > For large slices you need to check your code. e.g. do you anything > that reads lots of columns or very large columns or lets the user > select how many columns to read? > > The org.apache.cassandra.db.ArrayBackedSortedColumns in the trace back > is used during reads (.e.g. > org.apache.cassandra.db.filter.SliceQueryFilter) thanks for explanation, will try to provide some figures (but unfortunately not from the 2.0.2). > >>> You probably want the heap to be 4G to 8G in size, 10G will >>> encounter longer pauses. >>> Also the size of the new heap may be too big depending on the number >>> of cores. I would recommend trying 800M >> I tried to decrease it first to 384M then to 128M with no change in >> the behaviour. I don't really care extra memory overhead of the cache >> - to be able to actual point to it with objects, but I don't really >> see the reason why it should create/delete those many objects so >> quickly. > Not sure what you changed to 384M. Sorry for the confusion. I meant to say that I tried to decrease row cache size to 384M and then to 128M and the GC times did not change at all (still ~30% of the time). > >>> Shows the heap growing very quickly. This could be due to wide reads >>> or a high write throughput. >> Well, both prg01 and prg02 receive the same load which is about >> ~150-250 (during peak) read requests per seconds and 100-160 write >> requests per second. The only with heap growing rapidly and GC >> kicking in is on nodes with row cache enabled. > > This sounds like on a row cache miss cassandra is reading the whole > row, which happens to be a wide row. I would also guess some writes > are going to the rows and they are getting invalidated out of the row > cache. > > The row cache is not great for rows the update frequently and/or wide > rows. > > How big are the rows ? use nodetool cfstats and nodetool cfhistograms. I will get in touch with the developers and take the data from cf* commands in a few days (I am out of office for some days).
Thanks for the pointers, will get in touch. Cheers Jiri Horky
