We have the same problem.
2013/11/5 Jiri Horky <ho...@avast.com> > Hi there, > > we are seeing extensive memory allocation leading to quite long and > frequent GC pauses when using row cache. This is on cassandra 2.0.0 > cluster with JNA 4.0 library with following settings: > > key_cache_size_in_mb: 300 > key_cache_save_period: 14400 > row_cache_size_in_mb: 1024 > row_cache_save_period: 14400 > commitlog_sync: periodic > commitlog_sync_period_in_ms: 10000 > commitlog_segment_size_in_mb: 32 > > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G > -Xmn1024M -XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/data2/cassandra-work/instance-1/cassandra-1383566283-pid1893.hprof > -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark > > We have disabled row cache on one node to see the difference. Please > see attached plots from visual VM, I think that the effect is quite > visible. I have also taken 10x "jmap -histo" after 5s on a affected > server and plotted the result, attached as well. > > I have taken a dump of the application when the heap size was 10GB, most > of the memory was unreachable, which was expected. The majority was used > by 55-59M objects of HeapByteBuffer, byte[] and > org.apache.cassandra.db.Column classes. I also include a list of inbound > references to the HeapByteBuffer objects from which it should be visible > where they are being allocated. This was acquired using Eclipse MAT. > > Here is the comparison of GC times when row cache enabled and disabled: > > prg01 - row cache enabled > - uptime 20h45m > - ConcurrentMarkSweep - 11494686ms > - ParNew - 14690885 ms > - time spent in GC: 35% > prg02 - row cache disabled > - uptime 23h45m > - ConcurrentMarkSweep - 251ms > - ParNew - 230791 ms > - time spent in GC: 0.27% > > I would be grateful for any hints. Please let me know if you need any > further information. For now, we are going to disable the row cache. > > Regards > Jiri Horky > -- Atenciosamente, Sávio S. Teles de Oliveira voice: +55 62 9136 6996 http://br.linkedin.com/in/savioteles Mestrando em Ciências da Computação - UFG Arquiteto de Software Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG
