High CPU can be http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs
memory usage looks ok http://wiki.apache.org/cassandra/FAQ#mmap Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 3/07/2012, at 6:49 PM, Tamar Fraenkel wrote: > Hi! > I have a Cassandra cluster on Amazon EC2 Datastax AMIs with 3 nodes and > replication factor of 2. > As of July 1st the cluster is very slow and seems to be loaded. > > Running top I get: > > top - 06:40:58 up 99 days, 21:30, 2 users, load average: 12.45, 13.37, 14.01 > Tasks: 102 total, 1 running, 101 sleeping, 0 stopped, 0 zombie > Cpu(s): 21.0%us, 9.8%sy, 0.0%ni, 2.2%id, 0.0%wa, 0.4%hi, 0.4%si, 66.3%st > Mem: 7647812k total, 7135752k used, 512060k free, 60668k buffers > Swap: 0k total, 0k used, 0k free, 4234008k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 18729 cassandr 20 0 8866m 2.5g 275m S 108 34.0 1415:37 jsvc > 21798 root 20 0 478m 217m 9760 S 44 2.9 1743:47 java > 3 root 20 0 0 0 0 S 20 0.0 564:59.09 ksoftirqd/0 > > > > These are the cassandra processes > ps -ef | grep cassandra > root 18727 1 0 Jul02 ? 00:00:00 jsvc.exec -user cassandra > -home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/cassandra.pid > -errfile &1 -outfile /var/log/cassandra/output.log -cp > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.8.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar > -Dlog4j.configuration=log4j-server.properties > -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof > -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea > -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Djava.rmi.server.hostname=10.34.158.33 > -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > org.apache.cassandra.thrift.CassandraDaemon > 108 18729 18727 99 Jul02 ? 23:26:48 jsvc.exec -user cassandra > -home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/cassandra.pid > -errfile &1 -outfile /var/log/cassandra/output.log -cp > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.8.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar > -Dlog4j.configuration=log4j-server.properties > -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof > -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea > -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Djava.rmi.server.hostname=10.34.158.33 > -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > org.apache.cassandra.thrift.CassandraDaemon > > Attached is the cfstats from node #1. > Yesterday I restarted all the cluster, and it improved things a bit, but > still memory consumption is high and things are slower than they used to be + > many GC runs. > OpsCenter is running on node#1 and probes on other two nodes. > > Any help would be greatly appreciated! > Thanks, > > Tamar Fraenkel > Senior Software Engineer, TOK Media > > <tokLogo.png> > > ta...@tok-media.com > Tel: +972 2 6409736 > Mob: +972 54 8356490 > Fax: +972 2 5612956 > > > > <cfstats4>
