[
https://issues.apache.org/jira/browse/CASSANDRA-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023225#comment-13023225
]
Thibaut commented on CASSANDRA-2543:
------------------------------------
It flaps between one core and multiple cores. There was one thread taking 100%
cpu, and the other ones flapping as you said.
How can the gc run constantly if there are still 1.5 Gig free according to
nodetool info and I did shut down our application, so there shouldn't be any
requests (or just a very few read requests) at all? But the localhost info heap
info changed constantly (also grew close to the limit).
There also many small tables which haven't yet been compacted.
dynamic snitch is indeed disabled (because of data locality), but I will enable
it and then bring the node to life again.
> Node not responding, bringing down cluster, marked as up
> --------------------------------------------------------
>
> Key: CASSANDRA-2543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2543
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Thibaut
> Fix For: 0.7.6
>
> Attachments: jstack
>
>
> I have one node which constantly hangs and brings done the entire cluster
> (not giving any answers).
> If I restart the node, the node will hang after a certain number of time. I
> have no indication
> It's marked as up when executing the nodetool ring command.
> Executing the ring command on the node itself (without any traffic on the
> cluster) takes at least 2 minutes to execute. The node takes about 50%-100%
> of cpu over all cpus.
> Netstats doesn't list anything interesting:
> /software/cassandra/bin/nodetool -h localhost netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 51064
> Responses n/a 0 530479
> I attached the jstack of the node. There are no indications that the node has
> faulty hardware.
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
> -Xms5254M -Xmx5254M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=8080
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes -cp
> /software/cassandra/bin/../conf:/software/cassandra/bin/../build/classes:/software/cassandra/bin/../lib/antlr-3.1.3.jar:/software/cassandra/bin/../lib/apache-cassandra-0.7.4.jar:/software/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/software/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/software/cassandra/bin/../lib/commons-cli-1.1.jar:/software/cassandra/bin/../lib/commons-codec-1.2.jar:/software/cassandra/bin/../lib/commons-collections-3.2.1.jar:/software/cassandra/bin/../lib/commons-lang-2.4.jar:/software/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:/software/cassandra/bin/../lib/guava-r05.jar:/software/cassandra/bin/../lib/high-scale-lib.jar:/software/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/software/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/software/cassandra/bin/../lib/jetty-6.1.21.jar:/software/cassandra/bin/../lib/jetty-util-6.1.21.jar:/software/cassandra/bin/../lib/jline-0.9.94.jar:/software/cassandra/bin/../lib/json-simple-1.1.jar:/software/cassandra/bin/../lib/jug-2.0.0.jar:/software/cassandra/bin/../lib/libthrift-0.5.jar:/software/cassandra/bin/../lib/log4j-1.2.16.jar:/software/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/software/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/software/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/software/cassandra/bin/../lib/snakeyaml-1.6.jar
> org.apache.cassandra.thrift.CassandraDaemon
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
