Chris, Can you get me a stack dump of one of the busy nodes (kill -3)?
Gary On Thu, Jun 3, 2010 at 22:50, Chris Goffinet <[email protected]> wrote: > We're seeing this as well. We were testing with a 40+ node cluster on the > latest 0.6 branch from few days ago. > > -Chris > > On Jun 3, 2010, at 9:55 PM, Lu Ming wrote: > >> >> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to >> cassandra to 0.6.2 yesterday. >> But today I find six cassandra nodes have high CPU usage more than 400% in >> my 8-core CPU sever. >> The worst one is more than 760%. It is very serious. >> >> I use jvisualvm to watch the worst node, and I found that there are many >> running threads named "thread-xxx" >> the status of other threads is waiting and sleeping. >> >> "Thread-130" - Thread t...@240 >> java.lang.Thread.State: RUNNABLE >> at sun.misc.Unsafe.setMemory(Native Method) >> at sun.nio.ch.Util.erase(Util.java:202) >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> Locked ownable synchronizers: >> - None >> >> "Thread-126" - Thread t...@236 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:200) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> - locked java.lang.obj...@10808561 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> "Thread-119" - Thread t...@229 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.NativeThread.current(Native Method) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182) >> - locked java.lang.obj...@65b4abbd >> - locked java.lang.obj...@38773975 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> > >
