I found the problem and posted a patch on https://issues.apache.org/jira/browse/CASSANDRA-3451. If you build with that patch and rerun scrub the exception should go away.
On Thu, Nov 3, 2011 at 2:08 PM, Bryce Godfrey <bryce.godf...@azaleos.com> wrote: > A restart fixed the load numbers, they are back to where I expect them to be > now, but disk utilization is double the load #. I'm also still get the > cfstats exception from any node. > > -----Original Message----- > From: Jonathan Ellis [mailto:jbel...@gmail.com] > Sent: Thursday, November 03, 2011 11:52 AM > To: user@cassandra.apache.org > Subject: Re: Problem after upgrade to 1.0.1 > > Does restarting the node fix this? > > On Thu, Nov 3, 2011 at 1:51 PM, Bryce Godfrey <bryce.godf...@azaleos.com> > wrote: >> Disk utilization is actually about 80% higher than what is reported >> for nodetool ring across all my nodes on the data drive >> >> >> >> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: >> 206.926.1978 | M: 206.849.2477 >> >> >> >> From: Dan Hendry [mailto:dan.hendry.j...@gmail.com] >> Sent: Thursday, November 03, 2011 11:47 AM >> To: user@cassandra.apache.org >> Subject: RE: Problem after upgrade to 1.0.1 >> >> >> >> Regarding load growth, presumably you are referring to the load as >> reported by JMX/nodetool. Have you actually looked at the disk >> utilization on the nodes themselves? Potential issue I have seen: >> http://www.mail-archive.com/user@cassandra.apache.org/msg18142.html >> >> >> >> Dan >> >> >> >> From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] >> Sent: November-03-11 14:40 >> To: user@cassandra.apache.org >> Subject: Problem after upgrade to 1.0.1 >> >> >> >> I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go >> just fine with the rolling upgrade. But now I'm having extreme load >> growth on one of my nodes (and others are growing faster than usual >> also). I attempted to run a cfstats against the extremely large node >> that was seeing 2x the load of others and I get this error below. I'm >> also went into the o.a.c.db.HintedHandoffManager mbean and attempted >> to list pending hints to see if it was growing out of control for some >> reason, but that just times out eventually for any node. I'm not sure what >> to do next with this issue. >> >> >> >> Column Family: HintsColumnFamily >> >> SSTable count: 3 >> >> Space used (live): 12681676437 >> >> Space used (total): 10233130272 >> >> Number of Keys (estimate): 384 >> >> Memtable Columns Count: 117704 >> >> Memtable Data Size: 115107307 >> >> Memtable Switch Count: 66 >> >> Read Count: 0 >> >> Read Latency: NaN ms. >> >> Write Count: 21203290 >> >> Write Latency: 0.014 ms. >> >> Pending Tasks: 0 >> >> Key cache capacity: 3 >> >> Key cache size: 0 >> >> Key cache hit rate: NaN >> >> Row cache: disabled >> >> Compacted row minimum size: 30130993 >> >> Compacted row maximum size: 9223372036854775807 >> >> Exception in thread "main" java.lang.IllegalStateException: Unable to >> compute ceiling for max when histogram overflowed >> >> at >> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram. >> java:170) >> >> at >> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:39 >> 5) >> >> at >> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyS >> tore.java:293) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> ava:39) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> orImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe >> anIntrospector.java:93) >> >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe >> anIntrospector.java:27) >> >> at >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.ja >> va:208) >> >> at >> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65 >> ) >> >> at >> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:21 >> 6) >> >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Def >> aultMBeanServerInterceptor.java:666) >> >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.jav >> a:638) >> >> at >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectio >> nImpl.java:1404) >> >> at >> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnection >> Impl.java:72) >> >> at >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run( >> RMIConnectionImpl.java:1265) >> >> at >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RM >> IConnectionImpl.java:1360) >> >> at >> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnecti >> onImpl.java:600) >> >> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown >> Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> orImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> at >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:53 >> 5) >> >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport >> .java:790) >> >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport. >> java:649) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu >> tor.java:886) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. >> java:908) >> >> at java.lang.Thread.run(Thread.java:662) >> >> >> >> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: >> 206.926.1978 | M: 206.849.2477 >> >> >> >> No virus found in this incoming message. >> Checked by AVG - www.avg.com >> Version: 9.0.920 / Virus Database: 271.1.1/3993 - Release Date: >> 11/03/11 >> 03:39:00 > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
