I've recently added compaction rate (in bytes / second) to my monitors for cassandra and am seeing some odd values. I wasn't expecting the values for TotalBytesCompacted to sometimes decrease from one reading to the next. It seems that the value should be monotonically increasing while a server is running -- obviously it would start again at 0 when the server is restarted or if the counter rolls over (unlikely for a 64 bit long).
Below are two samples taken 60 seconds apart: the value decreased by 2,954,369,012 between the two readings. reported_metric=[timestamp:1349476449, status:200, request:[mbean:org.apache.cassandra.db:type=CompactionManager, attribute:TotalBytesCompacted, type:read], value:7548675470069] previous_metric=[timestamp:1349476389, status:200, request:[mbean:org.apache.cassandra.db:type=CompactionManager, attribute:TotalBytesCompacted, type:read], value:7551629839081] I briefly looked at the code for CompactionManager and a few related classes and don't see anyplace that is performing subtraction explicitly; however, there are many additions of signed long values that are not validated and could conceivably contain a negative value thus causing the totalBytesCompacted to decrease. It's interesting to note that the all of the differences I've seen so far are more than the overflow value of a signed 32 bit value. The OS (CentOS 5.7) and sun java vm (1.6.0_29) are both 64 bit. JNA is enabled. Is this expected and normal? If so, what is the correct interpretation of this metric? I'm seeing the negatives values a few times per hour when reading it once every 60 seconds. -Bryan
