Using the following cmd - sudo su cassandra -c "jstat -gccause 4162”
Gave this (not sure if it will present correctly on the webpage) But during load we only see data move between the survivor spaces in Eden and the old gen never really grows S0 S1 E O M CCS YGC YGCT FGC FGCT GCT LGCC GCC 0.00 70.57 48.69 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 49.02 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 78.38 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 83.99 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 90.07 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 90.30 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC 0.00 70.57 90.40 26.29 97.86 96.62 119 14.087 2 0.100 14.187 Allocation Failure No GC From: Walsh, Stephen [mailto:stephen.wa...@aspect.com] Sent: 02 October 2015 09:32 To: user@cassandra.apache.org Subject: RE: Consistency Issues Sorry for the late reply, I ran the nodetool resetlocalschema on all nodes but in the end it just removed all the schemas and crashed the applications. I need to reset and try again. I’ll try get you the gc stats today ☺ From: Jonathan Haddad [mailto:j...@jonhaddad.com] Sent: 01 October 2015 16:01 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Consistency Issues You say that you don't think GC is your issue... but did you actually check? The reasons you suggest aren't very convincing. Can you provide your GC settings, and take a look at jstat --gccause? http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html#gccause_option On Thu, Oct 1, 2015 at 4:50 AM Walsh, Stephen <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: If you’re looking for the clean-up of the old gen in the jvm heap, it doesn’t happen. We have a new gen turning 15 times before its pushed to old gen. Seems all our data only has a TTL of 10 seconds – very little data is sent to the old gen. Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is our issue. I’m more worried about error messages in the Cassandra log file that state. UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=cf411b50-6785-11e5-a435-e7be20c92086 and cassandra OutboundTcpConnection.java:313 - error writing to Connection. But I really need to understand this best practice that was mentioned (on number of CF’s) by Jack Krupansky. Anyone more information on this? Many thanks for all your help guys keep it coming ☺ Steve From: Ricardo Sancho [mailto:sancho.rica...@gmail.com<mailto:sancho.rica...@gmail.com>] Sent: 01 October 2015 09:39 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: RE: Consistency Issues Can you tell us how much time your gcs are taking? Do you see any especially long ones? On 1 Oct 2015 09:37, "Walsh, Stephen" <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: There is no load balancer in front of Cassandra, it’s in front of our application. Everyone seems hung up on this point? But it’s not the root causing of the inconsistency issue. Can anyone verify the best practice for number of CF’s? From: Robert Coli [mailto:rc...@eventbrite.com<mailto:rc...@eventbrite.com>] Sent: 30 September 2015 18:45 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Consistency Issues On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: We never had these issue with our first run. Its only when we added another 25% of writes. As Jack said, you are probably pushing your GC over a threshold, leading to long pause times and inability to meet quorum. As Sebastian said, you probably shouldn't need a load balancer in front of Cassandra. =Rob This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments.
