I think I narrowed down the constant Full GC due to accumulation of large
partitions as a result of upgrade to 3.11.
And how the large partitions were produced maybe related to
https://issues.apache.org/jira/browse/CASSANDRA-11887

I started to see duplicate records in one of the tables which probably
expanded the size of already semi large enough partitions to well over 300,
500MB and some cases 800MB.  My system.log started to report Writing large
partition during compaction quite frequently after the upgrade.

On CASSANDRA-11887, the suggestion is to run nodetool scrub which I will
try next to see if dup records are purged out.

Is anyone else familiar with this problem?  It says scrub fixes the
duplicate record issue but I'm not sure it would be that simple.  I think
the duplicate record issue is the cause of the problem but I don't see any
open issue on this other than 11887.

On Tue, Aug 28, 2018 at 12:14 PM ZAIDI, ASAD A <az1...@att.com> wrote:

> You may want to check if coincidentally you’re having expired cells in
> heap. GC log should be able to tell you OR look for tombstones in
> system.log file. See your compactions are under control and normal.  This
> may not be related to upgrade at all!
>
>
>
>
>
> *From:* Pradeep Chhetri [mailto:prad...@stashaway.com]
> *Sent:* Tuesday, August 28, 2018 3:32 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Upgrade from 2.1 to 3.11
>
>
>
> You may want to try upgrading to 3.11.3 instead which has some memory
> leaks fixes.
>
>
>
> On Tue, Aug 28, 2018 at 9:59 AM, Mun Dega <mundeg...@gmail.com> wrote:
>
> I am surprised that no one else ran into any issues with this version.  GC
> can't catch up fast enough and there is constant Full GC taking place.
>
>
>
> The result? unresponsive nodes makeing entire cluster unusable.
>
>
>
> Any insight on this issue from anyone that is using this version would be
> appreciated.
>
>
>
> Ma
>
>
>
> On Fri, Aug 24, 2018, 04:30 Mohamadreza Rostami <
> mohamadrezarosta...@gmail.com> wrote:
>
> You have very large heap,it’s take most of  cpu time in GC stage.you
> should in maximum set heap on 12GB and enable row cache to your cluster
> become faster.
>
> On Friday, 24 August 2018, Mun Dega <mundeg...@gmail.com> wrote:
>
> 120G data
>
> 28G heap out of 48 on system
>
> 9 node cluster, RF3
>
>
>
> On Thu, Aug 23, 2018, 17:19 Mohamadreza Rostami <
> mohamadrezarosta...@gmail.com> wrote:
>
> Hi,
>
> How much data do you have? How much RAM do your servers have? How much do
> you have a heep?
>
> On Thu, Aug 23, 2018 at 10:14 PM Mun Dega <mundeg...@gmail.com> wrote:
>
> Hello,
>
>
>
> We recently upgraded from Cassandra 2.1 to 3.11.2 on one cluster.  The
> process went OK including upgradesstable but we started to experience high
> latency for r/w, occasional OOM and long GC pause after.
>
>
>
> For the same cluster with 2.1, we didn't have any issues like this.  We
> also kept server specs, heap, all the same in post upgrade
>
>
>
> Has anyone else had similar issues going to 3.11 and what are the major
> changes that could have such a major setback in the new version?
>
>
>
> Ma Dega
>
>
>

Reply via email to