You could also be hitting CASSANDRA-11739, which was fixed on 3.0.7 and could potentially cause OOMs for long-running repairs.
2016-06-20 13:26 GMT-03:00 Robert Stupp <sn...@snazy.de>: > One possibility might be CASSANDRA-11206 (Support large partitions on the > 3.0 sstable format), which reduces heap usage for other operations (like > repair, compactions) as well. > You can verify that by setting column_index_cache_size_in_kb in c.yaml to > a really high value like 10000000 - if you see the same behaviour in 3.7 > with that setting, there’s not much you can do except upgrading to 3.7 as > that change went into 3.6 and not into 3.0.x. > > — > Robert Stupp > @snazy > > On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: > > Hi All, > > We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. > There has been a consistent issue with nodetool repair for a while and we > have tried issuing it with multiple options --pr, --local as well, > sometimes node went down with Out of Memory error and at times nodes did > stopped connecting any connection, even jmx nodetool commands. > > On trying with same data on 3.7 Repair Ran successfully without > encountering any of the above mentioned issues. I then tried increasing > heap to 16GB on 3.0.3 and repair ran successfully. > > I then analyzed memory usage during nodetool repair for 3.0.3(16GB heap) > vs 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times, whereas 3.7 > spiked between 1-4.5 GB while repair runs. As they ran on same dataset > and unrepaired data with full repair. > > We would like to know if it is a known bug that was fixed post 3.0.3 and > there could be a possible way by which we can run repair on 3.0.3 without > increasing heap size as for all other activities 8GB works for us. > > PFA the visualvm snapshots. > > <Screenshot from 2016-06-20 21:06:09.png> > 3.0.3 VisualVM Snapshot, consistent heap usage of greater than 12 GB. > > > <Screenshot from 2016-06-20 21:05:57.png> > 3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till about 5GB. > > Thanks & Regards, > Bhuvan Rawal > > > PS: In case if the snapshots are not visible, they can be viewed from the > following links: > 3.0.3: > https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png > 3.7: > https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png > > >
