On Thu, Sep 24, 2015 at 12:45 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Wed, Sep 23, 2015 at 7:09 AM, Tom van den Berge < > tom.vandenbe...@gmail.com> wrote: > >> So it seems that Cassandra simply doesn't have enough memory. I'm trying >> to understand if this can be caused by the use of vnodes? Is there an >> sensible reason why vnodes would consume more memory than regular nodes? Or >> does any of you have the same experience? If not, I might be barking up the >> wrong tree here, and I would love to know it before upgrading my servers >> with more memory. >> > > Yes, range ownership has a RAM/heap cost per-range-owned. This cost is > paid during many, but not all, operations. Owning 256 ranges > Owning 1 > range. > > I have not had the same experience but am not at all surprised to hear > that vnodes increase heap consumption for otherwise identical > configurations. I am surprised to hear that it makes a significant > difference in GC time, but you might have been close enough to heap > saturation that vnodes tip you over. > That's apparently exactly what's going on. We've just increased the memory from 8 to 16 GB, and all is fine now. This seems to confirm that using vnodes indeed increase heap consumption significantly. I think it would be great if this could be advertised in the documentation, as a warning. From the current documentation, it seems that vnodes don't come at any cost. What's also interesting is this: Before increasing the memory we have been changing our code not to use our secondary indexes anymore. We still had a number of those, and we were suspecting them to be the cause of the increased heap consumption. It did not eliminate the problem, but it definitely helped to bring the GC times dramatically. I already knew that secondary indexes are best not to use, but it seems that using them in combination with vnodes makes it far worse. > As an aside, one is likely to win very little net win from vnodes if one's > cluster is not now and will never be more than approximately 15 nodes. > That's a very interesting observation. Especially since vnodes are enabled by default for some time now, and apparently it has a (heap) price. And my guess is that a significant percentage of all clusters will never exceed 15 nodes. Thx, Tom
