Re: Cassandra high heap utilization under heavy reads and writes.
So, I did a lot of dial turning and heap tuning (came across this nice writeup about JVM tuning http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html) still no luck with 1.2.9. I gave up and upgraded to 1.2.12 and since then things are much much better. I don't run into the heap issue that I used to with 1.2.9. 1.2.12 also looks more stable and seems to work for us. -sandeep On Mon, Nov 25, 2013 at 11:10 AM, Christopher J. Bottaro < cjbott...@academicworks.com> wrote: > Yes, we saw this same behavior. > > A couple of months ago, we moved a large portion of our data out of > Postgres and into Cassandra. The initial migration was done in a > "distributed" manner: we had 600 (or 800, can't remember) processes > reading from Postgres and writing to Cassandra in tight loops. This caused > the exact behavior you described. We also did a read before a write. > > After we got through the initial data migration, our normal workload is > *much* less writes (and reads for that matter) such that our cluster can > easily handle it, so we didn't investigate further. > > -- C > > > On Sat, Nov 23, 2013 at 10:55 PM, srmore wrote: > >> Hello, >> We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap >> bloom filters and other improvements. >> >> We see a lot of messages dropped under high load conditions. We noticed >> that when we do heavy read AND write simultaneously (we read first and >> check whether the key exists if not we write it) Cassandra heap increases >> dramatically and then gossip marks the node down (as a result of high load >> on the node). >> >> >> Under heavy 'reads only' we don't see this behavior. Has anyone seen >> this behavior ? any suggestions. >> >> Thanks ! >> >> >> >
Re: Cassandra high heap utilization under heavy reads and writes.
Yes, we saw this same behavior. A couple of months ago, we moved a large portion of our data out of Postgres and into Cassandra. The initial migration was done in a "distributed" manner: we had 600 (or 800, can't remember) processes reading from Postgres and writing to Cassandra in tight loops. This caused the exact behavior you described. We also did a read before a write. After we got through the initial data migration, our normal workload is *much* less writes (and reads for that matter) such that our cluster can easily handle it, so we didn't investigate further. -- C On Sat, Nov 23, 2013 at 10:55 PM, srmore wrote: > Hello, > We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap > bloom filters and other improvements. > > We see a lot of messages dropped under high load conditions. We noticed > that when we do heavy read AND write simultaneously (we read first and > check whether the key exists if not we write it) Cassandra heap increases > dramatically and then gossip marks the node down (as a result of high load > on the node). > > > Under heavy 'reads only' we don't see this behavior. Has anyone seen this > behavior ? any suggestions. > > Thanks ! > > >
Cassandra high heap utilization under heavy reads and writes.
Hello, We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap bloom filters and other improvements. We see a lot of messages dropped under high load conditions. We noticed that when we do heavy read AND write simultaneously (we read first and check whether the key exists if not we write it) Cassandra heap increases dramatically and then gossip marks the node down (as a result of high load on the node). Under heavy 'reads only' we don't see this behavior. Has anyone seen this behavior ? any suggestions. Thanks !