Re: cassandra performance problems
Thanks all for your responses. We've downgraded from 2.0.3 to 2.0.0 and everything became normal. 2013/12/8 Nate McCall n...@thelastpickle.com If you are really set on using Cassandra as a cache, I would recommend disabling durable writes for the keyspace(s)[0]. This will bypass the commitlog (the flushing/rotation of which my be a good-sized portion of your performance problems given the number of tables). [0] http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/alter_keyspace_r.html On Fri, Dec 6, 2013 at 12:42 PM, J. Ryan Earl o...@jryanearl.us wrote: On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. If you just want a caching layer, why wouldn't you use Memcached or Redis instead? Cassandra is designed to be a persist store and not so much designed as caching layer. If you were replacing your use of Postgres completely, that would be appropriate. -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: cassandra performance problems
If you are really set on using Cassandra as a cache, I would recommend disabling durable writes for the keyspace(s)[0]. This will bypass the commitlog (the flushing/rotation of which my be a good-sized portion of your performance problems given the number of tables). [0] http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/alter_keyspace_r.html On Fri, Dec 6, 2013 at 12:42 PM, J. Ryan Earl o...@jryanearl.us wrote: On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. If you just want a caching layer, why wouldn't you use Memcached or Redis instead? Cassandra is designed to be a persist store and not so much designed as caching layer. If you were replacing your use of Postgres completely, that would be appropriate. -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: cassandra performance problems
On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. If you just want a caching layer, why wouldn't you use Memcached or Redis instead? Cassandra is designed to be a persist store and not so much designed as caching layer. If you were replacing your use of Postgres completely, that would be appropriate.
Re: cassandra performance problems
Do you mean high CPU usage or high load avg? (20 indicates load avg to me). High load avg means the CPU is waiting on something. Check iostat -dmx 1 100 to check your disk stats, you'll see the columns that indicate mb/s read write as well as % utilization. Once you understand the bottleneck we can start to narrow down the cause. On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: Hi all, We have a 3 node cluster setup, single keyspace, about 500 tables. The hardware is 2 cores + 16 GB RAM (Cassandra chose to have 4GB). Cassandra version is 2.0.3. Our replication factor is 3, read/write consistency is QUORUM. We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. Then the problems came. Our nodes began showing high cpu usage (around 20). The funny thing is that they were actually doing it one after another and there was always only node with high cpu usage. Using OpsCenter we saw that when the CPU was beginning to go high the node in question was performing compaction. But even after the compaction was performed the cpu remained still high, and in some cases didn't go down for hours. Our jmx monitoring showed that it was presumably in constant garbage collection. During that time cluster read latency goes from 2ms to 200ms What can be the reason? Can it be high number of tables? Do we need to adjust some settings for this setup? Is it ok to have so many tables? Theoretically we can stuck them all in 3-4 tables. Thanks in advance, Alexander -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: cassandra performance problems
On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: Cassandra version is 2.0.3. ... We've plugged it into our production environment as a cache in front of postgres. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ What can be the reason? Can it be high number of tables? Do we need to adjust some settings for this setup? Is it ok to have so many tables? Theoretically we can stuck them all in 3-4 tables. 500 column families (tables) is a lot but not necessarily too many. During that time cluster read latency goes from 2ms to 200ms In perfectly normal operation, things operating in the JVM may exhibit a few hundreds of milliseconds of latency, when the JVM pauses for garbage collection. If you absolutely require consistent latency below this time for all requests, something running in the JVM may not be for you. If compaction triggers runaway GC then you probably have too much steady state heap consumption. You don't mention anything about the hardware environment, how much heap is available, etc.. ? =Rob
Re: cassandra performance problems
Thanks for your answers, Jonathan, yes it was load avg and iowait was lower than 2% all that time - the only load was the user one. Robert, we had -Xmx4012m which was automatically calculated by the default cassandra-env.sh (1/4 of total memory - 16G) - we didn't change that. 2013/12/5 Robert Coli rc...@eventbrite.com On Thu, Dec 5, 2013 at 4:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: Cassandra version is 2.0.3. ... We've plugged it into our production environment as a cache in front of postgres. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ What can be the reason? Can it be high number of tables? Do we need to adjust some settings for this setup? Is it ok to have so many tables? Theoretically we can stuck them all in 3-4 tables. 500 column families (tables) is a lot but not necessarily too many. During that time cluster read latency goes from 2ms to 200ms In perfectly normal operation, things operating in the JVM may exhibit a few hundreds of milliseconds of latency, when the JVM pauses for garbage collection. If you absolutely require consistent latency below this time for all requests, something running in the JVM may not be for you. If compaction triggers runaway GC then you probably have too much steady state heap consumption. You don't mention anything about the hardware environment, how much heap is available, etc.. ? =Rob
