i am seeing this as well. i did a test with just 1 cassandra node,
ReplicationFactor=1, 'get' ConsistencyLevel.ONE, 'put'
ConsistencyLevel.QUORUM. The first test was writing and reading random
values starting from a fresh database. The put performance is staying
reasonabe, but the read performance falls off dramatically as the data
grows. The get performance fell from approx 6500 get/sec to 150 get/sec
(as reported by my client stats.) The database has grown to approx
500gig. i have the stats recorded on 5 second intervals and i see a
very linear drop off as the data grows.
i stopped the server and restarted it, let it do its thing during
restart and then rerun a read-only test using the exact same data. i am
still at about 150 get/sec. via JMX i can see the read latency at about
60, but this varies as the app runs.
my keyspace is simple:
<Keyspaces>
<Keyspace Name="uds">
<KeysCachedFraction>0.01</KeysCachedFraction>
<ColumnFamily CompareWith="BytesType" Name="bucket" />
</Keyspace>
</Keyspaces>
all values are exactly the same and are 2k in length.
i've tried to do some tuning to make things faster but don't necessarily
understand the options. here are some of the params i've changed in the
config file:
<CommitLogRotationThresholdInMB>256</CommitLogRotationThresholdInMB>
<MemtableSizeInMB>1024</MemtableSizeInMB>
<MemtableObjectCountInMillions>0.6</MemtableObjectCountInMillions>
<CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
<MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
hope this data helps, and any help you can provide is much appreciated.
On Tue, 2009-12-01 at 20:18 -0600, Jonathan Ellis wrote:
> On Tue, Dec 1, 2009 at 7:31 PM, Freeman, Tim <tim.free...@hp.com> wrote:
> > Looking at the Cassandra mbean's, the attributes of ROW-MUTATION-STAGE and
> > ROW-READ-STAGE and RESPONSE-STAGE are all less than 10.
> > MINOR-COMPACTION-POOL reports 1218 pending tasks.
>
> That's probably the culprit right there. Something is wrong if you
> have 1200 pending compactions.
>
> This is something that upgrading to trunk will help with right away
> since we parallelize compactions there.
>
> Another thing you can do is increase the memtable limits so you are
> not flushing + compacting so often with your insert traffic.
>
> -Jonathan
