Practical limit to number of keyspaces?
Hi, We are using 2.0.x in production env. Each node has 8G heap. Is there a practical limit to how many keyspaces I can have? Please note that many keyspaces remain mostly dormant while others have very high level of activity. -- Sourabh Agrawal Bangalore +91 9945657973
Re: keyspace with hundreds of columnfamilies
Yes, what about CQL style columns? Please clarify On Sat, Jul 5, 2014 at 12:32 PM, tommaso barbugli wrote: > Yes my question what about CQL-style columns. > > > 2014-07-04 12:40 GMT+02:00 Jens Rantil : > > Just so you guys aren't misunderstanding each other; Tommaso, you were not >> refering to CQL-style columns, right? >> >> /J >> >> >> On Fri, Jul 4, 2014 at 10:18 AM, Romain HARDOUIN < >> romain.hardo...@urssaf.fr> wrote: >> >>> Cassandra can handle many more columns (e.g. time series). >>> So 100 columns is OK. >>> >>> Best, >>> Romain >>> >>> >>> >>> tommaso barbugli a écrit sur 03/07/2014 21:55:18 : >>> >>> > De : tommaso barbugli >>> > A : user@cassandra.apache.org, >>> > Date : 03/07/2014 21:55 >>> > Objet : Re: keyspace with hundreds of columnfamilies >>> > >>> > thank you for the replies; I am rethinking the schema design, one >>> > possible solution is to "implode" one dimension and get N times less >>> CFs. >>> >>> > With this approach I would come up with (cql) tables with up to 100 >>> > columns; would that be a problem? >>> > >>> > Thank You, >>> > Tommaso >>> > >>> >> >> > -- Sourabh Agrawal Bangalore +91 9945657973
Re: Size-tiered Compaction runs out of memory
Thank you both! Robert, I read through the bug--it sounds like this behavior has been fixed (or the impact reduced) in 2.1, but given that our data is pretty uniform (with no overlap between rows/values), it doesn't look like we'll suffer from that. At least, that's what I understood from the bug. I was actually discouraged from using the leveled compaction because, in general, write-heavy workloads yield poorer performance, I/O wise. However, since these machines have very little reads, I don't think some delay will hurt. Is it possible to seamlessly migrate from SizeTiered to Leveled Compaction? On Thu, Jul 10, 2014 at 6:56 AM, Pavel Kogan wrote: > Moving to Leveled compaction resolved same problem for us. As Robert > mentioned, use it carefully. > Size tiered compaction requires having 50% free disk space (also according > to datastax documentation). > > Pavel > > > On Wed, Jul 9, 2014 at 8:39 PM, Robert Coli wrote: > >> On Wed, Jul 9, 2014 at 4:27 PM, Andrew wrote: >> >>> What kind of overhead should I expect for compaction, in terms of >>> size? In this use case, the primary use for compaction is more or less to >>> clean up tombstones for expired TTLs. >>> >> >> Compaction can result in output files >100% of the input, if compression >> is used and the input SSTables are also compressed. If you use size tiered >> compaction (STS), you therefore must have enough headroom to compact your >> largest [n] SSTables together successfully. >> >> Level compaction (LCS) has a different, significantly lower, amount of >> headroom. >> >> If you are making heavy use of TTL, you should be careful about using LCS >> in certain cases, read : >> >> https://issues.apache.org/jira/browse/CASSANDRA-6654 - "Droppable >> tombstones are not being removed from LCS table despite being above 20%" >> >> =Rob >> >> >
[RELEASE CANDIDATE] Apache Cassandra 2.1.0-rc3 released
The Cassandra team is pleased to announce the release of the second release candidate for the future Apache Cassandra 2.1.0. Let first stress that this is not yet the final release of 2.1.0 and as such is *not* ready for production use. We however encourage as much testing as possible of this release candidate as possible and please report any problem you may encounter[3,4]. Also make sure to have a look at the change log[1] and release notes[2] to see where Cassandra 2.1 differs from the previous series. Apache Cassandra 2.1.0-rc3[5] is available as usual from the cassandra website (http://cassandra.apache.org/download/) and a debian package is available using the 21x branch (see http://wiki.apache.org/cassandra/DebianPackaging). Thank you for your help in testing and have fun with it. [1]: http://goo.gl/mYfL0J (CHANGES.txt) [2]: http://goo.gl/kiBZNS (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA [4]: user@cassandra.apache.org [5]: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/cassandra-2.1.0-rc3
Re: Size-tiered Compaction runs out of memory
Moving to Leveled compaction resolved same problem for us. As Robert mentioned, use it carefully. Size tiered compaction requires having 50% free disk space (also according to datastax documentation). Pavel On Wed, Jul 9, 2014 at 8:39 PM, Robert Coli wrote: > On Wed, Jul 9, 2014 at 4:27 PM, Andrew wrote: > >> What kind of overhead should I expect for compaction, in terms of size? >> In this use case, the primary use for compaction is more or less to clean >> up tombstones for expired TTLs. >> > > Compaction can result in output files >100% of the input, if compression > is used and the input SSTables are also compressed. If you use size tiered > compaction (STS), you therefore must have enough headroom to compact your > largest [n] SSTables together successfully. > > Level compaction (LCS) has a different, significantly lower, amount of > headroom. > > If you are making heavy use of TTL, you should be careful about using LCS > in certain cases, read : > > https://issues.apache.org/jira/browse/CASSANDRA-6654 - "Droppable > tombstones are not being removed from LCS table despite being above 20%" > > =Rob > >
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
It seems that memtable tries to flush itself to SSTable of not existing
keyspace. I don't know why it is happens, but probably running nodetool
flush before drop should prevent this issue.
Pavel
On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher
wrote:
> Hello,
>
> I am using the 'development' version 2.1-rc2.
>
> With one node (=localhost), I get timeouts trying to connect to C* after
> running a 'DROP KEYSPACE' command. I have following error messages in
> system.log :
>
> INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578
> MigrationManager.java:319 - Drop Keyspace 'test_main'
> (...)
> ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178
> CassandraDaemon.java:166 - Exception in thread
> Thread[MemtableFlushWriter:6,5,main]
> java.lang.RuntimeException: Last written key
> DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
> 52446dde90244ca49789b41671e4ca7c) >= current key
> DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
> 52446dde90244ca49789b41671e4ca7c) writing into
> ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55]
>
> Then, I can not connect to the Cluster anymore from my app (Java Driver
> 2.1-SNAPSHOT) and got in application logs :
>
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1:9042
> (com.datastax.driver.core.exceptions.DriverException: Timeout during read))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
> at
> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258)
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
> (...)
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1:9042
> (com.datastax.driver.core.exceptions.DriverException: Timeout during read))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> I can still connect through CQLSH but if I run (again) a "DROP KEYSPACE"
> command from CQLSH, I get the following error :
> errors={}, last_host=127.0.0.1
>
> Now, on a 2 nodes cluster I also have a similar issue but the error's
> stacktrace is different :
>
> From application logs :
>
> 17971 [Cassandra Java Driver worker-2] WARN
> com.datastax.driver.core.Cluster - No schema agreement from live replicas
> after 1 ms. The schema may not be up to date on some nodes.
>
> From system.log :
>
> INFO [SharedPool-Worker-2] 2014-07-10 09:04:53,434
> MigrationManager.java:319 - Drop Keyspace 'test_main'
> (...)
> ERROR [MigrationStage:1] 2014-07-10 09:04:56,553
> CommitLogSegmentManager.java:304 - Failed waiting for a forced recycle of
> in-use commit log segments
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecycleAll(CommitLogSegmentManager.java:299)
> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
> at
> org.a
C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
Hello,
I am using the 'development' version 2.1-rc2.
With one node (=localhost), I get timeouts trying to connect to C* after
running a 'DROP KEYSPACE' command. I have following error messages in
system.log :
INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578
MigrationManager.java:319 - Drop Keyspace 'test_main'
(...)
ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178
CassandraDaemon.java:166 - Exception in thread
Thread[MemtableFlushWriter:6,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
52446dde90244ca49789b41671e4ca7c) >= current key
DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
52446dde90244ca49789b41671e4ca7c) writing into
./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55]
Then, I can not connect to the Cluster anymore from my app (Java Driver
2.1-SNAPSHOT) and got in application logs :
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
tried for query failed (tried: /127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during read))
at
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
at
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
(...)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (tried: /127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during read))
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
at
com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I can still connect through CQLSH but if I run (again) a "DROP KEYSPACE"
command from CQLSH, I get the following error :
errors={}, last_host=127.0.0.1
Now, on a 2 nodes cluster I also have a similar issue but the error's
stacktrace is different :
>From application logs :
17971 [Cassandra Java Driver worker-2] WARN
com.datastax.driver.core.Cluster - No schema agreement from live replicas
after 1 ms. The schema may not be up to date on some nodes.
>From system.log :
INFO [SharedPool-Worker-2] 2014-07-10 09:04:53,434
MigrationManager.java:319 - Drop Keyspace 'test_main'
(...)
ERROR [MigrationStage:1] 2014-07-10 09:04:56,553
CommitLogSegmentManager.java:304 - Failed waiting for a forced recycle of
in-use commit log segments
java.lang.AssertionError: null
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecycleAll(CommitLogSegmentManager.java:299)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:160)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:516)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:300)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.j
Re: Changes in the Metrics MBean
Look at change log on the Github repo for the MBean class Le 10 juil. 2014 09:14, "bhavishya.p" a écrit : > > Hi.. > There seems to be a new feature added in the Metrics MBean which provides > the overall metrics for a particular cluster. In which version was this > feature added? > Also, how can we determine in which version the changes are made in > general? > Thanks in advance. > > Regards, > Bhavishya Palukurthi >
Changes in the Metrics MBean
Hi.. There seems to be a new feature added in the Metrics MBean which provides the overall metrics for a particular cluster. In which version was this feature added? Also, how can we determine in which version the changes are made in general? Thanks in advance. Regards, Bhavishya Palukurthi
