Re: NPE during compaction in compare
Are you sure you're using the last version of the 1.2 branch? The exceptions are in RangeTombstoneList that has been introduced by the patch of CASSANDRA-5677 and the initial version of the patch were indeed able to produce that kind of error. Could that be you're using such development version of the patch? What I would suggest is to test the current artifacts for 1.2.7 at https://repository.apache.org/content/repositories/orgapachecassandra-013/org/apache/cassandra/apache-cassandra/1.2.7/. This is what should be released as soon as the vote period is complete. If you can reproduce with that version, we'd be very interested to know (you can even feel free to open a ticket on JIRA). -- Sylvain On Wed, Jul 24, 2013 at 10:52 PM, Paul Ingalls paulinga...@gmail.comwrote: Hey Chris, so I just tried dropping all my data and converting my column families to use leveled compaction. Now I'm getting exceptions like the following once I start inserting data. Have you seen these? ERROR 13:13:25,616 Exception in thread Thread[CompactionExecutor:34,1,main] java.lang.NullPointerException at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:69) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:396) at org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:205) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:180) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:46) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:115) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) and ERROR 13:17:11,327 Exception in thread Thread[CompactionExecutor:45,1,main] java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:396) at org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:205) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:180) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:46) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:115) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at
Re: unable to compact large rows
Obvisously, that should not happen. That being said, we did fix a bug yesterday that could produce that kind of trace: https://issues.apache.org/jira/browse/CASSANDRA-5799. It will be part of 1.2.7 that should be released tomorrow, though the artifacts that are being voted on are at https://repository.apache.org/content/repositories/orgapachecassandra-013/org/apache/cassandra/apache-cassandra/1.2.7/if you want to test sooner than later. If you still can reproduce with that version, feel free to open a ticket on JIRA (in which case, the more information you can give us to reproduce, the better the change we'd be able to fix it quickly). -- Sylvain On Wed, Jul 24, 2013 at 10:54 PM, Paul Ingalls paulinga...@gmail.comwrote: It is pretty much every row that hits the large threshold. I don't think I can delete every row that hits that… you can see the db size in the stack trace, do you want a different type of size? On Jul 24, 2013, at 11:07 AM, Jason Wee peich...@gmail.com wrote: Would it possible to delete this row and reinsert this row? By the way, how large is that one row? Jason On Wed, Jul 24, 2013 at 9:23 AM, Paul Ingalls paulinga...@gmail.comwrote: I'm getting constant exceptions during compaction of large rows. In fact, I have not seen one work, even starting from an empty DB. As soon as I start pushing in data, when a row hits the large threshold, it fails compaction with this type of stack trace: INFO [CompactionExecutor:6] 2013-07-24 01:17:53,592 CompactionController.java (line 156) Compacting large row fanzo/tweets_by_id:352567939972603904 (153360688 bytes) incrementally ERROR [CompactionExecutor:6] 2013-07-24 01:18:12,496 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:6,1,main] java.lang.AssertionError: incorrect row data size 5722610 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_id/fanzo-tweets_by_id-tmp-ic-1453-Data.db; correct is 5767384 at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) I'm not sure what to do or where to look. Help…:) Thanks, Paul
Re: disappointed
Yeah, Rob is smart. don't run crap in production. Run what others are stable at. If you are running the latest greatest dumbest craziest in prod then you ask for fail, and you will get just that. FAIL On Jul 24, 2013, at 12:06 PM, Robert Coli rc...@eventbrite.com wrote: A better solution would likely involve not running cutting edge code in production.. if you find yourself needing to upgrade production anything on the day of a release, you are probably ahead of the version it is reasonable to run in production. If you're already comfortable with this high level of risk in production, I don't really see small manual patches as significantly increasing your level of risk... =Rob
Re: disappointed
Mysql? -- Colin +1 320 221 9531 On Jul 25, 2013, at 6:08 AM, Derek Andree dand...@lacunasystems.com wrote: Yeah, Rob is smart. don't run crap in production. Run what others are stable at. If you are running the latest greatest dumbest craziest in prod then you ask for fail, and you will get just that. FAIL On Jul 24, 2013, at 12:06 PM, Robert Coli rc...@eventbrite.com wrote: A better solution would likely involve not running cutting edge code in production.. if you find yourself needing to upgrade production anything on the day of a release, you are probably ahead of the version it is reasonable to run in production. If you're already comfortable with this high level of risk in production, I don't really see small manual patches as significantly increasing your level of risk... =Rob
[BETA RELEASE] Apache Cassandra 2.0.0-beta2 released
The Cassandra team is pleased to announce the release of the second beta for the future Apache Cassandra 2.0.0. Let me first stress that this is still beta software and as such is *not* ready for production use. As with the first beta, the goal is to give a preview of what will become Cassandra 2.0 and to get wider testing before the final release. Please report any problem you may encounter[3,4] with this release and have a look at the change log[1] and release notes[2] to see where Cassandra 2.0 differs from the previous series. Apache Cassandra 2.0.0-beta2[5] is available as usual from the cassandra website (http://cassandra.apache.org/download/) and a debian package is available using the 20x branch (see http://wiki.apache.org/cassandra/DebianPackaging). Thank you for your help in testing and have fun with it. [1]: http://goo.gl/jY3FYT (CHANGES.txt) [2]: http://goo.gl/r1AcVZ (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA [4]: user@cassandra.apache.org [5]: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/cassandra-2.0.0-beta2
Re: sstable size change
Unfortunately the table in question is a CQL3 table so cassandra-cli will not
output its describe:
WARNING: CQL3 tables are intentionally omitted from 'describe' output.
See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
However, I did figure out that apparently I was not setting the change
properly. I was using the following alter via CQL3:
alter table shard_user_lookup with compaction_strategy_options =
{'sstable_size_in_mb':256};
But when I did a describe, I did not see the change.
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.10 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1',
'sstable_compression': 'LZ4Compressor'};
If I then execute the following via CQL2, I do see the property. Seems odd to
me?
alter table shard_user_lookup with
compaction_strategy_options:sstable_size_in_mb=256;
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.10 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '256', 'class':
'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1',
'sstable_compression': 'LZ4Compressor'};
From: Wei Zhu wz1...@yahoo.commailto:wz1...@yahoo.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org, Wei Zhu
wz1...@yahoo.commailto:wz1...@yahoo.com
Date: Wednesday, July 24, 2013 8:49 PM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: sstable size change
what is output of show keyspaces from cassandra-cli, did you see the new value?
Compaction Strategy:
org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compaction Strategy Options:
sstable_size_in_mb: XXX
From: Keith Wright kwri...@nanigans.commailto:kwri...@nanigans.com
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Sent: Wednesday, July 24, 2013 3:44 PM
Subject: Re: sstable size change
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter
command and saw at least one compaction run (I did not trigger a compaction via
nodetool nor upgrades stables nor removing the .json file). But so far my data
sizes appear the same at the default 5 MB (see below for output of ls –Sal as
well as relevant portion of cfstats). Is this expected? I was hoping to see
at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33
users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59
users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58
users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56
users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44
users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54
users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41
users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44
users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58
users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli kohlisank...@gmail.commailto:kohlisank...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Tuesday, July 23, 2013 3:04 PM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: sstable size change
Will Cassandra force any newly compacted files to my new setting as
compactions are naturally
TTL, Tombstones, and gc_grace
Hello, Quick question on Cassandra, TTLs, tombstones, and GC grace. If we have a column family whose only mechanism of deleting columns is utilizing TTLs, is repair really necessary to make tombstones consistent, and therefore would it be safe to set the gc grace period of the column family to a very low value? I ask because of this blog post based on Cassandra .7: http://www.datastax.com/dev/blog/whats-new-cassandra-07-expiring-columns. The first time the expired column is compacted, it is transformed into a tombstone. This transformation frees some disk space: the size of the value of the expired column. From that moment on, the column is a normal tombstone and follows the tombstone rules: it will be totally removed by compaction (including minor ones in most cases since Cassandra 0.6.6) after GCGraceSeconds. Since tombstones are not written using a replicated write, but instead written during compaction, theoretically, it shouldn't be possible to lose a tombstone? Or is this blog post inaccurate for later versions of cassandra? We are using cassandra 1.1.11. Thanks, -Mike
Re: TTL, Tombstones, and gc_grace
Hi Michael, yes, you should never loose a delete, because there are no real deletes. No matter what version you are using. btw: There is actually a ticket that builds an optimization on top of that assumption: CASSANDRA-4917. Basically, if TTLgc_grace then do not create tombstones for expiring-columns. This works because disappear anyway if TTL is over. cheers, Christian On Thu, Jul 25, 2013 at 3:24 PM, Michael Theroux mthero...@yahoo.comwrote: Hello, Quick question on Cassandra, TTLs, tombstones, and GC grace. If we have a column family whose only mechanism of deleting columns is utilizing TTLs, is repair really necessary to make tombstones consistent, and therefore would it be safe to set the gc grace period of the column family to a very low value? I ask because of this blog post based on Cassandra .7: http://www.datastax.com/dev/blog/whats-new-cassandra-07-expiring-columns. The first time the expired column is compacted, it is transformed into a tombstone. This transformation frees some disk space: the size of the value of the expired column. From that moment on, the column is a normal tombstone and follows the tombstone rules: it will be totally removed by compaction (including minor ones in most cases since Cassandra 0.6.6) after GCGraceSeconds. Since tombstones are not written using a replicated write, but instead written during compaction, theoretically, it shouldn't be possible to lose a tombstone? Or is this blog post inaccurate for later versions of cassandra? We are using cassandra 1.1.11. Thanks, -Mike
Re: Strange cassandra-stress results with 2.0.0 beta1
Some bug was fixed in 2.0.0-beta2 by C* developers. Try it! 2013/7/22 Andrew Cobley a.e.cob...@dundee.ac.uk I've been noticing some strange casandra-stress results with 2.0.0 beta 1. I've set up a single node on a Mac (4 gig ram, 2.8Ghz core 2 duo) and installed 2.0.0 beta1. When I run ./cassandra-stress -d 134.36.36.218 I'm seeing the interval-op-rate drop from a peek of 11562 at the start to 0 -20 after about 358 seconds. Here's the output: ./cassandra-stress -d 134.36.36.218 Created keyspaces. Sleeping 1s for propagation. total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time 13629,1362,1362,15.3,117.6,531.1,10 37545,2391,2391,9.4,74.0,438.4,20 105410,6786,6786,3.3,38.0,216.4,30 209014,10360,10360,2.7,26.2,216.5,41 320024,11101,11101,2.5,14.5,216.5,51 430216,11019,11019,2.4,10.5,215.4,61 531905,10168,10168,2.4,8.3,177.1,72 619672,8776,8776,2.4,6.2,145.8,82 653977,3430,3430,2.4,5.9,145.4,92 682648,2867,2867,2.4,6.1,145.4,102 692771,1012,1012,2.4,6.1,145.4,113 712163,1939,1939,2.4,6.0,4461.7,123 723761,1159,1159,2.4,6.0,4461.7,133 731115,735,735,2.4,6.1,4461.7,143 737627,651,651,2.4,6.2,4461.7,153 743853,622,622,2.4,7.0,5056.1,164 747345,349,349,2.4,7.0,5056.1,174 755501,815,815,2.5,6.9,6351.7,184 758692,319,319,2.5,6.9,5056.1,195 763960,526,526,2.5,7.3,5292.6,205 767966,400,400,2.5,7.4,5292.6,215 769514,154,154,2.5,7.4,5292.6,225 773492,397,397,2.5,7.2,6435.7,236 775374,188,188,2.5,7.4,5292.6,246 776035,66,66,2.6,7.5,5347.1,256 777896,186,186,2.6,7.9,6407.3,266 778791,89,89,2.6,8.0,6438.9,276 779646,85,85,2.6,8.0,6523.2,287 780785,113,113,2.6,11.1,6523.2,297 781336,55,55,2.6,11.7,6523.2,307 781684,34,34,2.6,13.3,16734.0,317 781818,13,13,2.7,13.9,16764.2,328 781952,13,13,2.7,13.9,16900.1,338 782195,24,24,2.7,15.3,16900.1,348 782294,9,9,2.7,15.3,16900.1,358 782362,6,6,2.7,15.5,21487.3,368 782589,22,22,2.7,20.1,21487.3,379 782775,18,18,2.7,28.8,21487.3,389 783012,23,23,2.8,29.4,21487.3,399 783248,23,23,2.8,68.2,21487.3,409 783270,2,2,2.8,68.2,21487.3,419 783283,1,1,2.8,68.2,21487.3,430 783283,0,0,2.8,68.2,21487.3,440 783337,5,5,2.8,198.6,42477.2,450 783492,15,15,2.9,5057.6,52687.3,460 783581,8,8,2.9,5349.0,59306.9,471 783807,22,22,3.0,6429.1,59306.9,481 783816,0,0,3.0,6438.3,59306.9,491 783945,12,12,3.2,11566.9,59306.9,501 783962,1,1,3.2,11612.0,59306.9,512 783962,0,0,3.2,11612.0,59306.9,522 I installed 1.2.6 for comparision and that gave expected results: ./cassandra-stress -d 134.36.36.218 Unable to create stress keyspace: Keyspace names must be case-insensitively unique (Keyspace1 conflicts with Keyspace1) total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time 22158,2215,2215,0.8,8.9,549.0,10 64192,4203,4203,1.0,7.0,486.2,20 114236,5004,5004,1.3,6.5,411.9,30 178407,6417,6417,1.4,5.5,411.9,40 281714,10330,10330,2.2,5.1,409.6,51 371684,8997,8997,2.4,5.1,407.5,61 464401,9271,9271,2.5,5.0,236.7,72 562797,9839,9839,2.6,5.2,87.5,82 655755,9295,9295,2.6,5.2,87.5,92 751560,9580,9580,2.6,4.7,179.9,103 848022,9646,9646,2.6,4.5,826.0,113 914539,6651,6651,2.6,4.7,823.2,123 100,8546,8546,2.6,5.1,1048.5,133 Any ideas whats causing the slow down ? Andy The University of Dundee is a registered Scottish Charity, No: SC015096 -- Atenciosamente, Sávio S. Teles de Oliveira voice: +55 62 9136 6996 http://br.linkedin.com/in/savioteles Mestrando em Ciências da Computação - UFG Arquiteto de Software Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG
Re: About column family
With 1.2.7 you can use -Dcassandra.unsafesystem. That will speed up cf creation. So you will get in even more trouble even faster! On Tue, Jul 23, 2013 at 12:23 PM, bjbylh bjb...@me.com wrote: Hi all: i have two questions to ask: 1,how many column families can be created in a cluster?is there a limit to the number of it? 2,it spents 2-5 seconds to create a new cf while the cluster contains about 1 cfs(if the cluster is empty,it spents about 0.5s).is it normal?how to improve the efficiency of creating cf? btw C* is 1.2.4. thanks a lot. Sent from Samsung Mobile
Re: Strange cassandra-stress results with 2.0.0 beta1
I'm getting something similar for beta 2, you can see here it's dropping from 12000 to 400 quite quickly. The output from cassandra is available n dropbox here: https://dl.dropboxusercontent.com/u/1638201/output.txt (184kBytes) I'm wondering if it's a GC issue ? ./cassandra-stress -d 134.36.36.218 -n 3000 Unable to create stress keyspace: Keyspace names must be case-insensitively unique (Keyspace1 conflicts with Keyspace1) total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time 73761,7376,7376,2.1,7.6,181.0,10 199159,12539,12539,2.4,7.5,102.4,20 316978,11781,11781,2.4,7.8,172.7,30 434060,11708,11708,2.4,8.1,170.6,40 552650,11859,11859,2.5,7.3,170.8,50 602432,4978,4978,2.5,7.3,170.8,60 614364,1193,1193,2.5,7.5,5490.4,71 634199,1983,1983,2.5,8.1,5490.4,81 653293,1909,1909,2.5,8.9,5490.4,91 658398,510,510,2.5,8.7,5490.4,101 671805,1340,1340,2.6,8.7,5490.4,111 678677,687,687,2.6,9.1,5490.4,121 685247,657,657,2.6,10.2,5626.8,131 691519,627,627,2.6,10.7,5626.8,141 693682,216,216,2.6,10.7,5626.8,151 700082,640,640,2.7,11.9,6460.1,161 703952,387,387,2.7,12.5,6460.1,172 706007,205,205,2.7,12.5,6460.1,182 709988,398,398,2.7,12.7,6647.0,192 ^Cmacaroon:bin administrator$ Andy C On 25 Jul 2013, at 15:28, Sávio Teles savio.te...@lupa.inf.ufg.brmailto:savio.te...@lupa.inf.ufg.br wrote: Some bug was fixed in 2.0.0-beta2 by C* developers. Try it! 2013/7/22 Andrew Cobley a.e.cob...@dundee.ac.ukmailto:a.e.cob...@dundee.ac.uk I've been noticing some strange casandra-stress results with 2.0.0 beta 1. I've set up a single node on a Mac (4 gig ram, 2.8Ghz core 2 duo) and installed 2.0.0 beta1. When I run ./cassandra-stress -d 134.36.36.218 I'm seeing the interval-op-rate drop from a peek of 11562 at the start to 0 -20 after about 358 seconds. Here's the output: The University of Dundee is a registered Scottish Charity, No: SC015096
maximum storage per node
Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne
Re: maximum storage per node
Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.comwrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? ** ** Thanks, Anne
RE: maximum storage per node
We're storing fairly large files (about 1MB apiece) for a few months and then deleting the oldest to get more space to add new ones. We have large requirements (maybe up to 100 TB), so having a 1TB limit would be unworkable. What is the reason for the limit? Does something fail after that? If there are hardware issues, what's recommended? BTW, we're using Cassandra 1.2 Anne From: cem [mailto:cayiro...@gmail.com] Sent: Thursday, July 25, 2013 11:41 AM To: user@cassandra.apache.org Subject: Re: maximum storage per node Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.commailto:pru...@avaya.com wrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne
RE: maximum storage per node
Issues with large data nodes would be - * Nodetool repair will be impossible to run * Your read i/o will suffer since you will almost always go to disk (each read will take 3 IOPS worst case) * Boot-straping the node in case of failures will take days/weeks From: Pruner, Anne (Anne) [mailto:pru...@avaya.com] Sent: 25 July 2013 10:45 To: user@cassandra.apache.org Subject: RE: maximum storage per node We're storing fairly large files (about 1MB apiece) for a few months and then deleting the oldest to get more space to add new ones. We have large requirements (maybe up to 100 TB), so having a 1TB limit would be unworkable. What is the reason for the limit? Does something fail after that? If there are hardware issues, what's recommended? BTW, we're using Cassandra 1.2 Anne From: cem [mailto:cayiro...@gmail.com] Sent: Thursday, July 25, 2013 11:41 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: maximum storage per node Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.commailto:pru...@avaya.com wrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne
Re: maximum storage per node
You will suffer from long compactions if you are planning to get rid of from old records by TTL. Best Regards, Cem. On Thu, Jul 25, 2013 at 5:51 PM, Kanwar Sangha kan...@mavenir.com wrote: Issues with large data nodes would be – ** ** **· **Nodetool repair will be impossible to run **· **Your read i/o will suffer since you will almost always go to disk (each read will take 3 IOPS worst case) **· **Boot-straping the node in case of failures will take days/weeks ** ** ** ** *From:* Pruner, Anne (Anne) [mailto:pru...@avaya.com] *Sent:* 25 July 2013 10:45 *To:* user@cassandra.apache.org *Subject:* RE: maximum storage per node ** ** We’re storing fairly large files (about 1MB apiece) for a few months and then deleting the oldest to get more space to add new ones. We have large requirements (maybe up to 100 TB), so having a 1TB limit would be unworkable. ** ** What is the reason for the limit? Does something fail after that? ** ** If there are hardware issues, what’s recommended? ** ** BTW, we’re using Cassandra 1.2 ** ** Anne ** ** *From:* cem [mailto:cayiro...@gmail.com cayiro...@gmail.com] *Sent:* Thursday, July 25, 2013 11:41 AM *To:* user@cassandra.apache.org *Subject:* Re: maximum storage per node ** ** Between 500GB - 1TB is recommended. ** ** But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? ** ** Best Regards, Cem ** ** On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.com wrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne ** **
RE: maximum storage per node
I actually wrote my own compactor that deals with this problem. Anne From: cem [mailto:cayiro...@gmail.com] Sent: Thursday, July 25, 2013 11:59 AM To: user@cassandra.apache.org Subject: Re: maximum storage per node You will suffer from long compactions if you are planning to get rid of from old records by TTL. Best Regards, Cem. On Thu, Jul 25, 2013 at 5:51 PM, Kanwar Sangha kan...@mavenir.commailto:kan...@mavenir.com wrote: Issues with large data nodes would be - * Nodetool repair will be impossible to run * Your read i/o will suffer since you will almost always go to disk (each read will take 3 IOPS worst case) * Boot-straping the node in case of failures will take days/weeks From: Pruner, Anne (Anne) [mailto:pru...@avaya.commailto:pru...@avaya.com] Sent: 25 July 2013 10:45 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: RE: maximum storage per node We're storing fairly large files (about 1MB apiece) for a few months and then deleting the oldest to get more space to add new ones. We have large requirements (maybe up to 100 TB), so having a 1TB limit would be unworkable. What is the reason for the limit? Does something fail after that? If there are hardware issues, what's recommended? BTW, we're using Cassandra 1.2 Anne From: cem [mailto:cayiro...@gmail.com] Sent: Thursday, July 25, 2013 11:41 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: maximum storage per node Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.commailto:pru...@avaya.com wrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne
Re: Strange cassandra-stress results with 2.0.0 beta1
So is that in Cassandra somewhere ? Any idea on how I can go about pinpointing the problem to raise a JIRA issue ? Andy On 25 Jul 2013, at 17:50, Radim Kolar h...@filez.com wrote: I'm wondering if it's a GC issue ? yes it is: 1039280992 used; max is 1052770304 most likely memory leak. The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Strange cassandra-stress results with 2.0.0 beta1
Dne 25.7.2013 20:03, Andrew Cobley napsal(a): Any idea on how I can go about pinpointing the problem to raise a JIRA issue ? http://www.ehow.com/how_8705297_create-java-heap-dump.html
deleting columns with CAS (2.0.0-beta2)
Hi, I've been playing around with CAS (compare-and-swap) in the 2.0 beta, using the Thrift API. I can't at all figure out how to delete columns while using CAS, however. I'm able to update/insert columns by adding them to the list of updates that is passed as a parameter to org.apache.cassandra.thrift.Cassandra.Client#cas(). But there seems to be no indication as to how to delete. I presume it can be done because conditional deletes work through CQL3. However, I'm interested in using Hector/Java (particularly as CQL3 does not yet allow batched conditional statements). Any help would be great! Thanks, Kalpana
cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Hi, I am upgrading my cassandra cluster from 0.8 to 1.2.5. In cassandra 1.2.5 the 'num_token' attribute confuses me. I understand that it distributes multiple tokens per node but I am not clear how that is helpful for performance or load balancing. Can anyone elaborate? has anyone used this feature and knows its advantages/disadvantages? Thanks, Rashmi
Re: NPE during compaction in compare
Hey Sylvain, I pulled the latest from the 1.2 branch and these exceptions were fixed. The only issue that remains is compacting large rows which fails with an assertion error. Thanks for the tip! Paul On Jul 25, 2013, at 12:26 AM, Sylvain Lebresne sylv...@datastax.com wrote: Are you sure you're using the last version of the 1.2 branch? The exceptions are in RangeTombstoneList that has been introduced by the patch of CASSANDRA-5677 and the initial version of the patch were indeed able to produce that kind of error. Could that be you're using such development version of the patch? What I would suggest is to test the current artifacts for 1.2.7 at https://repository.apache.org/content/repositories/orgapachecassandra-013/org/apache/cassandra/apache-cassandra/1.2.7/. This is what should be released as soon as the vote period is complete. If you can reproduce with that version, we'd be very interested to know (you can even feel free to open a ticket on JIRA). -- Sylvain On Wed, Jul 24, 2013 at 10:52 PM, Paul Ingalls paulinga...@gmail.com wrote: Hey Chris, so I just tried dropping all my data and converting my column families to use leveled compaction. Now I'm getting exceptions like the following once I start inserting data. Have you seen these? ERROR 13:13:25,616 Exception in thread Thread[CompactionExecutor:34,1,main] java.lang.NullPointerException at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:69) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:396) at org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:205) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:180) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:46) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:115) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) and ERROR 13:17:11,327 Exception in thread Thread[CompactionExecutor:45,1,main] java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:396) at org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:205) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:180) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:46) at
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
It is very useful to upgrade the apps perfomance. For example, if you have a machine with X capacity, you can put the num_token=256. If you add a machine in your cluster with (X*2) capacity you can put the num_token=512. So, this new machine will receive twice the load automatically. Moreover, you can minimize how long a machine replacement is going to take and will minimize the time to repair a disk failure. Read the docs to more informations: http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 Cheers! 2013/7/25 rash aroskar rashmi.aros...@gmail.com Hi, I am upgrading my cassandra cluster from 0.8 to 1.2.5. In cassandra 1.2.5 the 'num_token' attribute confuses me. I understand that it distributes multiple tokens per node but I am not clear how that is helpful for performance or load balancing. Can anyone elaborate? has anyone used this feature and knows its advantages/disadvantages? Thanks, Rashmi -- Atenciosamente, Sávio S. Teles de Oliveira voice: +55 62 9136 6996 http://br.linkedin.com/in/savioteles Mestrando em Ciências da Computação - UFG Arquiteto de Software Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG
Re: unable to compact large rows
I ran with the latest from the 1.2 branch, and although the NPE and index out of bounds exception went away, these still stuck around. I'll see if I can figure out something to put into JIRA... On Jul 25, 2013, at 12:32 AM, Sylvain Lebresne sylv...@datastax.com wrote: Obvisously, that should not happen. That being said, we did fix a bug yesterday that could produce that kind of trace: https://issues.apache.org/jira/browse/CASSANDRA-5799. It will be part of 1.2.7 that should be released tomorrow, though the artifacts that are being voted on are at https://repository.apache.org/content/repositories/orgapachecassandra-013/org/apache/cassandra/apache-cassandra/1.2.7/ if you want to test sooner than later. If you still can reproduce with that version, feel free to open a ticket on JIRA (in which case, the more information you can give us to reproduce, the better the change we'd be able to fix it quickly). -- Sylvain On Wed, Jul 24, 2013 at 10:54 PM, Paul Ingalls paulinga...@gmail.com wrote: It is pretty much every row that hits the large threshold. I don't think I can delete every row that hits that… you can see the db size in the stack trace, do you want a different type of size? On Jul 24, 2013, at 11:07 AM, Jason Wee peich...@gmail.com wrote: Would it possible to delete this row and reinsert this row? By the way, how large is that one row? Jason On Wed, Jul 24, 2013 at 9:23 AM, Paul Ingalls paulinga...@gmail.com wrote: I'm getting constant exceptions during compaction of large rows. In fact, I have not seen one work, even starting from an empty DB. As soon as I start pushing in data, when a row hits the large threshold, it fails compaction with this type of stack trace: INFO [CompactionExecutor:6] 2013-07-24 01:17:53,592 CompactionController.java (line 156) Compacting large row fanzo/tweets_by_id:352567939972603904 (153360688 bytes) incrementally ERROR [CompactionExecutor:6] 2013-07-24 01:18:12,496 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:6,1,main] java.lang.AssertionError: incorrect row data size 5722610 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_id/fanzo-tweets_by_id-tmp-ic-1453-Data.db; correct is 5767384 at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) I'm not sure what to do or where to look. Help…:) Thanks, Paul
Cassandra 2.0: Paxos Prepare response always false
Hi, I have test setup where clients randomly make a controlled number of cas() requests (among other requests) at a cluster of cassandra 2.0 servers. After one point, I'm seeing that all requests are pending and my client's throughput has reduced to 0.0 for all kinds of requests. For this specific case I had 10 clients each making around 30 cas() requests per second at a cluster of 72 instances of cassandra. Clients are set up to register a request as a success after the cas() call returns with CASResult.success = true, else an exception is thrown. Since I see that no client requests were actually registered and no exceptions were thrown, which indicates that the cas() call itself is hung. On the server side, I see Paxos logs as follows - they go on for 50 log files for each of the servers involved, and they span at least an hour. I have marked a particular instance where the prepare response is true but the propose response is false from all the involved servers: *At the Paxos Initiator: * None of the files among the 50 system logs have the phrase 'Propose response true', these logs just go on and on. * * DEBUG [RequestResponseStage:110] 2013-07-25 15:09:05,332 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d2093120-f576-11e2-a57e-a154d605509d, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.195 * * DEBUG [RequestResponseStage:92] 2013-07-25 15:09:05,346 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d2093120-f576-11e2-a57e-a154d605509d, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.184 DEBUG [RequestResponseStage:98] 2013-07-25 15:09:05,347 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d2093120-f576-11e2-a57e-a154d605509d, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.20 DEBUG [RequestResponseStage:93] 2013-07-25 15:09:05,350 ProposeCallback.java (line 44) Propose response false from /17.163.7.20 DEBUG [RequestResponseStage:100] 2013-07-25 15:09:05,350 ProposeCallback.java (line 44) Propose response false from /17.163.7.184 DEBUG [RequestResponseStage:111] 2013-07-25 15:09:05,350 ProposeCallback.java (line 44) Propose response false from /17.163.7.195 DEBUG [RequestResponseStage:102] 2013-07-25 15:09:05,351 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d20c3e60-f576-11e2-9bbe-bf2ad4fe6707, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.195 DEBUG [RequestResponseStage:107] 2013-07-25 15:09:05,352 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d20c3e60-f576-11e2-9bbe-bf2ad4fe6707, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.20 DEBUG [RequestResponseStage:108] 2013-07-25 15:09:05,352 PrepareCallback.java (line 58) Prepare response PrepareResponse(true, Commit(d145fe46f5d02a54b5ea95852f94c402, 1a0c4220-f561-11e2-a409-019f62d610d7, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])), Commit(d145fe46f5d02a54b5ea95852f94c402, d20c3e60-f576-11e2-9bbe-bf2ad4fe6707, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,]))) from / 17.163.7.184 DEBUG [RequestResponseStage:104] 2013-07-25 15:09:05,352 ProposeCallback.java (line 44) Propose response false from /17.163.7.20 DEBUG [RequestResponseStage:99] 2013-07-25 15:09:05,353 ProposeCallback.java (line 44) Propose response false from /17.163.7.195 DEBUG [RequestResponseStage:105] 2013-07-25 15:09:05,353 ProposeCallback.java (line 44) Propose response false from /17.163.7.184 *At 17.163.7.20:* * * DEBUG [MutationStage:58] 2013-07-25 15:09:05,347 PaxosState.java (line 100) accept requested for Commit(d145fe46f5d02a54b5ea95852f94c402, d20b05e0-f576-11e2-9bbe-bf2ad4fe6707, ColumnFamily(P [81d271b2125c59cb0003:false:27@1374780817218000,])) but inProgress is
Re: cassandra 1.2.6 - Start key's token sorts after end token
We managed to make it work with RandomPartitioner... Guess we will rely on it for now... 2013/7/23 Hiller, Dean dean.hil...@nrel.gov Oh, and in the past 0.20.x has been pretty stable by the wayŠ..they finally switched their numbering scheme thank god. Dean On 7/23/13 2:13 PM, Hiller, Dean dean.hil...@nrel.gov wrote: Perhaps try 0.20.2 as 1. The maven pom files have cassandra depending on 0.20.2 2. The 0.20.2 default was murmur and we had to change it to random partitioner or it wouldn't work for us Ie. I suspect they will change the pom file to a more recent version of hadoop at some point but I wonder if test suites suck in 0.20.2 because the pom file points to that versionŠ.depends on if they actually have tests for map/reduce which is probably a bit hard. Dean From: Marcelo Elias Del Valle mvall...@gmail.commailto:mvall...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Tuesday, July 23, 2013 1:54 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: cassandra 1.2.6 - Start key's token sorts after end token Dean, I am using hadoop 1.0.3. Indeed, using Cassandra 1.2.3 with Random partitioner, it worked. However, it's the only reason for me to use randompartitioner, I really would like to move forward. Besides, I tried to use Cassandra 1.2.6 with RandomPartitioner and I got problems when inserting data, even stopping Cassandra, cleaning my entire data folder and then starting it again. I am also really curious to know if there is anyone else having these problems or if it is just me... Best regards, Marcelo. 2013/7/23 Hiller, Dean dean.hil...@nrel.govmailto:dean.hil...@nrel.gov Out of curiosity, what version of hadoop are you using with cassandra? I think we are trying 0.20.2 if I remember(I have to ask my guy working on it to be sure). I do remember him saying the cassandra maven dependency was odd in that it is in the older version and not a newer hadoop version. We are using RandomPartitioner though right now which I have personally used in the past with success. We are in the process of map/reducing to a cassandra with MurmurPartitioner (our real reason to map/reduce is some refactorings in our model though and we just thought we would switch to murmur). Has anyone else used map/reduce with murmur partitioner? Dean From: Marcelo Elias Del Valle mvall...@gmail.commailto:mvall...@gmail.commailto:mvall...@gmail.com m ailto:mvall...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto: user@c assandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto: user@c assandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, July 22, 2013 4:04 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto: user@c assandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto: user@c assandra.apache.orgmailto:user@cassandra.apache.org Subject: cassandra 1.2.6 - Start key's token sorts after end token Hello, I am trying to figure what might be cause this error. I am using Cassandra 1.2.6 (tried with 1.2.3 as well) and I am trying to read data from cassandra on hadoop using column family input format. I also got the same error using pure astyanax on a test. I am using Murmur3Partitioner and I created the keyspace using Cassandra 1.2.6, there is nothing from prior versions. I created the keyspace with SimpleStrategy and replication factor 1. Here is the exception I am getting: 2013-07-22 21:53:05,824 WARN org.apache.hadoop.mapred.Child (main): Error running child java.lang.RuntimeException: InvalidRequestException(why:Start key's token sorts after end token) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.maybe Init(ColumnFamilyRecordReader.java:453) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.compu teNext(ColumnFamilyRecordReader.java:459) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.compu teNext(ColumnFamilyRecordReader.java:406) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat or.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1 38) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFam ilyRecordReader.java:103) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTa sk.java:522) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapT ask.java:547) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at
Re: maximum storage per node
Try putting multiple instances per machine with each instance mapped to its own disk. This might not work with v-nodes On Thu, Jul 25, 2013 at 9:04 AM, Pruner, Anne (Anne) pru...@avaya.comwrote: I actually wrote my own compactor that deals with this problem. ** ** Anne ** ** *From:* cem [mailto:cayiro...@gmail.com] *Sent:* Thursday, July 25, 2013 11:59 AM *To:* user@cassandra.apache.org *Subject:* Re: maximum storage per node ** ** You will suffer from long compactions if you are planning to get rid of from old records by TTL. ** ** Best Regards, Cem. ** ** On Thu, Jul 25, 2013 at 5:51 PM, Kanwar Sangha kan...@mavenir.com wrote: Issues with large data nodes would be – · Nodetool repair will be impossible to run · Your read i/o will suffer since you will almost always go to disk (each read will take 3 IOPS worst case) · Boot-straping the node in case of failures will take days/weeks* *** *From:* Pruner, Anne (Anne) [mailto:pru...@avaya.com] *Sent:* 25 July 2013 10:45 *To:* user@cassandra.apache.org *Subject:* RE: maximum storage per node We’re storing fairly large files (about 1MB apiece) for a few months and then deleting the oldest to get more space to add new ones. We have large requirements (maybe up to 100 TB), so having a 1TB limit would be unworkable. What is the reason for the limit? Does something fail after that? If there are hardware issues, what’s recommended? BTW, we’re using Cassandra 1.2 Anne *From:* cem [mailto:cayiro...@gmail.com cayiro...@gmail.com] *Sent:* Thursday, July 25, 2013 11:41 AM *To:* user@cassandra.apache.org *Subject:* Re: maximum storage per node Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) pru...@avaya.com wrote: Does anyone have opinions on the maximum amount of data reasonable to store on one Cassandra node? If there are limitations, what are the reasons for it? Thanks, Anne ** **
