Re: CompactionExecutor holds 8000+ SSTableReader 6G+ memory

2013-07-01 Thread sulong
These two fields: CompressedRandomAccessReader.buffer CompressedRandomAccessReader.compressed in the queue SSTableReader.dfile.pool consumed those memory. I think the SSTableReader.dfile is the cache of the SSTable file. On Sat, Jun 29, 2013 at 1:09 PM, aaron morton

Re: How to do a CAS UPDATE on single column CF?

2013-07-01 Thread Sylvain Lebresne
You're right, there is currently no way to do this since 1) insert can't have a IF currently and 2) update can't update such table. We'll fix that: https://issues.apache.org/jira/browse/CASSANDRA-5715 -- Sylvain On Sat, Jun 29, 2013 at 9:51 PM, Blair Zajac bl...@orcaware.com wrote: On

Re: Cassandra as storage for cache data

2013-07-01 Thread Dmitry Olshansky
Hello, thanks to all for your answers and comments. What we've done: - increased Java heap memory up to 6 Gb - changed replication factor to 1 - set durable_writes to false - set memtable_total_space_in_mb to 5000 - set commitlog_total_space_in_mb to 6000 If I understand correctly the last

C* 1.2.5 AssertionError in ColumnSerializer:40

2013-07-01 Thread horschi
Hi, using C* 1.2.5 I just found a weird AssertionError in our logfiles: ... INFO [OptionalTasks:1] 2013-07-01 09:15:43,608 MeteredFlusher.java (line 58) flushing high-traffic column family CFS(Keyspace='Monitoring', ColumnFamily='cfDateOrderedMessages') (estimated 5242880 bytes) INFO

10,000s of column families/keyspaces

2013-07-01 Thread Kirk True
Hi all, I know it's an old topic, but I want to see if anything's changed on the number of column families that C* supports, either in 1.2.x or 2.x. For a number of reasons [1], we'd like to support multi-tenancy via separate column families. The problem is that there are around 5,000 tenants to

Re: 10,000s of column families/keyspaces

2013-07-01 Thread Hiller, Dean
We use playorm to do 80,000 virtual column families(a playorm feature though the pattern could be copied). We did find out later and we are working on this now that we wanted to map 80,000 virtual CF's into 10 real CF's so leveled compaction can run more in parallel though or else we get stuck

Re: 10,000s of column families/keyspaces

2013-07-01 Thread Hiller, Dean
Oh and if you are using STCS, I don't think the below is an issue at all since that can run in parallel if needed already. Dean On 7/1/13 10:24 AM, Hiller, Dean dean.hil...@nrel.gov wrote: We use playorm to do 80,000 virtual column families(a playorm feature though the pattern could be copied).

Re: CorruptBlockException

2013-07-01 Thread Robert Coli
On Sat, Jun 29, 2013 at 8:39 PM, Glenn Thompson gatman1...@gmail.com wrote: I'm Glenn Thompson and new to Cassandra. I have been trying to figure out how to recover from a CorruptBlockException. ... One of my nodes must have a hardware problem. Although I've been unable to find anything

Re: CorruptBlockException

2013-07-01 Thread Glenn Thompson
Hi Rob, It was hardware. Memory. I've been loading data since I originally posted. No exceptions so far. I had some issues with OOMs when I first started playing with cassandra. I increased the amount RAM to the VM and reduced the memtable size. I'm guessing it's because I'm using I3s.

Re: How to do a CAS UPDATE on single column CF?

2013-07-01 Thread Blair Zajac
Thanks! On 7/1/13 1:41 AM, Sylvain Lebresne wrote: You're right, there is currently no way to do this since 1) insert can't have a IF currently and 2) update can't update such table. We'll fix that: https://issues.apache.org/jira/browse/CASSANDRA-5715 -- Sylvain On Sat, Jun 29, 2013 at 9:51

Re: How to do a CAS UPDATE on single column CF?

2013-07-01 Thread Hiller, Dean
What does CAS stand for? And is that the row locking feature like hbase's setAndReadWinner that you give the previous val and next val and your next val is returned if you won otherwise the current result is returned and you know some other node won? Thanks, Dean On 7/1/13 12:09 PM, Blair Zajac

Re: Patterns for enabling Compute apps which only request Local Node's

2013-07-01 Thread Robert Coli
On Sun, Jun 30, 2013 at 1:48 AM, rekt...@voodoowarez.com wrote: Question; if we're co-locating our Cassandra and our compute application on the same nodes, are there any in-use patterns in Cassandra user (or Cassandra dev) applications for having the compute application only pull data off the

Re: How to do a CAS UPDATE on single column CF?

2013-07-01 Thread Francisco Andrades Grassi
http://en.wikipedia.org/wiki/Compare-and-swap I believe C* uses Paxos for CAS but not completely sure? -- Francisco Andrades Grassi www.bigjocker.com @bigjocker On Jul 1, 2013, at 1:49 PM, Hiller, Dean dean.hil...@nrel.gov wrote: What does CAS stand for? And is that the row locking feature

Re: How to do a CAS UPDATE on single column CF?

2013-07-01 Thread Andrew Cobley
According to Jonathan Ellis talk at Cassandra 13 it does use Paxos: http://www.youtube.com/watch?v=PcUpPR4nSr4list=PLqcm6qE9lgKJzVvwHprow9h7KMpb5hcUU http://www.slideshare.net/jbellis/cassandra-summit-2013-keynote Andy On 1 Jul 2013, at 19:40, Francisco Andrades Grassi

Re: 10,000s of column families/keyspaces

2013-07-01 Thread Robert Coli
On Mon, Jul 1, 2013 at 9:19 AM, Kirk True kirktrue...@gmail.com wrote: What will it take for C* to support 50,000 column families? As I understand it, a (the?) big problem with huge numbers of Column Families is that each ColumnFamily has a large number of MBeans associated with it, each of

Re: 10,000s of column families/keyspaces

2013-07-01 Thread Edward Capriolo
There is another problem. You now need to run repair for a large number of column families and keyspaces and manage that, look out for schema mismatches etc. On Mon, Jul 1, 2013 at 4:09 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Jul 1, 2013 at 9:19 AM, Kirk True kirktrue...@gmail.com

Re: Cassandra as storage for cache data

2013-07-01 Thread Robert Coli
The most effective way to deal with obsolete Tombstones in the short lived cache case seems to be to drop them on the floor en masse... :D a) have two column families that the application alternates between, modulo time_period b) truncate and populate the cold one c) read from the hot one d)

Dynamic Snitch and EC2MultiRegionSnitch

2013-07-01 Thread Daning Wang
How does dynamic snitch work with EC2MultiRegionSnitch? Can dynamic routing only happen in one data center? We don't wan to have the requests routed to another center even nodes are idle in other side since the network could be slow. Thanks in advance, Daning

RE: about FlushWriter All time blocked

2013-07-01 Thread Arindam Barua
Thanks guys, these sound like good suggestions, will try those out. Aaron, we have around 80 CFs. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Friday, June 28, 2013 10:05 PM To: user@cassandra.apache.org Subject: Re: about FlushWriter All time blocked We do not use secondary

schema management

2013-07-01 Thread Franc Carter
Hi, I've been giving some thought to the way we deploy schemas and am looking for something better than out current approach, which is to use cassandra-cli scripts. What do people use for this ? cheers -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au

query deadlock(?) after flushing a table

2013-07-01 Thread Mohica Jasha
Hey, I created a table with a wide row. Query on the wide row after removing the entries and flushing the table becomes very slow. I am aware of the impact of tombstones but it seems that there is a deadlock which prevents the query to be completed. step by step: 1. creating the keyspace and

Re: schema management

2013-07-01 Thread sankalp kohli
You can generate schema through the code. That is also one option. On Mon, Jul 1, 2013 at 4:10 PM, Franc Carter franc.car...@sirca.org.auwrote: Hi, I've been giving some thought to the way we deploy schemas and am looking for something better than out current approach, which is to use

Re: schema management

2013-07-01 Thread Todd Fast
Franc-- I think you will find Mutagen Cassandra very interesting; it is similar to schema management tools like Flyway for SQL databases: Mutagen Cassandra is a framework (based on Mutagen) that provides schema versioning and mutation for Apache Cassandra. Mutagen is a lightweight framework

Re: schema management

2013-07-01 Thread Franc Carter
On Tue, Jul 2, 2013 at 10:33 AM, Todd Fast t...@digitalexistence.comwrote: Franc-- I think you will find Mutagen Cassandra very interesting; it is similar to schema management tools like Flyway for SQL databases: Oops - forgot to mention in my original email that we will be looking into

Streaming performance with 1.2.6

2013-07-01 Thread Mike Heffner
Hi, We've recently been testing some of the higher performance instance classes on EC2, specifically the hi1.4xlarge, with Cassandra. For those that are not familiar with them, they have two SSD disks and 10 gige. While we have observed much improved raw performance over our current instances,

very inefficient operation with tombstones

2013-07-01 Thread Mohica Jasha
Querying a table with 5000 thousands tombstones take 3 minutes to complete! But Querying the same table with the same data pattern with 10,000 entries takes a fraction of second to complete! Details: 1. created the following table: CREATE KEYSPACE test WITH replication = {'class':