Cluster imbalance caused due to #Num_Tokens

Hi, While setting up a cluster for our POC, when we installed Cassandra on the 1st node we gave num_tokens: 256 , while on next 2 nodes which were added later we left it blank in Cassandra.yaml. This made our cluster an unbalanced one with nodetool status showing 99% load on one server. Now ev

Re: LCS Strategy, compaction pending tasks keep increasing

Thanks Brice for the input, I am confused as to how to calculate the value of concurrent_read, following is what i found recommended on sites and in configuration docs. concurrent_read : some places its 16 X number of drives or 4 X number of cores which of the above should i pick ? i have 40 cor

Re: LCS Strategy, compaction pending tasks keep increasing

Yes I was referring referring to multithreaded_compaction, but just because we didn’t get bitten by this setting just doesn’t mean it’s right, and the jira is a clear indication of that ;) @Anishek that reminds me of these settings to look at as well: - concurrent_write and concurrent_read bot

Re: Handle Write Heavy Loads in Cassandra 2.0.3

Hi, I cannot really answer your question as some rock solid truth. When we had problems, we did mainly two things - Analyzed the GC logs (with censum from jClarity, this tool IS really awesome, it’s good investment even better if the production is running other java applications) - He

Re: LCS Strategy, compaction pending tasks keep increasing

I want to draw a distinction between a) multithreaded compaction (the jira I just pointed to) and b) concurrent_compactors. I'm not clear on which one you are recommending at this stage. a) Multithreaded compaction is what I warned against in my last note. b) Concurrent compactors is the number of

Error while building from source code

Hi, I am trying to build a project the source bundled downloaded from http://apache.arvixe.com/cassandra/2.1.4/apache-cassandra-2.1.4-src.tar.gz but when I run "ant build" I get following error during build. Any idea why I am getting build Failed? Seems looking for dependencies org.apache.cassan

Re: Is 2.1.5 ready for upgrade?

Robert, Can you elaborate more please? Cheers, Brian On Tuesday, April 21, 2015, Robert Coli wrote: > On Tue, Apr 21, 2015 at 2:25 PM, Dikang Gu > wrote: > >> We have some issues with streaming in 2.1.2. We find that there are a lot >> of patches in 2.1.5. Is it ready for upgrade? >> > > I per

Re: LCS Strategy, compaction pending tasks keep increasing

Oh, thank you Sebastian for this input and the ticket reference ! We did notice an increase in CPU usage, but kept the concurrent compaction low enough for our usage, by default it takes the number of cores. We did use a number up to 30% of our available cores. But under heavy load clearly CPU is t

Re: Is 2.1.5 ready for upgrade?

On Tue, Apr 21, 2015 at 2:25 PM, Dikang Gu wrote: > We have some issues with streaming in 2.1.2. We find that there are a lot > of patches in 2.1.5. Is it ready for upgrade? > I personally would not run either version in production at this time, but if forced, would prefer 2.1.5 over 2.1.2. =Ro

Is 2.1.5 ready for upgrade?

Hi guys, We have some issues with streaming in 2.1.2. We find that there are a lot of patches in 2.1.5. Is it ready for upgrade? Thanks. -- Dikang

Re: LCS Strategy, compaction pending tasks keep increasing

Do not enable multithreaded compaction. Overhead usually outweighs any benefit. It's removed in 2.1 because it harms more than helps: https://issues.apache.org/jira/browse/CASSANDRA-6142 All the best, [image: datastax_logo.png] Sebastián Estévez Solutions Architect

Re: Cassandra tombstones being created by updating rows with TTL's

Whats ur sstable count for the CF? I hope compactions are working fine. Also check the full stacktrace of FileNotFoundException ..if its related to compactionyou can try cleaning compactions_in_progress folder in system folder in data directory..there are JIRA issues relating to that. Than

Re: Handle Write Heavy Loads in Cassandra 2.0.3

Thanks Brice!! We are using Red Hat Linux 6.4..24 cores...64Gb Ram..SSDs in RAID5..CPU are not overloaded even in peak load..I dont think IO is an issue as iostat shows await<17 all times..util attrbute in iostat usually increases from 0 to 100..and comes back immediately..m not an expert on a

Re: Bootstrap performance.

On Mon, Apr 20, 2015 at 8:09 PM, Dikang Gu wrote: > Why do you say steaming is single threaded? I see a lot of background > streaming threads running, for example: > Imprecise : "Each stream is a single thread". As I said, first place to look is throttles... but I would not be surprised if the

Re: CQL 3.x Update ...USING TIMESTAMP...

On Mon, Apr 20, 2015 at 4:02 PM, Sachin Nikam wrote: > #1. We have 2 data centers located close by with plans to expand to more > data centers which are even further away geographically. > #2. How will this impact light weight transactions when there is high > level of network contention for cros

Re: Handle Write Heavy Loads in Cassandra 2.0.3

This is an intricate matter, I cannot say for sure what are good parameters from the wrong ones, too many things changed at once. However there’s many things to consider - What is your OS ? - Do your nodes have SSDs or mechanical drives ? How many cores do you have ? - Is it the CPUs

Re: Cassandra tombstones being created by updating rows with TTL's

Hmm - we read/write with Local Quorum always - I'd recommend that as that is your 'consistency' defense. We use python, so I am not familiar with the java driver - but 'file not found' indicates something is inconsistent. On Tue, Apr 21, 2015 at 12:22 PM, Walsh, Stephen wrote: > Thanks for all

RE: Cassandra tombstones being created by updating rows with TTL's

Thanks for all your help Michael, Our data will change through the day, so data with a TTL will eventually get dropped, and new data will appear. I’d imagine the entire table maybe expire and start over 7-10 times a day. But on the GC topic, now java Driver now gives this error on the query I

Re: Cassandra tombstones being created by updating rows with TTL's

Discussions previously on the list show why this is not a problem in much more detail. If something changes in your cluster: node down, new node, etc - you run repair for sure. We also run periodic repairs prophylactically. But if you never delete and always ttl by the same amount, you do not ha

RE: Cassandra tombstones being created by updating rows with TTL's

Maybe thanks Michael, I will give these setting a go, How do you do you periodic node-tool repairs in the situation, for what I read we need to start doing this also. https://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair From: Laing, Michael [mailto:michael.la...@nytimes.com

Re: Cassandra tombstones being created by updating rows with TTL's

If you never delete except by ttl, and always write with the same ttl (or monotonically increasing), you can set gc_grace_seconds to 0. That's what we do. There have been discussions on the list over the last few years re this topic. ml On Tue, Apr 21, 2015 at 11:14 AM, Walsh, Stephen wrote: >

Re: Handle Write Heavy Loads in Cassandra 2.0.3

Any suggestions or comments on this one??  Thanks Anuj Wadhera Sent from Yahoo Mail on Android From:"Anuj Wadehra" Date:Mon, 20 Apr, 2015 at 11:51 pm Subject:Re: Handle Write Heavy Loads in Cassandra 2.0.3 Small correction: we are making writes in 5 cf an reading frm one at high speeds. 

Cassandra tombstones being created by updating rows with TTL's

We were chatting to Jon Haddena about a week ago about our tombstone issue using Cassandra 2.0.14 To Summarize We have a 3 node cluster with replication-factor=3 and compaction = SizeTiered We use 1 keyspace with 1 table Each row have about 40 columns Each row has a TTL of 10 seconds We insert a

Re: LCS Strategy, compaction pending tasks keep increasing

I’m not sure I get everything about storm stuff, but my understanding of LCS is that compaction count may increase the more one update data (that’s why I was wondering about duplicate primary keys). Another option is that the code is sending too much write request/s to the cassandra cluster. I don

Re: LCS Strategy, compaction pending tasks keep increasing

sorry i take that back we will modify different keys across threads not the same key, our storm topology is going to use field grouping to get updates for same keys to same set of bolts. On Tue, Apr 21, 2015 at 6:17 PM, Anishek Agarwal wrote: > @Bruice : I dont think so as i am giving each threa

Re: LCS Strategy, compaction pending tasks keep increasing

@Bruice : I dont think so as i am giving each thread a specific key range with no overlaps this does not seem to be the case now. However we will have to test where we have to modify the same key across threads -- do u think that will cause a problem ? As far as i have read LCS is recommended for s

Re: LCS Strategy, compaction pending tasks keep increasing

Could it that the app is inserting _duplicate_ keys ? -- Brice On Tue, Apr 21, 2015 at 1:52 PM, Marcus Eriksson wrote: > nope, but you can correlate I guess, tools/bin/sstablemetadata gives you > sstable level information > > and, it is also likely that since you get so many L0 sstables, you wi

RE: Connecting to Cassandra cluster in AWS from local network

Thanks everyone for the suggestions! I have used the following code to create my cluster from my dev environment and it seems to be working perfectly: cluster = Cluster.*builder* ().addContactPoints(nodes).withAddressTranslater(*new* AddressTranslater() { *publi

Re: LCS Strategy, compaction pending tasks keep increasing

nope, but you can correlate I guess, tools/bin/sstablemetadata gives you sstable level information and, it is also likely that since you get so many L0 sstables, you will be doing size tiered compaction in L0 for a while. On Tue, Apr 21, 2015 at 1:40 PM, Anishek Agarwal wrote: > @Marcus I did l

Network transfer to one node twice as others

Hello, We are using cassandra 2.0.14 and have a cluster of 3 nodes. I have a writer test (written in java) that runs 50 threads to populate data to a single table in a single keyspace. when i look at the "iftop" I see that the amount of network transfer happening on two nodes is same but on one

Re: LCS Strategy, compaction pending tasks keep increasing

@Marcus I did look and that is where i got the above but it doesnt show any detail about moving from L0 -L1 any specific arguments i should try with ? On Tue, Apr 21, 2015 at 4:52 PM, Marcus Eriksson wrote: > you need to look at nodetool compactionstats - there is probably a big L0 > -> L1 compa

Re: LCS Strategy, compaction pending tasks keep increasing

I am on version 2.0.14, will update once i get the stats up for the writes again On Tue, Apr 21, 2015 at 4:46 PM, Carlos Rolo wrote: > Are you on version 2.1.x? > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant > > Pythian - Love your data > > rolo@pythian | Twitter: cjrolo | Linkedin

Re: LCS Strategy, compaction pending tasks keep increasing

Are you on version 2.1.x? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo * Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com O

LCS Strategy, compaction pending tasks keep increasing

Hello, I am inserting about 100 million entries via datastax-java driver to a cassandra cluster of 3 nodes. Table structure is as create keyspace test with replication = {'class': 'NetworkTopologyStrategy', 'DC' : 3}; CREATE TABLE test_bits(id bigint primary key , some_bits text) with gc_grace_

Re: LCS Strategy, compaction pending tasks keep increasing

you need to look at nodetool compactionstats - there is probably a big L0 -> L1 compaction going on that blocks other compactions from starting On Tue, Apr 21, 2015 at 1:06 PM, Anishek Agarwal wrote: > the "some_bits" column has about 14-15 bytes of data per key. > > On Tue, Apr 21, 2015 at 4:34

Re: LCS Strategy, compaction pending tasks keep increasing

the "some_bits" column has about 14-15 bytes of data per key. On Tue, Apr 21, 2015 at 4:34 PM, Anishek Agarwal wrote: > Hello, > > I am inserting about 100 million entries via datastax-java driver to a > cassandra cluster of 3 nodes. > > Table structure is as > > create keyspace test with replic