Re: CQL flow control

On Wed, Feb 5, 2014 at 7:32 PM, Edward Capriolo wrote: > I agree you can not really ask your database to capacity plan for you. > Cassandra does have backpressure of sorts if requests fail with > TimedOutException or UnavailableException. You might be having a capacity > problem. > > The way I wo

Re: CQL flow control

On Wed, Feb 5, 2014 at 6:55 PM, Robert Coli wrote: > I think most deploys of Cassandra deal with this reality by carefully > managing available capacity so that they don't risk getting in this > situation. This is what I have done in my production apps. Basically I have found the system's sweet s

First SSTable file is not being compacted

Hi, I'm running C* 2.0.4 and when I have a handful of SSTable files and trigger a manual compaction with 'nodetool compact' the first SSTable file doesn't get compacted away. Is there something special about the first SSTable that it remains even after a SizedTierCompaction? - SF

Re: Move to smaller nodes

Thank you Rob this is very helpful. I'll keep you posted on any progress. Are others running some what large nodes on CentOS 6.4 or similar? Using java 7? We are also hosted through SoftLayer? Any help is much appreciated. In general I think Cassandra meets our needs but this is a blocker fo

Re: Move to smaller nodes

On Wed, Feb 5, 2014 at 11:22 AM, Keith Wright wrote: > Also there is one more option which is we could upgrade to 2.0 in the > hopes that our issue is fixed as part of the streaming overhaul. But > seeing as this is a production cluster and 2.0 does not yet appear > production ready, that makes

Re: Move to smaller nodes

On Wed, Feb 5, 2014 at 11:18 AM, Keith Wright wrote: > Hi Rob, thanks for the response! Interestingly if we run a repair we > don't see the bootstrap issue so I am considering doing the empty node > repair methodology. > Weird. Bootstrap should not be more fragile than repair. > >- Update

Re: Periodic rpc_timeout errors on select query

Hi Chap, If you have the ability to test the 2.0.0rc2 driver, I would recommend doing so, even from a dedicated test client or a JUnit test case. There are other benefits to the change, such as being able to use BatchStatements, aside from possible impact on your read timeouts. Steve On Wed, F

Re: Periodic rpc_timeout errors on select query

Hi Steve, Thanks for the reply. After all that information in my initial message I would forget one of the most important bits. We're running Cassandra 2.0.3 with the 1.0.4 version of the DataStax driver. I'd seen mention of those timeouts under earlier 2.x versions and really hoped they were

Re: Periodic rpc_timeout errors on select query

Hi Chap, You don't indicate which version of Cassandra and what client side driver you are using, but I have seen the same behavior with Cassandra 2.0.2 and earlier versions of the Java Driver. With Cassandra 2.0.3 and the 2.0.0rc2 driver, my read timeouts are basically nonexistent at my current l

Re: Lots of deletions results in death by GC

You should find that the patch will apply cleanly to the 2.0.5 release, so you could apply it yourself. On 5 February 2014 18:56, Robert Wille wrote: > Thank you so much. Everything I had seen pointed to this being the case. > I'm glad that someone in the know has confirmed this bug and fixed i

Periodic rpc_timeout errors on select query

Hi, We're seeing pretty regular rpc timeout errors on what appear to be simple queries. We're running a three node cluster under pretty light load. We're averaging 30-40 writes/sec and about 8 reads/sec according to OpsCenter. The failures don't seem to be related to any changes in load. A si

Re: Question about how DataStax python driver chooses a coordinator

On Wed, Feb 5, 2014 at 2:45 PM, Sameer Farooqui wrote: > Does the new DataStax Python Driver 1.0.0 intelligently choose a > coordinator that is also likely to be a replica partner for that row-key > when using vnodes in C* 2.0? If so, how does it do it... just hash the > row-key and see which part

Re: pycassa get column_start and column_finish with less than or greater than

On Wed, Feb 5, 2014 at 1:05 PM, Kumar Ranjan wrote: > Does pycassa get column_start takes greater than equal to option? column_start is inclusive at the Cassandra API level, so pycassa matches that behavior (and there's nothing else it can really do). > What I know so far is, you have to exac

Question about how DataStax python driver chooses a coordinator

Does the new DataStax Python Driver 1.0.0 intelligently choose a coordinator that is also likely to be a replica partner for that row-key when using vnodes in C* 2.0? If so, how does it do it... just hash the row-key and see which partition range it falls in and which node owns that range? Here is

Re: CQL flow control

I agree you can not really ask your database to capacity plan for you. Cassandra does have backpressure of sorts if requests fail with TimedOutException or UnavailableException. You might be having a capacity problem. The way I would handle this is 1) prototype at scale (dark launches, similar ha

Re: Move to smaller nodes

Also there is one more option which is we could upgrade to 2.0 in the hopes that our issue is fixed as part of the streaming overhaul. But seeing as this is a production cluster and 2.0 does not yet appear production ready, that makes me nervous. From: Keith Wright mailto:kwri...@nanigans.com>

Re: Move to smaller nodes

Hi Rob, thanks for the response! Interestingly if we run a repair we don’t see the bootstrap issue so I am considering doing the empty node repair methodology. Its just that it usually takes a week for that to work. As I see it, we could try the following to fix the bootstrap issue: * Upd

Re: Move to smaller nodes

On Wed, Feb 5, 2014 at 11:00 AM, Keith Wright wrote: > Earlier today I emailed about issues we're having bootstrapping nodes > into our existing cluster. One theory we have is that our nodes are simply > too large and are considering moving to more, smaller nodes. However, > because we cann

pycassa get column_start and column_finish with less than or greater than

Hey Folks, Does pycassa get column_start takes greater than equal to option? What I know so far is, you have to exact column or super_column value for column_start and column_finish to work. In my case, column is value of epoch time.

Move to smaller nodes

Hi all, Earlier today I emailed about issues we’re having bootstrapping nodes into our existing cluster. One theory we have is that our nodes are simply too large and are considering moving to more, smaller nodes. However, because we cannot bootstrap it makes it difficult. As I see it, w

Re: Lots of deletions results in death by GC

Thank you so much. Everything I had seen pointed to this being the case. I¹m glad that someone in the know has confirmed this bug and fixed it. Now I just need to figure out where to go from here: do I wait, use the dev branch or work around. Robert From: Benedict Elliott Smith Reply-To: Date

Re: CQL flow control

On Wed, Feb 5, 2014 at 6:14 AM, Ben Hood <0x6e6...@gmail.com> wrote: > What is the general approach to this from a server perspective? Is > there any flow control that the server can apply to back pressure onto > the sending driver? No. In theory the client could look at dynamic snitch scores, I

Re: Question 1: JMX binding, Question 2: Logging

Unfortunately, you can't change the interface jmx binds to initially. So in order to accomplish what you want you will need to use different jmx ports for each process. This is how ccm (https://github.com/pcmanus/ccm) accomplishes this as well. On Wed, Feb 5, 2014 at 9:35 AM, Kyle Crumpton (kcru

Re: Question 1: JMX binding, Question 2: Logging

Hi Andrey, I only see a port for JMX. I do not see a value for the interface. I am using Cassandra 1.2.6 From: Andrey Ilinykh mailto:ailin...@gmail.com>> Reply-To: "user@cassandra.apache.org" mailto:user@cassandra.apache.org>> Date: Tuesday, February 4, 2014 4:

Re: Lots of deletions results in death by GC

I believe there is a bug, and I have filed a ticket for it: https://issues.apache.org/jira/browse/CASSANDRA-6655 I will have a patch uploaded shortly, but it's just missed the 2.0.5 release window, so you'll either need to grab the development branch once it's committed or wait until 2.0.6 On 5

Re: Lots of deletions results in death by GC

Yes. It¹s kind of an unusual workload. An insertion phase followed by a deletion phase, generally not overlapping. From: Benedict Elliott Smith Reply-To: Date: Tuesday, February 4, 2014 at 5:29 PM To: Subject: Re: Lots of deletions results in death by GC Is it possible you are generating

Re: Bootstrap failure

I did find a stop the world GC on one of the non-bootstrapping nodes during one of our previous bootstrap failures (see below) as well as the flags being passed to the java process. Perhaps this is just a GC tuning issue? >From what I’ve read, bootstrap is supposed to be a light weight operatio

CQL flow control

Hi, A discussion has arisen in the gocql team about how to handle saturation when CQL clients are sending in packets at a faster rate than the Cassandra cluster can sustain. What is the general approach to this from a server perspective? Is there any flow control that the server can apply to back

Bootstrap failure

Hi all, We have been struggling with the inability to bootstrap nodes into our 1.2.13 environment with Vnodes using centos 6.4 with Java 7. We have an 8 node cluster (32 GB RAM, dual hex core, SSDs, 8 GB heap with 1200 MB eden space, RF3) with around 1 TB per node using murmur3. When we

Re: what tool will create noncql columnfamilies in cassandra 3a

I agree having two tools is no good. I ended up discussing this in the IRC channel for a bit yesterday for this column family. 1. create column family BrowserStuff 2. with column_type = 'Standard' 3. and comparator = 'UTF8Type' 4. and default_validation_class = 'UTF8Type' 5.

Re: what tool will create noncql columnfamilies in cassandra 3a

> Cassandra 2.0.4 cli is informing me that it will no longer exist in the next major. I know you meant 3.0 by "next major" but just for the sake of not confusing anyone reading, the current plan is to remove the cassandra-cli in Cassandra 3.0 which will not be the next major release (the next majo