Re: consistency ONE and "null"

that makes sense. thanks! On Apr 7, 2011, at 8:36 AM, Stephen Connolly wrote: > also there is a configuration parameter that controls the probability of any > read request triggering a read repair > > - Stephen > > --- > Sent from my Android phone, so random spelling mistakes, random nonsense

problem with large batch mutation set

Hi, I am using the thrift client batch_mutate method with Cassandra 0.7.0 on Ubuntu 10.10. When the size of the mutations gets too large, the client fails with the following exception: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at

Re: consistency ONE and "null"

also there is a configuration parameter that controls the probability of any read request triggering a read repair - Stephen --- Sent from my Android phone, so random spelling mistakes, random nonsense words and other nonsense are a direct result of using swype to type on the screen On 7 Apr 2011

Re: consistency ONE and "null"

as I understand, the read repair is a background task triggered by the read request, but once the consistency requirement has been met you will be given a response. the coordinator at CL.ONE is allowed to return your responce once it has one response (empty or not) from any replica. if the first r

Re: Stress tests failed with secondary index

Thank you Aaron. It does not seem to be an overload problem. I have 16 cores and 48G ram on the single node, and I reduced the concurrent threads to be 1. Still, it just suddenly dies of a timeout, while the cpu, ram, disk load are below 10% and write latency is about 0.5ms for the past 10 minute

Re: RE: batch_mutate failed: out of sequence response

"out of sequence response" is thrift's way of saying "I got a response for request Y when I expected request X." my money is on using a single connection from multiple threads. don't do that. On Wed, Apr 6, 2011 at 5:59 PM, Dan Washusen wrote: > Pelops will retry when TimedOutException, TTransp

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

Hypothesis: it's probably the flush causing the CMS, not the snapshot linking. Confirmation possibility #1: Add a logger.warn to CLibrary.createHardLinkWithExec -- with JNA enabled it shouldn't be called, but let's rule it out. Confirmation possibility #2: Force some flushes w/o snapshot. Either

Re: Compaction threshold does not save with nodetool

Thanks. I think this feature could be clarified on the wiki. 2011/4/6 Dan Hendry > There are two layers of settings, the default, cluster wide, settings part > of the schema and exposed/modifiable via the cli and individual settings > exposed/modifiable via JMX and nodetool. Using nodetool, you

Re: Stress tests failed with secondary index

TimedOutException means that the less than CL number of nodes responded to the coordinator before the rpc_timeout. So it was overloaded. Which makes sense when you say it only happens with secondary indexes. Consider things like - reducing the throughput - reducing the number of clients - ensuri

Re: Test idea on cassandra

Stress tools in contrib directory use multiple threads/processes. 2011/4/7 Mengchen Yu > I'm trying to simulate a multi-user scenario. The reason why I > want to use MPJ is to create different processes act like individual > users. Do any one have idea how to do this clearly? > Sorry for duplica

Re: Abnormal memory consumption

Hello Paul, Thank you for the tip. The random port attribution policy of JMX was really making me mad ! Good to know there is a solution for that problem. Concerning the rest of the conversation, my only concern is that as an administrator and a student it is hard to constantly watch Cassandra i

consistency ONE and "null"

Let's say you have RF of 3 and a write was written to 2 nodes. 1 was not written because the node had a network hiccup (but came back online again). My question is, if you are reading a key with a CL of ONE, and you happen to land on that node that didn't get the write, will the read fail imm

Re: nodetool repair & compact

Thanks a lot. It has became clear for me. From iPhone On 2011/04/06, at 23:51, Sylvain Lebresne wrote: > On Tue, Apr 5, 2011 at 9:03 PM, Maki Watanabe wrote: >> Thanks Sylvain, it's very clear. >> But should I still need to force major compaction regularly to clear >> tombstones? >> I know t

Re: RE: batch_mutate failed: out of sequence response

Pelops will retry when TimedOutException, TTransportException or UnavailableException exceptions are thrown but not TApplicationException. TApplicationException has a type property which looks like it could be used to retry based on specific values. Based on the names the INTERNAL_ERROR and BAD

Re: CL.ONE reads / RR / badness_threshold interaction

Ok, I took this opportunity to look look a bit more on this part of the code. My reading of StorageProxy.fetchRows() and related is as follows, but please allow for others to say I'm wrong/missing something (and sorry, this is more a stream of consciousness that is probably more useful to me for le

Re: Abnormal memory consumption

You can use JMX over ssh by doing this: http://blog.reactive.org/2011/02/connecting-to-cassandra-jmx-via-ssh.html Basically, you use SSH -D to do dynamic application port forwarding. In terms of scaling, you'll be able to afford 120GB RAM/node in 3 years if you're successful. Or, a machine with m

Test idea on cassandra

I'm trying to simulate a multi-user scenario. The reason why I want to use MPJ is to create different processes act like individual users. Do any one have idea how to do this clearly? Sorry for duplicated mails if any

CL.ONE reads / RR / badness_threshold interaction

My understanding for For CL.ONE. For the node that receives the request: (A) If RR is enabled and this node contains the needed row --> return immediately and do RR to remaining replicas in background. (B) If RR is off and this node contains the needed row --> return the needed data immediately.

Flush / Snapshot Triggering Full GCs, Leaving Ring

Hello, We're running a six-node 0.7.4 ring in EC2 on m1.xlarge instances with 4GB heap (15GB total memory, 4 cores, dataset fits in RAM, storage on ephemeral disk). We've noticed a brief flurry of query failures during the night corresponding with our backup schedule. More specifically, our log

Re: Is Cassandra and MPI compatible?

https://github.com/pcmanus/ccm is probably better. On Wed, Apr 6, 2011 at 12:41 PM, Mengchen Yu wrote: > I'm thinking about using MPI (MPI Java actually) to test cassandra on > my laptop before test it in clusters. > Is there anything I'd better look to know more? > > Thanks. > -- Jonathan El

Re: Introduce Random latency

Thanks! This is a much better way. On Wed, Apr 6, 2011 at 1:27 PM, Scott Brooks wrote: > I'm not sure of your exact requirements for why you would do that > there, but have you thought about putting the delay in the network > instead? > http://stackoverflow.com/questions/614795/simulate-delayed-a

Re: LB scenario

I think best is to have global load balancer in front of web servers/app servers. And leave app servers to handle requests at local quoram. If data center goes down then load balancer will simply hand out only one DCs ips. -- View this message in context: http://cassandra-user-incubator-apache-or

Is Cassandra and MPI compatible?

I'm thinking about using MPI (MPI Java actually) to test cassandra on my laptop before test it in clusters. Is there anything I'd better look to know more? Thanks.

Re: 0.7.4 - Cassandra Nodes Do Not Start

apparently ... it occurs in mine too ... it doesn't appear to cause harm to anything ... On Wed, Apr 6, 2011 at 7:24 PM, mcasandra wrote: > I see this error in the logs posted. Is this normal? > > java.io.IOError: java.io.EOFException >        at > org.apache.cassandra.net.IncomingTcpConnection.r

Re: Introduce Random latency

I'm not sure of your exact requirements for why you would do that there, but have you thought about putting the delay in the network instead? http://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux That SO page has a pretty good description of how you can do it on t

Re: LB scenario

I have done some more research. My question now is: 1. From my tests I see that it matters a lot whether the co-ordinator node is local (geographically) to the client or not. I will have a cassandra cluster where nodes will be distributed across the globe. Say 3 in US east_cost, 3 in US west_coast

Re: 0.7.4 - Cassandra Nodes Do Not Start

I see this error in the logs posted. Is this normal? java.io.IOError: java.io.EOFException at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at or

Introduce Random latency

I want to run some tests where I incorporate random latency in some of the nodes (that varies over time). I think the best place is to place a 'sleep (random(10 to 50 seconds)) function' somewhere in the source code of cassandra. Or maybe the sleep is not random but the seconds to sleep are read fr

Re: Location-aware replication based on objects' access pattern

I use the global ring to provide quick reference data. I don't see it as a bottle neck because there isn't a lot of data floating around it. Just enough to satisfy specific use cases we have. If the end user needs to access the data in the US ring while in Asia via the web, I would opt to transf

Re: Secondary Index keeping track of column names

No, 0.7 indexes handle equality queries; you're basically asking for a IS NOT NULL query. On Wed, Apr 6, 2011 at 11:23 AM, Jeremiah Jordan wrote: >        In 0.7.X is there a way to have an automatic secondary index > which keeps track of what keys contain a certain column?  Right now we > are ke

Re: Location-aware replication based on objects' access pattern

On Wed, Apr 6, 2011 at 3:55 AM, Sasha Dolgy wrote: > I had been asked this question from a strategy point of view, and > referenced how linkedin.com appears to handle this. > > > Specific region data is stored on a ring in that region.  While based > in the middle east, my linkedin.com profile wa

Re: Minor Follow-up: reduced cached mem; resident set size growth

On 04/05/2011 04:38 PM, Peter Schuller wrote: >> - Different collectors: -XX:+UseParallelGC -XX:+UseParallelOldGC > > Unless you also removed the -XX:+UseConcMarkSweepGC I *think* it takes > precedence, so that the above options would have no effect. I didn't > test. In either case, did you defini

Secondary Index keeping track of column names

In 0.7.X is there a way to have an automatic secondary index which keeps track of what keys contain a certain column? Right now we are keeping track of this manually, so we can quickly get all of the rows which contain a given column, it would be nice if it was automatic. -Jeremiah _

Re: 0.7.4 - Cassandra Nodes Do Not Start

Nothing looks suspicious there. Try casandra -f, no & On Wed, Apr 6, 2011 at 10:20 AM, B R wrote: > Hi guys, > > We are running a 2 node Cassandra cluster which has suddenly started giving > errors and the nodes do not start. I am pasting the start-up logs below. Any > help would be great. Thank

0.7.4 - Cassandra Nodes Do Not Start

Hi guys, We are running a 2 node Cassandra cluster which has suddenly started giving errors and the nodes do not start. I am pasting the start-up logs below. Any help would be great. Thanks. Error Log 1 [root@idata1_34 bin]# ./cassandra & [1] 16945 [root@idata1_34 bin]# INFO 20:33:16,550 Loggin

Re: nodetool repair & compact

On Tue, Apr 5, 2011 at 9:03 PM, Maki Watanabe wrote: > Thanks Sylvain, it's very clear. > But should I still need to force major compaction regularly to clear > tombstones? > I know that minor compaction clear the tombstones after 0.7, but > maximumCompactionThreshold limits the maximum number of

RE: Compaction threshold does not save with nodetool

There are two layers of settings, the default, cluster wide, settings part of the schema and exposed/modifiable via the cli and individual settings exposed/modifiable via JMX and nodetool. Using nodetool, you are only modifying the in memory settings for a single node, changes to those settings are

Stress tests failed with secondary index

I used py_stress module to insert 10m test data with a secondary index. I got the following exceptions. # python stress.py -d xxx -o insert -n 1000 -c 5 -s 34 -C 5 -x keys total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time 265322,26532,26541,0.00186140829433,10 630300,36497,3650

Re: Disable Swap? batch_mutate failed: out of sequence response

El mié, 06-04-2011 a las 09:18 +0200, Héctor Izquierdo Seliva escribió: > I took a look at vmstats, and there was no swap. Also, our monitoring > tools showed no swap being used at all. It's running with mlockall and > all that. 8GB heap on a 16GB machine > I tried disabling swap completely, and

Re: RE: batch_mutate failed: out of sequence response

El mié, 06-04-2011 a las 09:06 +1000, Dan Washusen escribió: > Pelops raises a RuntimeException? Can you provide more info please? > org.scale7.cassandra.pelops.exceptions.ApplicationException: batch_mutate failed: out of sequence response > -- > Dan Washusen > Make big files fly > visit digit

Re: Location-aware replication based on objects' access pattern

I had been asked this question from a strategy point of view, and referenced how linkedin.com appears to handle this. Specific region data is stored on a ring in that region. While based in the middle east, my linkedin.com profile was kept on the middle east part of linkedin.com ... when I moved

Re: Location-aware replication based on objects' access pattern

good to see a discussion on this. This also has practical use for business continuity where you can control that the clients in a given data center first write replicas to its own data center, then to the other data center for backup. If I understand correctly, a write takes the token into ac

Re: Disable Swap? batch_mutate failed: out of sequence response

I took a look at vmstats, and there was no swap. Also, our monitoring tools showed no swap being used at all. It's running with mlockall and all that. 8GB heap on a 16GB machine El mar, 05-04-2011 a las 21:24 +0200, Peter Schuller escribió: > > Would you recommend to disable system swap as a rule?