Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
Sure is: scale7-pelops-1.1-0.7.x-20110418.225848-3.jar Cheers, -- Dan Washusen On Tuesday, 19 April 2011 at 3:58 PM, Héctor Izquierdo Seliva wrote: > Thanks Dan for fixing that! Is the change integrated in the latest maven > snapshot? > > El mar, 19-04-2011 a las 10:48 +1000, Dan Washusen escri

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Héctor Izquierdo Seliva
Thanks Dan for fixing that! Is the change integrated in the latest maven snapshot? El mar, 19-04-2011 a las 10:48 +1000, Dan Washusen escribió: > An example scenario (that is now fixed in Pelops): > 1. Attempt to write a column with a null value > 2. Cassandra throws a TProtocolException

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
An example scenario (that is now fixed in Pelops): Attempt to write a column with a null value Cassandra throws a TProtocolException which renders the connection useless for future operations Pelops returns the corrupt connection to the pool A second read operation is attempted with the corrupt co

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Jonathan Ellis
Any idea what's causing the original TPE? On Mon, Apr 18, 2011 at 6:22 PM, Dan Washusen wrote: > It turns out that once a TProtocolException is thrown from Cassandra the > connection is useless for future operations. Pelops was closing connections > when it detected TimedOutException, TTransportE

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
It turns out that once a TProtocolException is thrown from Cassandra the connection is useless for future operations. Pelops was closing connections when it detected TimedOutException, TTransportException and UnavailableException but not TProtocolException. We have now changed Pelops to close c

Re: RE: batch_mutate failed: out of sequence response

2011-04-14 Thread Dan Washusen
I've looked over the Pelops code again and I really can't see how it could be at fault here... -- Dan Washusen On Wednesday, 13 April 2011 at 3:20 AM, Stephen McKamey wrote: > [I wrote this Apr 10, 2011 at 12:09 but my message seems to have gotten lost > along the way.] > > I use Pelops (the

RE: batch_mutate failed: out of sequence response

2011-04-12 Thread Stephen McKamey
[I wrote this Apr 10, 2011 at 12:09 but my message seems to have gotten lost along the way.] I use Pelops (the 1.0-0.7.x build from the Github Maven repo) and have occasionally seen this message (under load or during GC). I have a test app running in two separate single-threaded processes doing a

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Dan Washusen
Pelops uses a single connection per operation from a pool that is backed by Apache Commons Pool (assuming you're using Cassandra 0.7). I'm not saying it's perfect but it's NOT sharing a connection over multiple threads. Dan Hendry mentioned that he sees these errors. Is he also using Pelops? Fro

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió: > "out of sequence response" is thrift's way of saying "I got a response > for request Y when I expected request X." > > my money is on using a single connection from multiple threads. don't do > that. > I'm not using thrift direct

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió: > "out of sequence response" is thrift's way of saying "I got a response > for request Y when I expected request X." > > my money is on using a single connection from multiple threads. don't do > that. I'm not using thrift direc

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Jonathan Ellis
vide more info please? > > org.scale7.cassandra.pelops.exceptions.ApplicationException: > batch_mutate failed: out of sequence response > > -- > Dan Washusen > Make big files fly > visit digitalpigeon.com > > On Tuesday, 5 April 2011 at 11:43 PM, Héctor Izquierdo Seliva wr

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Dan Washusen
> org.scale7.cassandra.pelops.exceptions.ApplicationException: > batch_mutate failed: out of sequence response > > > -- > > Dan Washusen > > Make big files fly > > visit digitalpigeon.com > > > > On Tuesday, 5 April 2011 at 11:43 PM, Héctor Izquierdo Seliva wrote: > >

Re: Disable Swap? batch_mutate failed: out of sequence response

2011-04-06 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 09:18 +0200, Héctor Izquierdo Seliva escribió: > I took a look at vmstats, and there was no swap. Also, our monitoring > tools showed no swap being used at all. It's running with mlockall and > all that. 8GB heap on a 16GB machine > I tried disabling swap completely, and

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 09:06 +1000, Dan Washusen escribió: > Pelops raises a RuntimeException? Can you provide more info please? > org.scale7.cassandra.pelops.exceptions.ApplicationException: batch_mutate failed: out of sequence response > -- > Dan Washusen > Make big fi

Re: Disable Swap? batch_mutate failed: out of sequence response

2011-04-06 Thread Héctor Izquierdo Seliva
I took a look at vmstats, and there was no swap. Also, our monitoring tools showed no swap being used at all. It's running with mlockall and all that. 8GB heap on a 16GB machine El mar, 05-04-2011 a las 21:24 +0200, Peter Schuller escribió: > > Would you recommend to disable system swap as a rule?

Re: RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Dan Washusen
our response! > > Héctor > > > -Original Message- > > From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] > > Sent: April-05-11 8:30 > > To: user@cassandra.apache.org > > Subject: batch_mutate failed: out of sequence response > > > > Hi

Re: Disable Swap? batch_mutate failed: out of sequence response

2011-04-05 Thread Peter Schuller
> Would you recommend to disable system swap as a rule?   I'm running on Debian > 64bit and am seeing light swapping: I'm not Jonathan, but *yes*. I would go so far as to say that disabling swap is a good rule of thumb for *most* production systems that serve latency sensitive traffic. For a mach

Disable Swap? batch_mutate failed: out of sequence response

2011-04-05 Thread Jonathan Colby
round 5 seconds. I'm running cassandra with a heap of 8 GB. Should I tune >> this somehow? >> >> Is any of this wrong? >>> >>>>> -Original Message- >>>>> From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] >>>

Re: batch_mutate failed: out of sequence response

2011-04-05 Thread Jonathan Ellis
---Original Message- >> > > From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] >> > > Sent: April-05-11 8:30 >> > > To: user@cassandra.apache.org >> > > Subject: batch_mutate failed: out of sequence response >> > > >> > > Hi e

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
t: April-05-11 8:30 > > > To: user@cassandra.apache.org > > > Subject: batch_mutate failed: out of sequence response > > > > > > Hi everyone. I'm having trouble while inserting big amounts of data into > > > cassandra. I'm getting this exception: >

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
> -Original Message- > > From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] > > Sent: April-05-11 8:30 > > To: user@cassandra.apache.org > > Subject: batch_mutate failed: out of sequence response > > > > Hi everyone. I'm having trouble while

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
your response! Héctor > -Original Message- > From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] > Sent: April-05-11 8:30 > To: user@cassandra.apache.org > Subject: batch_mutate failed: out of sequence response > > Hi everyone. I'm having trouble while

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Dan Hendry
age/xc3tskhhvsf5awz7). What OS are you running? Dan -Original Message- From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com] Sent: April-05-11 8:30 To: user@cassandra.apache.org Subject: batch_mutate failed: out of sequence response Hi everyone. I'm having trouble while inserting big am

batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
Hi everyone. I'm having trouble while inserting big amounts of data into cassandra. I'm getting this exception: batch_mutate failed: out of sequence response I'm gessing is due to very big mutates. I have made the batch mutates smaller and it seems to be behaving. Can somebody