Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-28 Thread Kyle Gibson
Thanks for the reply Peter, you may have discovered the problem, I'll explain below. On Sun, Aug 28, 2011 at 8:51 AM, Peter Schuller wrote: >> Understood. In the code example I provided, I am writing the same >> value, but I am doing so in quick succession, so perhaps a few second >> sleep might

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-28 Thread Peter Schuller
> Understood. In the code example I provided, I am writing the same > value, but I am doing so in quick succession, so perhaps a few second > sleep might be helpful. It is worth noting also that the code I > provided is only the second step 2 in the process. There is a php > script that receives th

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-26 Thread Kyle Gibson
Update: I scaled my cluster down from 7 nodes to 3 nodes, and kept RF=3. I did a complete cluster rebuild, so everything was fresh. Kept my reads and writes at CL.ALL. For a while there it seemed like I had succeeded in eliminating the problem. Unfortunately about an hour ago a duplicate came thro

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-22 Thread Kyle Gibson
Thanks for the reply. On Mon, Aug 22, 2011 at 1:11 PM, Dominic Williams wrote: > Hi there, here's my tuppence! > 1. Something to look at first: > > If you write two different values to the same column quickly in succession, > if both writes go out with the same timestamp, then it is indeterminate

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-22 Thread Dominic Williams
Hi there, here's my tuppence! 1. Something to look at first: If you write two different values to the same column quickly in succession, if both writes go out with the same timestamp, then it is indeterminate which one wins i.e. write order doesn't necessarily matter. 2. Don't trust PayPal (anyo

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-22 Thread Kyle Gibson
I made some changes to my code base that uses cassandra. I went back to using the "processed" column, but instead of using "0" or "1" I decided to use "no" and "yes" You can view the code here: http://pastebin.com/gRBC16e7 As you can see from the code, I perform an insert, get, check the result,

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-20 Thread Peter Schuller
> Do you mean the cassandra log, or just logging in the script itself? The script itself. I.e, some "independent" verification that the line of code after the insert is in fact running, just in case there's some kind of silent failure. Sounds like you've tried to address it though with the E-Mail

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-19 Thread Kyle Gibson
The cron script doesn't do much. It pulls new IPNs (usually only 1 in a given 5 minute period), inserts a row, and then sends an email. As for failure handling in the script itself, I rely on python exception handling, and whenever an exception occurs I do get an email with the exception details.

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-19 Thread Peter Schuller
> Is it possible for instance that sometimes your cron job takes longer > than five minutes? Or just a lack of failure handling in the cron job for that matter. Are you *SURE* the the "processed" flag truly got set? Do you have a log statement (written *AFTER* successful write to Cassandra) that i

Re: Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-18 Thread Jonathan Ellis
There are a lot of people on 0.7 for whom CL is working as advertised. Not saying it's impossible that there's a bug, but the odds are against it. Is it possible for instance that sometimes your cron job takes longer than five minutes? On Thu, Aug 18, 2011 at 9:49 AM, Kyle Gibson wrote: > I am

Occasionally getting old data back with ConsistencyLevel.ALL

2011-08-18 Thread Kyle Gibson
I am running cassandra 0.7.8. pycassa 1.1.0 Nodes=7, RF=3 This problem started a few months ago and only occurs sporadically. I receive notifications from paypal's IPN. The IPN data is saved into a column family. I add another column for "processed" which is set to 0. Every 5 minutes, a cron sc