I am running cassandra 0.7.8. pycassa 1.1.0 Nodes=7, RF=3
This problem started a few months ago and only occurs sporadically. I receive notifications from paypal's IPN. The IPN data is saved into a column family. I add another column for "processed" which is set to 0. Every 5 minutes, a cron script runs and pulls down IPN events that haven't been processed. It does some work, and then writes back processed to 1. Usually this worked, but then (when exactly I don't recall), occasionally, it started having a problem. Processed wasn't been set to 1. So, IPN events would be processed twice. I tried a few things to fix this. Repair, compact, restarting the cluster, upgrading. I even did a complete rebuild of the cluster, wiping the data directory and starting fresh on 0.7.8. I then ditched the "processed" column and decide to use two column families. IPNs are put into column family A, and after being processed, it is inserted into column family B and deleted from A. The problem still persisted. At this point I was using CL.QUORUM. So, I started using CL.ALL. And the problem still persists. Having IPN events processed twice causes a fair bit of problems, so this is something I really need to get resolved. Thanks, Kyle