Re: Cassandra reverting deletes?

2010-04-30 Thread Joost Ouwerkerk
Great, thank you. Do you have a hypothesis about where things might be going wrong? Let me know what I can do to help. On Fri, Apr 30, 2010 at 9:33 AM, Jonathan Ellis wrote: > https://issues.apache.org/jira/browse/CASSANDRA-1040 > > On Thu, Apr 29, 2010 at 6:55 PM, Joost Ouwerkerk wrote: >> Ok

Re: Cassandra reverting deletes?

2010-04-30 Thread Jonathan Ellis
https://issues.apache.org/jira/browse/CASSANDRA-1040 On Thu, Apr 29, 2010 at 6:55 PM, Joost Ouwerkerk wrote: > Ok, I reproduced without mapred.  Here is my recipe: > > On a single-node cassandra cluster with basic config (-Xmx:1G) > loop { >   * insert 5,000 records in a single columnfamily with

Re: Cassandra reverting deletes?

2010-04-29 Thread Joost Ouwerkerk
Ok, I reproduced without mapred. Here is my recipe: On a single-node cassandra cluster with basic config (-Xmx:1G) loop { * insert 5,000 records in a single columnfamily with UUID keys and random string values (between 1 and 1000 chars) in 5 different columns spanning two different supercolumn

Re: Cassandra reverting deletes?

2010-04-28 Thread Jonathan Ellis
Good! :) Can you reproduce w/o map/reduce, with raw get_range_slices? On Wed, Apr 28, 2010 at 3:56 PM, Joost Ouwerkerk wrote: > Yes! Reproduced on single-node cluster: > > 10/04/28 16:30:24 INFO mapred.JobClient:     ROWS=274884 > 10/04/28 16:30:24 INFO mapred.JobClient:     TOMBSTONES=951083 >

Re: Cassandra reverting deletes?

2010-04-28 Thread Joost Ouwerkerk
Yes! Reproduced on single-node cluster: 10/04/28 16:30:24 INFO mapred.JobClient: ROWS=274884 10/04/28 16:30:24 INFO mapred.JobClient: TOMBSTONES=951083 10/04/28 16:42:49 INFO mapred.JobClient: ROWS=166580 10/04/28 16:42:49 INFO mapred.JobClient: TOMBSTONES=1059387 On Wed, Apr 28,

Re: Cassandra reverting deletes?

2010-04-28 Thread Jonathan Ellis
It sounds like either there is a fairly obvious bug, or you're doing something wrong. :) Can you reproduce against a single node? On Tue, Apr 27, 2010 at 5:14 PM, Joost Ouwerkerk wrote: > Update: I ran a test whereby I deleted ALL the rows in a column > family, using a consistency level of ALL.

Re: Cassandra reverting deletes?

2010-04-27 Thread Joost Ouwerkerk
Update: I ran a test whereby I deleted ALL the rows in a column family, using a consistency level of ALL. To do this, I mapped the ColumnFamily and called remove on each row id. There were 1.5 million rows, so 1.5 million rows were deleted. I ran a counter job immediately after. This job maps t

Re: Cassandra reverting deletes?

2010-04-27 Thread Joost Ouwerkerk
Clocks are in sync: cluster04:~/cassandra$ dsh -g development "date" Tue Apr 27 17:36:33 EDT 2010 Tue Apr 27 17:36:33 EDT 2010 Tue Apr 27 17:36:33 EDT 2010 Tue Apr 27 17:36:33 EDT 2010 Tue Apr 27 17:36:34 EDT 2010 Tue Apr 27 17:36:34 EDT 2010 Tue Apr 27 17:36:34 EDT 2010 Tue Apr 27 17:36:34 EDT 20

Re: Cassandra reverting deletes?

2010-04-27 Thread Nathan McCall
Have you confirmed that your clocks are all synced in the cluster? This may be the result of an unintentional read-repair occurring if that were the case. -Nate On Tue, Apr 27, 2010 at 2:20 PM, Joost Ouwerkerk wrote: > Hmm... Even after deleting with cl.ALL, I'm getting data back for some > rows

Re: Cassandra reverting deletes?

2010-04-27 Thread Joost Ouwerkerk
Hmm... Even after deleting with cl.ALL, I'm getting data back for some rows after having deleted them. Which rows return data is inconsistent from one run of the job to the next. On Tue, Apr 27, 2010 at 1:44 PM, Joost Ouwerkerk wrote: > To check that rows are gone, I check that KeySlice.columns

Re: Cassandra reverting deletes?

2010-04-27 Thread Joost Ouwerkerk
To check that rows are gone, I check that KeySlice.columns is empty. And as I mentioned, immediately after the delete job, this returns the expected number. Unfortunately I reproduced with QUORUM this morning. No node outages. I am going to try ALL to see if that changes anything, but I am star

Re: Cassandra reverting deletes?

2010-04-26 Thread Jonathan Ellis
How are you checking that the rows are gone? Are you experiencing node outages during this? DC_QUORUM is unfinished code right now, you should avoid using it. Can you reproduce with normal QUORUM? On Sat, Apr 24, 2010 at 12:23 PM, Joost Ouwerkerk wrote: > I'm having trouble deleting rows in Cas

Cassandra reverting deletes?

2010-04-24 Thread Joost Ouwerkerk
I'm having trouble deleting rows in Cassandra. After running a job that deletes hundreds of rows, I run another job that verifies that the rows are gone. Both jobs run correctly. However, when I run the verification job an hour later, the rows have re-appeared. This is not a case of "ghosting"