Re: Deleted columns reappear after repair

2015-03-26 Thread Robert Coli
On Wed, Mar 25, 2015 at 6:53 PM, Roman Tkachenko ro...@mailgunhq.com wrote: Yup, I increased in_memory_compaction_limit_in_mb to 512MB so the row in question fits into it and ran repair on a couple of nodes owning its key. The log entries about this particular row went away and those columns

Re: Deleted columns reappear after repair

2015-03-26 Thread Roman Tkachenko
Yep, good point: https://issues.apache.org/jira/browse/CASSANDRA-9045. On Thu, Mar 26, 2015 at 4:23 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Mar 25, 2015 at 6:53 PM, Roman Tkachenko ro...@mailgunhq.com wrote: Yup, I increased in_memory_compaction_limit_in_mb to 512MB so the row

Re: Deleted columns reappear after repair

2015-03-25 Thread Roman Tkachenko
Okay, so I'm positively going crazy :) Increasing gc_grace + repair + decreasing gc_grace didn't help. The columns still appear after the repair. I checked in cassandra-cli and timestamps for these columns are old, not in the future, so it shouldn't be the reason. I also did a test: updated one

Re: Deleted columns reappear after repair

2015-03-25 Thread Robert Coli
On Wed, Mar 25, 2015 at 1:57 PM, Roman Tkachenko ro...@mailgunhq.com wrote: Okay, so I'm positively going crazy :) Increasing gc_grace + repair + decreasing gc_grace didn't help. The columns still appear after the repair. I checked in cassandra-cli and timestamps for these columns are old,

Re: Deleted columns reappear after repair

2015-03-25 Thread Roman Tkachenko
Thanks Robert. Yup, I increased in_memory_compaction_limit_in_mb to 512MB so the row in question fits into it and ran repair on a couple of nodes owning its key. The log entries about this particular row went away and those columns haven't reappeared, yet. If that was the reason, that's

Re: Deleted columns reappear after repair

2015-03-24 Thread Duncan Sands
Hi Roman, On 24/03/15 18:05, Roman Tkachenko wrote: Hi Duncan, Thanks for the response! I can try increasing gc_grace_seconds and run repair on all nodes. It does not make sense though why all *new* deletes (for the same column that resurrects after repair) I do are forgotten as well after

Re: Deleted columns reappear after repair

2015-03-24 Thread Roman Tkachenko
Hi Duncan, Thanks for the response! I can try increasing gc_grace_seconds and run repair on all nodes. It does not make sense though why all *new* deletes (for the same column that resurrects after repair) I do are forgotten as well after repair? Doesn't Cassandra insert a new tombstone every

Re: Deleted columns reappear after repair

2015-03-24 Thread Duncan Sands
Hi Roman, On 24/03/15 17:32, Roman Tkachenko wrote: Hey guys, Has anyone seen anything like this behavior or has an explanation for it? If not, I think I'm gonna file a bug report. this can happen if repair is run after the tombstone gc_grace_period has expired. I suggest you increase

Re: Deleted columns reappear after repair

2015-03-24 Thread Roman Tkachenko
Hey guys, Has anyone seen anything like this behavior or has an explanation for it? If not, I think I'm gonna file a bug report. Thanks! Roman On Mon, Mar 23, 2015 at 4:45 PM, Roman Tkachenko ro...@mailgunhq.com wrote: Hey guys, We're having a very strange issue: deleted columns get

Deleted columns reappear after repair

2015-03-23 Thread Roman Tkachenko
Hey guys, We're having a very strange issue: deleted columns get resurrected when repair is run on a node. Info about the setup. Cassandra 2.0.13, multi datacenter with 12 nodes in one datacenter and 6 nodes in another one. Schema: cqlsh describe keyspace blackbook; CREATE KEYSPACE blackbook