Re: Real time bad query logging framework in C*

2018-06-20 Thread Jaydeep Chovatia
Thanks Stefan for reviewing this, please find my comments inline: >We already provide tons of metrics and provide some useful logging (e.g. when reading too many tombstones), but I think we should still be able to implement further >checks in-code that highlight potentially issues. Maybe we could

Re: Tombstone passed GC period causes un-repairable inconsistent data

2018-06-20 Thread kurt greaves
That makes sense going forward (assuming it works), but this is still pretty surprising behaviour. Although disregarding the read repair factor entirely, the result will *eventually* come true when the tombstones are purged, we're still returning a result that doesn't match up with what we have on

Re: Tombstone passed GC period causes un-repairable inconsistent data

2018-06-20 Thread sankalp kohli
I agree with Stefan that we should use incremental repair and use patches from Marcus to drop tombstones only from repaired data. Regarding deep repair, you can bump the read repair and run the repair. The issue will be that you will stream lot of data and also your blocking read repair will go up

Re: Real time bad query logging framework in C*

2018-06-20 Thread Stefan Podkowinski
Jaydeep, thanks for taking this discussion to the dev list. I think it's the best place to introduce new idea, discuss them in general and how they potentially fit in. As already mention in the ticket, I do share your assessment that we should try to improve making operational issue more visible to

Re: Tombstone passed GC period causes un-repairable inconsistent data

2018-06-20 Thread Stefan Podkowinski
Sounds like an older issue that I tried to address two years ago: https://issues.apache.org/jira/browse/CASSANDRA-11427 As you can see, the result hasn't been as expected and we got some unintended side effects based on the patch. I'm not sure I'd be willing to give this another try, considering t