That is not happening anymore since I am repairing a keyspace with much less data (the other one is still there in write-only mode). The command I am using is the most boring (even shed the -pr option so to keep anticompactions to a minimum): nodetool -h localhost repair <keyspace> It's executed sequentially on each node (no overlapping, next node waits for the previous to complete).
Regards, Stefano Ortolani On Mon, Oct 31, 2016 at 11:18 PM, kurt Greaves <[email protected]> wrote: > Blowing out to 1k SSTables seems a bit full on. What args are you passing to > repair? > > Kurt Greaves > [email protected] > www.instaclustr.com > > On 31 October 2016 at 09:49, Stefano Ortolani <[email protected]> wrote: >> >> I've collected some more data-points, and I still see dropped >> mutations with compaction_throughput_mb_per_sec set to 8. >> The only notable thing regarding the current setup is that I have >> another keyspace (not being repaired though) with really wide rows >> (100MB per partition), but that shouldn't have any impact in theory. >> Nodes do not seem that overloaded either and don't see any GC spikes >> while those mutations are dropped :/ >> >> Hitting a dead end here, any further idea where to look for further ideas? >> >> Regards, >> Stefano >> >> On Wed, Aug 10, 2016 at 12:41 PM, Stefano Ortolani <[email protected]> >> wrote: >> > That's what I was thinking. Maybe GC pressure? >> > Some more details: during anticompaction I have some CFs exploding to 1K >> > SStables (to be back to ~200 upon completion). >> > HW specs should be quite good (12 cores/32 GB ram) but, I admit, still >> > relying on spinning disks, with ~150GB per node. >> > Current version is 3.0.8. >> > >> > >> > On Wed, Aug 10, 2016 at 12:36 PM, Paulo Motta <[email protected]> >> > wrote: >> >> >> >> That's pretty low already, but perhaps you should lower to see if it >> >> will >> >> improve the dropped mutations during anti-compaction (even if it >> >> increases >> >> repair time), otherwise the problem might be somewhere else. Generally >> >> dropped mutations is a signal of cluster overload, so if there's >> >> nothing >> >> else wrong perhaps you need to increase your capacity. What version are >> >> you >> >> in? >> >> >> >> 2016-08-10 8:21 GMT-03:00 Stefano Ortolani <[email protected]>: >> >>> >> >>> Not yet. Right now I have it set at 16. >> >>> Would halving it more or less double the repair time? >> >>> >> >>> On Tue, Aug 9, 2016 at 7:58 PM, Paulo Motta <[email protected]> >> >>> wrote: >> >>>> >> >>>> Anticompaction throttling can be done by setting the usual >> >>>> compaction_throughput_mb_per_sec knob on cassandra.yaml or via >> >>>> nodetool >> >>>> setcompactionthroughput. Did you try lowering that and checking if >> >>>> that >> >>>> improves the dropped mutations? >> >>>> >> >>>> 2016-08-09 13:32 GMT-03:00 Stefano Ortolani <[email protected]>: >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> I am running incremental repaird on a weekly basis (can't do it >> >>>>> every >> >>>>> day as one single run takes 36 hours), and every time, I have at >> >>>>> least one >> >>>>> node dropping mutations as part of the process (this almost always >> >>>>> during >> >>>>> the anticompaction phase). Ironically this leads to a system where >> >>>>> repairing >> >>>>> makes data consistent at the cost of making some other data not >> >>>>> consistent. >> >>>>> >> >>>>> Does anybody know why this is happening? >> >>>>> >> >>>>> My feeling is that this might be caused by anticompacting column >> >>>>> families with really wide rows and with many SStables. If that is >> >>>>> the case, >> >>>>> any way I can throttle that? >> >>>>> >> >>>>> Thanks! >> >>>>> Stefano >> >>>> >> >>>> >> >>> >> >> >> > > >
