Hi Alain, thank you for your response. I'm using incremental repair. I'm afraid subrange repair is not a viable alternative, because it's very slow - takes over a week to complete.
I've found at least a partial solution - specifying '-local' or '-dc' parameter will also disable anticompaction, but the repair will skip SSTables that are already marked as repaired. Our data is about 50% repaired, so this significantly reduces repair time. What if I ran 'sstablerepairedset --really-set --is-repaired' on every table that was repaired by a subrange repair? Would it prevent these tables from being anticompacted, and allow us to use incremental repair again? Regards, Martin On Wed, Sep 12, 2018 at 1:31 PM Alain RODRIGUEZ <arodr...@gmail.com> wrote: > > Hello Martin, > > How do you perform the repairs? > > Are you using incremental repairs or full repairs but without subranges? Alex > described issues related to these repairs here: > http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html. > > tl;dr: > >> The only way to perform repair without anticompaction in “modern” versions >> of Apache Cassandra is subrange repair, which fully skips anticompaction. To >> perform a subrange repair correctly, you have three options : >> - Compute valid token subranges yourself and script repairs accordingly >> - Use the Cassandra range repair script which performs subrange repair >> - Use Cassandra Reaper, which also performs subrange repair > > > If you can prevent anti-compaction, disk space growth should be more > predictable. > > There might be more solutions now out there, C* should also soon be shipped > with a side-car it's being actively discussed. Finally, Incremental repairs > will receive important fixes in Cassandra 4.0, Alex also wrote about this too > (yes, this guy loves repairs ¯\_(ツ)_/¯) > http://thelastpickle.com/blog/2018/09/10/incremental-repair-improvements-in-cassandra-4.html > > I believe (and hope) this information is relevant to help you fix this issue. > > C*heers, > ----------------------- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > Le mer. 12 sept. 2018 à 10:14, Martin Mačura <m.mac...@gmail.com> a écrit : >> >> Hi, >> we're on cassandra 3.11.2 . During an anticompaction after repair, >> TotalDiskSpaceUsed value of one table gradually went from 700GB to >> 1180GB, and then suddenly jumped back to 700GB. This happened on all >> nodes involved in the repair. There was no change in PercentRepaired >> during or after this process. SSTable count is currently 857, with a >> peak of 2460 during the repair. >> >> Table is using TWCS with 1-day time window. Most daily SSTables are >> around 1 GB but the oldest one is 156 GB - caused by a major >> compaction. >> >> system.log.6.zip:INFO [CompactionExecutor:9923] 2018-09-10 >> 15:29:54,238 CompactionManager.java:649 - [repair >> #88c36e30-b4cb-11e8-bebe-cd3efd73ed33] Starting anticompaction for ... >> on 519 [...] SSTables >> ... >> system.log:INFO [CompactionExecutor:9923] 2018-09-12 00:29:39,262 >> CompactionManager.java:1524 - Anticompaction completed successfully, >> anticompacted from 0 to 518 sstable(s). >> >> What could be the cause of the temporary increase, and how can we >> prevent it? We are concerned about running out of disk space soon. >> >> Thanks for any help >> >> Martin >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
