You are right... Repair makes the data consistent between nodes. I understand that you have 2 issues going on.
You need to run repair periodically without errors and need to decrease the numbers of compactions pending. So I suggest: 1) Run repair -pr on all nodes. If you upgrade to the new 2.1.3, you can use incremental repairs. There were some bugs on 2.1.2. 2) Run cleanup on all nodes 3) Since you have too many cold SSTables, set cold_reads_to_omit to 0.0, and increase setcompactionthroughput for some time and see if the number of SSTables is going down. Let us know what errors are you getting when running repairs. Regards, Roni Balthazar On Wed, Feb 18, 2015 at 1:31 PM, Ja Sam <ptrstp...@gmail.com> wrote: > Can you explain me what is the correlation between growing SSTables and > repair? > I was sure, until your mail, that repair is only to make data consistent > between nodes. > > Regards > > > On Wed, Feb 18, 2015 at 4:20 PM, Roni Balthazar <ronibaltha...@gmail.com> > wrote: > >> Which error are you getting when running repairs? >> You need to run repair on your nodes within gc_grace_seconds (eg: >> weekly). They have data that are not read frequently. You can run >> "repair -pr" on all nodes. Since you do not have deletes, you will not >> have trouble with that. If you have deletes, it's better to increase >> gc_grace_seconds before the repair. >> >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html >> After repair, try to run a "nodetool cleanup". >> >> Check if the number of SSTables goes down after that... Pending >> compactions must decrease as well... >> >> Cheers, >> >> Roni Balthazar >> >> >> >> >> On Wed, Feb 18, 2015 at 12:39 PM, Ja Sam <ptrstp...@gmail.com> wrote: >> > 1) we tried to run repairs but they usually does not succeed. But we had >> > Leveled compaction before. Last week we ALTER tables to STCS, because >> guys >> > from DataStax suggest us that we should not use Leveled and alter >> tables in >> > STCS, because we don't have SSD. After this change we did not run any >> > repair. Anyway I don't think it will change anything in SSTable count - >> if I >> > am wrong please give me an information >> > >> > 2) I did this. My tables are 99% write only. It is audit system >> > >> > 3) Yes I am using default values >> > >> > 4) In both operations I am using LOCAL_QUORUM. >> > >> > I am almost sure that READ timeout happens because of too much SSTables. >> > Anyway firstly I would like to fix to many pending compactions. I still >> > don't know how to speed up them. >> > >> > >> > On Wed, Feb 18, 2015 at 2:49 PM, Roni Balthazar < >> ronibaltha...@gmail.com> >> > wrote: >> >> >> >> Are you running repairs within gc_grace_seconds? (default is 10 days) >> >> >> >> >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html >> >> >> >> Double check if you set cold_reads_to_omit to 0.0 on tables with STCS >> >> that you do not read often. >> >> >> >> Are you using default values for the properties >> >> min_compaction_threshold(4) and max_compaction_threshold(32)? >> >> >> >> Which Consistency Level are you using for reading operations? Check if >> >> you are not reading from DC_B due to your Replication Factor and CL. >> >> >> >> >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html >> >> >> >> >> >> Cheers, >> >> >> >> Roni Balthazar >> >> >> >> On Wed, Feb 18, 2015 at 11:07 AM, Ja Sam <ptrstp...@gmail.com> wrote: >> >> > I don't have problems with DC_B (replica) only in DC_A(my system >> write >> >> > only >> >> > to it) I have read timeouts. >> >> > >> >> > I checked in OpsCenter SSTable count and I have: >> >> > 1) in DC_A same +-10% for last week, a small increase for last 24h >> (it >> >> > is >> >> > more than 15000-20000 SSTables depends on node) >> >> > 2) in DC_B last 24h shows up to 50% decrease, which give nice >> >> > prognostics. >> >> > Now I have less then 1000 SSTables >> >> > >> >> > What did you measure during system optimizations? Or do you have an >> idea >> >> > what more should I check? >> >> > 1) I look at CPU Idle (one node is 50% idle, rest 70% idle) >> >> > 2) Disk queue -> mostly is it near zero: avg 0.09. Sometimes there >> are >> >> > spikes >> >> > 3) system RAM usage is almost full >> >> > 4) In Total Bytes Compacted most most lines are below 3MB/s. For >> total >> >> > DC_A >> >> > it is less than 10MB/s, in DC_B it looks much better (avg is like >> >> > 17MB/s) >> >> > >> >> > something else? >> >> > >> >> > >> >> > >> >> > On Wed, Feb 18, 2015 at 1:32 PM, Roni Balthazar >> >> > <ronibaltha...@gmail.com> >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> You can check if the number of SSTables is decreasing. Look for the >> >> >> "SSTable count" information of your tables using "nodetool cfstats". >> >> >> The compaction history can be viewed using "nodetool >> >> >> compactionhistory". >> >> >> >> >> >> About the timeouts, check this out: >> >> >> >> >> >> >> http://www.datastax.com/dev/blog/how-cassandra-deals-with-replica-failure >> >> >> Also try to run "nodetool tpstats" to see the threads statistics. It >> >> >> can lead you to know if you are having performance problems. If you >> >> >> are having too many pending tasks or dropped messages, maybe will >> you >> >> >> need to tune your system (eg: driver's timeout, concurrent reads and >> >> >> so on) >> >> >> >> >> >> Regards, >> >> >> >> >> >> Roni Balthazar >> >> >> >> >> >> On Wed, Feb 18, 2015 at 9:51 AM, Ja Sam <ptrstp...@gmail.com> >> wrote: >> >> >> > Hi, >> >> >> > Thanks for your "tip" it looks that something changed - I still >> don't >> >> >> > know >> >> >> > if it is ok. >> >> >> > >> >> >> > My nodes started to do more compaction, but it looks that some >> >> >> > compactions >> >> >> > are really slow. >> >> >> > In IO we have idle, CPU is quite ok (30%-40%). We set >> >> >> > compactionthrouput >> >> >> > to >> >> >> > 999, but I do not see difference. >> >> >> > >> >> >> > Can we check something more? Or do you have any method to monitor >> >> >> > progress >> >> >> > with small files? >> >> >> > >> >> >> > Regards >> >> >> > >> >> >> > On Tue, Feb 17, 2015 at 2:43 PM, Roni Balthazar >> >> >> > <ronibaltha...@gmail.com> >> >> >> > wrote: >> >> >> >> >> >> >> >> HI, >> >> >> >> >> >> >> >> Yes... I had the same issue and setting cold_reads_to_omit to 0.0 >> >> >> >> was >> >> >> >> the solution... >> >> >> >> The number of SSTables decreased from many thousands to a number >> >> >> >> below >> >> >> >> a hundred and the SSTables are now much bigger with several >> >> >> >> gigabytes >> >> >> >> (most of them). >> >> >> >> >> >> >> >> Cheers, >> >> >> >> >> >> >> >> Roni Balthazar >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Feb 17, 2015 at 11:32 AM, Ja Sam <ptrstp...@gmail.com> >> >> >> >> wrote: >> >> >> >> > After some diagnostic ( we didn't set yet cold_reads_to_omit ). >> >> >> >> > Compaction >> >> >> >> > are running but VERY slow with "idle" IO. >> >> >> >> > >> >> >> >> > We had a lot of "Data files" in Cassandra. In DC_A it is about >> >> >> >> > ~120000 >> >> >> >> > (only >> >> >> >> > xxx-Data.db) in DC_B has only ~4000. >> >> >> >> > >> >> >> >> > I don't know if this change anything but: >> >> >> >> > 1) in DC_A avg size of Data.db file is ~13 mb. I have few a >> really >> >> >> >> > big >> >> >> >> > ones, >> >> >> >> > but most is really small (almost 10000 files are less then >> 100mb). >> >> >> >> > 2) in DC_B avg size of Data.db is much bigger ~260mb. >> >> >> >> > >> >> >> >> > Do you think that above flag will help us? >> >> >> >> > >> >> >> >> > >> >> >> >> > On Tue, Feb 17, 2015 at 9:04 AM, Ja Sam <ptrstp...@gmail.com> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> I set setcompactionthroughput 999 permanently and it doesn't >> >> >> >> >> change >> >> >> >> >> anything. IO is still same. CPU is idle. >> >> >> >> >> >> >> >> >> >> On Tue, Feb 17, 2015 at 1:15 AM, Roni Balthazar >> >> >> >> >> <ronibaltha...@gmail.com> >> >> >> >> >> wrote: >> >> >> >> >>> >> >> >> >> >>> Hi, >> >> >> >> >>> >> >> >> >> >>> You can run "nodetool compactionstats" to view statistics on >> >> >> >> >>> compactions. >> >> >> >> >>> Setting cold_reads_to_omit to 0.0 can help to reduce the >> number >> >> >> >> >>> of >> >> >> >> >>> SSTables when you use Size-Tiered compaction. >> >> >> >> >>> You can also create a cron job to increase the value of >> >> >> >> >>> setcompactionthroughput during the night or when your IO is >> not >> >> >> >> >>> busy. >> >> >> >> >>> >> >> >> >> >>> From http://wiki.apache.org/cassandra/NodeTool: >> >> >> >> >>> 0 0 * * * root nodetool -h `hostname` setcompactionthroughput >> >> >> >> >>> 999 >> >> >> >> >>> 0 6 * * * root nodetool -h `hostname` >> setcompactionthroughput 16 >> >> >> >> >>> >> >> >> >> >>> Cheers, >> >> >> >> >>> >> >> >> >> >>> Roni Balthazar >> >> >> >> >>> >> >> >> >> >>> On Mon, Feb 16, 2015 at 7:47 PM, Ja Sam <ptrstp...@gmail.com >> > >> >> >> >> >>> wrote: >> >> >> >> >>> > One think I do not understand. In my case compaction is >> >> >> >> >>> > running >> >> >> >> >>> > permanently. >> >> >> >> >>> > Is there a way to check which compaction is pending? The >> only >> >> >> >> >>> > information is >> >> >> >> >>> > about total count. >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > On Monday, February 16, 2015, Ja Sam <ptrstp...@gmail.com> >> >> >> >> >>> > wrote: >> >> >> >> >>> >> >> >> >> >> >>> >> Of couse I made a mistake. I am using 2.1.2. Anyway night >> >> >> >> >>> >> build >> >> >> >> >>> >> is >> >> >> >> >>> >> available from >> >> >> >> >>> >> http://cassci.datastax.com/job/cassandra-2.1/ >> >> >> >> >>> >> >> >> >> >> >>> >> I read about cold_reads_to_omit It looks promising. >> Should I >> >> >> >> >>> >> set >> >> >> >> >>> >> also >> >> >> >> >>> >> compaction throughput? >> >> >> >> >>> >> >> >> >> >> >>> >> p.s. I am really sad that I didn't read this before: >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> On Monday, February 16, 2015, Carlos Rolo < >> r...@pythian.com> >> >> >> >> >>> >> wrote: >> >> >> >> >>> >>> >> >> >> >> >>> >>> Hi 100% in agreement with Roland, >> >> >> >> >>> >>> >> >> >> >> >>> >>> 2.1.x series is a pain! I would never recommend the >> current >> >> >> >> >>> >>> 2.1.x >> >> >> >> >>> >>> series >> >> >> >> >>> >>> for production. >> >> >> >> >>> >>> >> >> >> >> >>> >>> Clocks is a pain, and check your connectivity! Also check >> >> >> >> >>> >>> tpstats >> >> >> >> >>> >>> to >> >> >> >> >>> >>> see >> >> >> >> >>> >>> if your threadpools are being overrun. >> >> >> >> >>> >>> >> >> >> >> >>> >>> Regards, >> >> >> >> >>> >>> >> >> >> >> >>> >>> Carlos Juzarte Rolo >> >> >> >> >>> >>> Cassandra Consultant >> >> >> >> >>> >>> >> >> >> >> >>> >>> Pythian - Love your data >> >> >> >> >>> >>> >> >> >> >> >>> >>> rolo@pythian | Twitter: cjrolo | Linkedin: >> >> >> >> >>> >>> linkedin.com/in/carlosjuzarterolo >> >> >> >> >>> >>> Tel: 1649 >> >> >> >> >>> >>> www.pythian.com >> >> >> >> >>> >>> >> >> >> >> >>> >>> On Mon, Feb 16, 2015 at 8:12 PM, Roland Etzenhammer >> >> >> >> >>> >>> <r.etzenham...@t-online.de> wrote: >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> Hi, >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> 1) Actual Cassandra 2.1.3, it was upgraded from 2.1.0 >> >> >> >> >>> >>>> (suggested >> >> >> >> >>> >>>> by >> >> >> >> >>> >>>> Al >> >> >> >> >>> >>>> Tobey from DataStax) >> >> >> >> >>> >>>> 7) minimal reads (usually none, sometimes few) >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> those two points keep me repeating an anwser I got. >> First >> >> >> >> >>> >>>> where >> >> >> >> >>> >>>> did >> >> >> >> >>> >>>> you >> >> >> >> >>> >>>> get 2.1.3 from? Maybe I missed it, I will have a look. >> But >> >> >> >> >>> >>>> if >> >> >> >> >>> >>>> it >> >> >> >> >>> >>>> is >> >> >> >> >>> >>>> 2.1.2 >> >> >> >> >>> >>>> whis is the latest released version, that version has >> many >> >> >> >> >>> >>>> bugs - >> >> >> >> >>> >>>> most of >> >> >> >> >>> >>>> them I got kicked by while testing 2.1.2. I got many >> >> >> >> >>> >>>> problems >> >> >> >> >>> >>>> with >> >> >> >> >>> >>>> compactions not beeing triggred on column families not >> >> >> >> >>> >>>> beeing >> >> >> >> >>> >>>> read, >> >> >> >> >>> >>>> compactions and repairs not beeing completed. See >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> https://www.mail-archive.com/search?l=user@cassandra.apache.org&q=subject:%22Re%3A+Compaction+failing+to+trigger%22&o=newest&f=1 >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> >> https://www.mail-archive.com/user%40cassandra.apache.org/msg40768.html >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> Apart from that, how are those both datacenters >> connected? >> >> >> >> >>> >>>> Maybe >> >> >> >> >>> >>>> there >> >> >> >> >>> >>>> is a bottleneck. >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> Also do you have ntp up and running on all nodes to keep >> >> >> >> >>> >>>> all >> >> >> >> >>> >>>> clocks >> >> >> >> >>> >>>> in >> >> >> >> >>> >>>> thight sync? >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> Note: I'm no expert (yet) - just sharing my 2 cents. >> >> >> >> >>> >>>> >> >> >> >> >>> >>>> Cheers, >> >> >> >> >>> >>>> Roland >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> -- >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > >> > >
