It sounds like the commitlog has been replayed however I have really no idea whether this could have happened. Anyone?
2012/4/13 Alain RODRIGUEZ <arodr...@gmail.com> > The commitlog_total_space_in_mb was not set, I set it to avoid having the > same problem in the future. > > I am aware of the over-counting problem introduced by the counters. The > point is that I use them to make statistics per hours. I can understand > having some wrong counts in the column corresponding to the crash time, but > how to explain that all my counts since the start (months ago) have become > wrong after the crash ? > > After the crash I tried to repair my entire keyspace from one of the 2 > nodes and this made my server crash again, no idea why. Can this failed > repair be at the origin of the corrupted data ? > > I'm still replaying all my counts of the past months and I'm afraid this > kind of bug could happen again... > > I was using cassandra for months without any issue. > > Alain > > 2012/4/11 aaron morton <aa...@thelastpickle.com> > >> However after recovering from this issue (freeing some space and fixing >>> the value of "commitlog_total_space_in_mb" in cassandra.yaml) >>> >> Did the commit log grow larger than commitlog_total_space_in_mb ? >> >> I realized that all statistics were all destroyed. I have bad values on >>> every single counter since I start using them (september) ! >>> >> Counter operations are not idempotent. If you client retries a counter >> operation it may result in the increment been applied twice. Could this >> have been your issue ? >> >> Cheers >> >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 11/04/2012, at 2:35 AM, Alain RODRIGUEZ wrote: >> >> By the way, I am using Cassandra 1.0.7, CL = ONE (R/W), RF = 2, 2 EC2 >> c1.medium nodes cluster >> >> Alain >> >> 2012/4/10 Alain RODRIGUEZ <arodr...@gmail.com> >> >>> Hi, I'm experimenting a strange and very annoying phenomena. >>> >>> I had a problem with the commit log size which grew too much and full >>> one of the hard disks in all my nodes almost at the same time (2 nodes >>> only, RF=2, so the 2 nodes are behaving exactly in the same way) >>> >>> My data are mounted in an other partition that was not full. However >>> after recovering from this issue (freeing some space and fixing the value >>> of "commitlog_total_space_in_mb" in cassandra.yaml) I realized that all >>> statistics were all destroyed. I have bad values on every single counter >>> since I start using them (september) ! >>> >>> Does anyone experimented something similar or have any clue on this ? >>> >>> Do you need more information ? >>> >>> Alain >>> >> >> >> > -- With kind regards, Robin Verlangen www.robinverlangen.nl
