Re: .TombstoneOverwhelmingException
Thanks Robert. I guess the issue is resolved now after doing the mentioned step. Can you tell me how to verify the current tombstone counts in order to do a regular check? Thanks Aravindan Thangavelu -Robert Coli rc...@eventbrite.com wrote: - To: user@cassandra.apache.org user@cassandra.apache.org From: Robert Coli rc...@eventbrite.com Date: 08/23/2014 03:30AM Subject: Re: .TombstoneOverwhelmingException On Fri, Aug 22, 2014 at 2:47 PM, Aravindan T aravinda...@tcs.com wrote: 1. Only insertion of records are done since the cassandra started. But, when looked into the system.log messages, i see this following error. ... Now, could you please tell how the tombstone got created when there are no deletes executed You stored a bunch of hints via hinted handoff. Then they got tombstoned. 2. How and where to monitor the current tombstone counts ? 3. done compaction, nodetool repair, increased the threshold count but still this tombstone is not getting resolved. How to resolve this issue? Either major compact the hints columnfamily (might not be done by nodetool compact because it's system keyspace) or : 1) nodetool drain 2) stop node 3) delete all sstables in system/hintscolumnfamily 4) start node 5) repair You should also figure out why you're storing so many hints, and stop doing so. =Rob =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Question about incremental backup
Hi, I am setting backup and restoration tooling for a Cassandra cluster and have a specific question regarding incremental backup. Let’s say I’m running incremental backups and take a snapshot. At the exact(ish) same time as my snapshot it taken another incremental *.db file is hard linked into the backups directory. My question is, how do I know which snapshot my incremental file belongs to? If it was made half a second _before_ my snapshot, it belongs to the previous snapshot. If it was made half a second after my snapshot, I guess it belongs to my latest snapshot. Or, is this not an issue since I can always include the uncertain incremental file when restoring (since timestamps are always included with every column value)? Thanks, Jens ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
Re: Question about incremental backup
On Sat, Aug 23, 2014 at 8:06 AM, Jens Rantil jens.ran...@tink.se wrote: I am setting backup and restoration tooling for a Cassandra cluster and have a specific question regarding incremental backup. Let’s say I’m running incremental backups and take a snapshot. At the exact(ish) same time as my snapshot it taken another incremental *.db file is hard linked into the backups directory. My question is, how do I know which snapshot my incremental file belongs to? Tablesnap avoids this race by snapshotting files directly from the data directory, and backing it up with a meta-information file that contains a list of all SSTables in the data directory at the time it notices a new one. You can probably do something similar with the incremental snapshot system, but you might want to consider if you need to. :D https://github.com/JeremyGrosser/tablesnap =Rob
Re: Question about incremental backup
keep in mind backing up SSTables is not enough. To have truly incremental backup you have to store commit logs also. Thank you, Andrey On Sat, Aug 23, 2014 at 11:30 AM, Robert Coli rc...@eventbrite.com wrote: On Sat, Aug 23, 2014 at 8:06 AM, Jens Rantil jens.ran...@tink.se wrote: I am setting backup and restoration tooling for a Cassandra cluster and have a specific question regarding incremental backup. Let’s say I’m running incremental backups and take a snapshot. At the exact(ish) same time as my snapshot it taken another incremental *.db file is hard linked into the backups directory. My question is, how do I know which snapshot my incremental file belongs to? Tablesnap avoids this race by snapshotting files directly from the data directory, and backing it up with a meta-information file that contains a list of all SSTables in the data directory at the time it notices a new one. You can probably do something similar with the incremental snapshot system, but you might want to consider if you need to. :D https://github.com/JeremyGrosser/tablesnap =Rob
How to free disk space after decreasing replication factor?
Hi, I am using leveled compaction and I changed the replication factor from 3 to 2, but after a few days the disk space weren't freed. I tried to trigger the compaction or clean up, but looks like it didn't take any effect.
