Yes, continued deletions of the same columns/rows will prevent removing them from final sstable upon compaction due to new timestamp. You're getting sliding tombstone gc grace period in that case.
During compaction of selected sstables Cassandra checks the whole Column Family for the latest timestamp of the column/row, including other sstables and memtable. You need to review your application logic. Best regards/ Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063. Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania [Adform news]<http://www.adform.com/> [Visit us!] Follow: [twitter]<http://twitter.com/#!/adforminsider> Visit our blog<http://www.adform.com/site/blog> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Ross Black [mailto:ross.w.bl...@gmail.com] Sent: Friday, March 23, 2012 07:16 To: user@cassandra.apache.org Subject: Re: tombstones problem with 1.0.8 Hi Victor, Thanks for your response. Is there a possibility that continual deletions during compact could be blocking removal of the tombstones? The full manual compact takes about 4 hours per node for our data, so there is a large number of deletes occurring during that time. This is the description from cassandra-cli Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: weekly Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 3600 Compaction min/max thresholds: 3/8 Read repair chance: 1.0 Replicate on write: true Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Ross On 23 March 2012 02:55, Viktor Jevdokimov <viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com>> wrote: Just tested 1.0.8 before upgrading from 1.0.7: tombstones created by TTL or by delete operation are perfectly deleted after either compaction or cleanup. Have no idea about any other settings than gc_grace_seconds, check you schema from cassandra-cli. Best regards/ Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com> Phone: +370 5 212 3063<tel:%2B370%205%20212%203063>. Fax: +370 5 261 0453<tel:%2B370%205%20261%200453> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania [Adform news]<http://www.adform.com/> [Visit us!] Follow: [twitter]<http://twitter.com/#%21/adforminsider> Visit our blog<http://www.adform.com/site/blog> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Ross Black [mailto:ross.w.bl...@gmail.com<mailto:ross.w.bl...@gmail.com>] Sent: Thursday, March 22, 2012 03:38 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: tombstones problem with 1.0.8 Hi, We recently moved from 0.8.2 to 1.0.8 and the behaviour seems to have changed so that tombstones are now not being deleted. Our application continually adds and removes columns from Cassandra. We have set a short gc_grace time (3600) since our application would automatically delete zombies if they appear. Under 0.8.2, the tombstones remained at a relatively constant number. Under 1.0.8, the tombstones have been continually increasing so that they exceed the size of our real data (at this stage we have over 100G of tombstones). Even after running a full compact the new compacted SSTable contains a massive number of tombstones, many that are several weeks old. Have I missed some new configuration option to allow deletion of tombstones? I also noticed that one of the changes between 0.8.2 and 1.0.8 was https://issues.apache.org/jira/browse/CASSANDRA-2786 which changed code to "avoid dropping tombstones when they might still be needed to shadow data in another sstable". Could this be having an impact since we continually add and remove columns even while a major compact is executing? Thanks, Ross
<<inline: image001.png>>
<<inline: image002.png>>
<<inline: image003.png>>
<<inline: signature-logo29.png>>
<<inline: dm-exco4823.png>>
<<inline: tweet18be.png>>
