When you ran this test, is that the exact schema you used? I'm not seeing where you are setting gc_grace to 0 (although I could just be blind, it happens).
On Thu, Jan 17, 2013 at 5:01 PM, Bryan Talbot <btal...@aeriagames.com>wrote:
> I'm able to reproduce this behavior on my laptop using 1.1.5, 1.1.7,
> 1.1.8, a trivial schema, and a simple script that just inserts rows. If
> the TTL is small enough so that all LCS data fits in generation 0 then the
> rows seem to be removed with TTL expires as desired. However, if the
> insertion rate is high enough or the TTL long enough then the data keep
> accumulating for far longer than expected.
>
> Using 120 second TTL and a single threaded php insertion script my MBP
> with SSD retained almost all of the data. 120 seconds should accumulate
> 5-10 MB of data. I would expect that TTL rows to be removed eventually and
> for the cassandra load to level off at some reasonable value near 10 MB.
> After running for 2 hours and with a cassandra load of ~550 MB I stopped
> the test.
>
> The schema is
>
> create keyspace test
> with placement_strategy = 'SimpleStrategy'
> and strategy_options = {replication_factor : 1}
> and durable_writes = true;
>
> use test;
>
> create column family test
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'TimeUUIDType'
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
> and caching = 'NONE'
> and bloom_filter_fp_chance = 1.0
> and column_metadata = [
> {column_name : 'a',
> validation_class : LongType}];
>
>
> and the insert script is
>
> <?php
>
> require_once('phpcassa/1.0.a.5/autoload.php');
>
> use phpcassa\Connection\ConnectionPool;
> use phpcassa\ColumnFamily;
> use phpcassa\SystemManager;
> use phpcassa\UUID;
>
> // Connect to test keyspace and column family
> $sys = new SystemManager('127.0.0.1');
>
> // Start a connection pool, create our ColumnFamily instance
> $pool = new ConnectionPool('test', array('127.0.0.1'));
> $testCf = new ColumnFamily($pool, 'test');
>
> // Insert records
> while( 1 ) {
> $testCf->insert(UUID::uuid1(), array("a" => 1), null, 120);
> }
>
> // Close our connections
> $pool->close();
> $sys->close();
>
> ?>
>
>
> -Bryan
>
>
>
>
> On Thu, Jan 17, 2013 at 10:11 AM, Bryan Talbot <btal...@aeriagames.com>wrote:
>
>> We are using LCS and the particular row I've referenced has been involved
>> in several compactions after all columns have TTL expired. The most recent
>> one was again this morning and the row is still there -- TTL expired for
>> several days now with gc_grace=0 and several compactions later ...
>>
>>
>> $> ./bin/nodetool -h localhost getsstables metrics request_summary
>> 459fb460-5ace-11e2-9b92-11d67b6163b4
>>
>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-448955-Data.db
>>
>> $> ls -alF
>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-448955-Data.db
>> -rw-rw-r-- 1 sandra sandra 5246509 Jan 17 06:54
>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-448955-Data.db
>>
>>
>> $> ./bin/sstable2json
>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-448955-Data.db
>> -k $(echo -n 459fb460-5ace-11e2-9b92-11d67b6163b4 | hexdump -e '36/1 "%x"')
>> {
>> "34353966623436302d356163652d313165322d396239322d313164363762363136336234":
>> [["app_name","50f21d3d",1357785277207001,"d"],
>> ["client_ip","50f21d3d",1357785277207001,"d"],
>> ["client_req_id","50f21d3d",1357785277207001,"d"],
>> ["mysql_call_cnt","50f21d3d",1357785277207001,"d"],
>> ["mysql_duration_us","50f21d3d",1357785277207001,"d"],
>> ["mysql_failure_call_cnt","50f21d3d",1357785277207001,"d"],
>> ["mysql_success_call_cnt","50f21d3d",1357785277207001,"d"],
>> ["req_duration_us","50f21d3d",1357785277207001,"d"],
>> ["req_finish_time_us","50f21d3d",1357785277207001,"d"],
>> ["req_method","50f21d3d",1357785277207001,"d"],
>> ["req_service","50f21d3d",1357785277207001,"d"],
>> ["req_start_time_us","50f21d3d",1357785277207001,"d"],
>> ["success","50f21d3d",1357785277207001,"d"]]
>> }
>>
>>
>> My experience with TTL columns so far has been pretty similar to Viktor's
>> in that the only way to keep them row count under control is to force major
>> compactions. In real world use, STCS and LCS both leave TTL expired rows
>> around forever as far as I can tell. When testing with minimal data,
>> removal of TTL expired rows seem to work as expected but in this case there
>> seems to be some divergence from real life work and test samples.
>>
>> -Bryan
>>
>>
>>
>>
>> On Thu, Jan 17, 2013 at 1:47 AM, Viktor Jevdokimov <
>> viktor.jevdoki...@adform.com> wrote:
>>
>>> @Bryan,****
>>>
>>> ** **
>>>
>>> To keep data size as low as possible with TTL columns we still use STCS
>>> and nightly major compactions.****
>>>
>>> ** **
>>>
>>> Experience with LCS was not successful in our case, data size keeps too
>>> high along with amount of compactions.****
>>>
>>> ** **
>>>
>>> IMO, before 1.2, LCS was good for CFs without TTL or high delete rate. I
>>> have not tested 1.2 LCS behavior, we’re still on 1.0.x****
>>>
>>> ** **
>>>
>>> ** **
>>> Best regards / Pagarbiai
>>> *Viktor Jevdokimov*
>>> Senior Developer
>>>
>>> Email: viktor.jevdoki...@adform.com
>>> Phone: +370 5 212 3063, Fax +370 5 261 0453
>>> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>>> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
>>> Take a ride with Adform's Rich Media
>>> Suite<http://vimeo.com/adform/richmedia>
>>> [image: Adform News] <http://www.adform.com>
>>> [image: Adform awarded the Best Employer 2012]
>>> <http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>> *From:* aaron morton [mailto:aa...@thelastpickle.com]
>>> *Sent:* Thursday, January 17, 2013 06:24
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: LCS not removing rows with all TTL expired columns****
>>>
>>> ** **
>>>
>>> Minor compaction (with Size Tiered) will only purge tombstones if all
>>> fragments of a row are contained in the SSTables being compacted. So if you
>>> have a long lived row, that is present in many size tiers, the columns will
>>> not be purged. ****
>>>
>>> ** **
>>>
>>> (thus compacted compacted) 3 days after all columns for that row had
>>> expired****
>>>
>>> Tombstones have to get on disk, even if you set the gc_grace_seconds to
>>> 0. If not they do not get a chance to delete previous versions of the
>>> column which already exist on disk. So when the compaction ran your
>>> ExpiringColumn was turned into a DeletedColumn and placed on disk. ****
>>>
>>> ** **
>>>
>>> I would expect the next round of compaction to remove these columns. ***
>>> *
>>>
>>> ** **
>>>
>>> There is a new feature in 1.2 that may help you here. It will do a
>>> special compaction of individual sstables when they have a certain
>>> proportion of dead columns
>>> https://issues.apache.org/jira/browse/CASSANDRA-3442 ****
>>>
>>> ** **
>>>
>>> Also interested to know if LCS helps. ****
>>>
>>> ** **
>>>
>>> Cheers****
>>>
>>> ****
>>>
>>> ** **
>>>
>>> -----------------****
>>>
>>> Aaron Morton****
>>>
>>> Freelance Cassandra Developer****
>>>
>>> New Zealand****
>>>
>>> ** **
>>>
>>> @aaronmorton****
>>>
>>> http://www.thelastpickle.com****
>>>
>>> ** **
>>>
>>> On 17/01/2013, at 2:55 PM, Bryan Talbot <btal...@aeriagames.com> wrote:*
>>> ***
>>>
>>>
>>>
>>> ****
>>>
>>> According to the timestamps (see original post) the SSTable was written
>>> (thus compacted compacted) 3 days after all columns for that row had
>>> expired and 6 days after the row was created; yet all columns are still
>>> showing up in the SSTable. Note that the column shows now rows when a
>>> "get" for that key is run so that's working correctly, but the data is
>>> lugged around far longer than it should be -- maybe forever.****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> -Bryan****
>>>
>>> ** **
>>>
>>> On Wed, Jan 16, 2013 at 5:44 PM, Andrey Ilinykh <ailin...@gmail.com>
>>> wrote:****
>>>
>>> To get column removed you have to meet two requirements ****
>>>
>>> 1. column should be expired****
>>>
>>> 2. after that CF gets compacted****
>>>
>>> ** **
>>>
>>> I guess your expired columns are propagated to high tier CF, which gets
>>> compacted rarely.****
>>>
>>> So, you have to wait when high tier CF gets compacted. ****
>>>
>>> ** **
>>>
>>> Andrey****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> On Wed, Jan 16, 2013 at 11:39 AM, Bryan Talbot <btal...@aeriagames.com>
>>> wrote:****
>>>
>>> On cassandra 1.1.5 with a write heavy workload, we're having problems
>>> getting rows to be compacted away (removed) even though all columns have
>>> expired TTL. We've tried size tiered and now leveled and are seeing the
>>> same symptom: the data stays around essentially forever. ****
>>>
>>> ** **
>>>
>>> Currently we write all columns with a TTL of 72 hours (259200 seconds)
>>> and expect to add 10 GB of data to this CF per day per node. Each node
>>> currently has 73 GB for the affected CF and shows no indications that old
>>> rows will be removed on their own.****
>>>
>>> ** **
>>>
>>> Why aren't rows being removed? Below is some data from a sample row
>>> which should have been removed several days ago but is still around even
>>> though it has been involved in numerous compactions since being expired.
>>> ****
>>>
>>> ** **
>>>
>>> $> ./bin/nodetool -h localhost getsstables metrics request_summary
>>> 459fb460-5ace-11e2-9b92-11d67b6163b4****
>>>
>>>
>>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-386179-Data.db
>>> ****
>>>
>>> ** **
>>>
>>> $> ls -alF
>>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-386179-Data.db
>>> ****
>>>
>>> -rw-rw-r-- 1 sandra sandra 5252320 Jan 16 08:42
>>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-386179-Data.db
>>> ****
>>>
>>> ** **
>>>
>>> $> ./bin/sstable2json
>>> /virtual/cassandra/data/data/metrics/request_summary/metrics-request_summary-he-386179-Data.db
>>> -k $(echo -n 459fb460-5ace-11e2-9b92-11d67b6163b4 | hexdump -e '36/1 "%x"')
>>> ****
>>>
>>> {****
>>>
>>> "34353966623436302d356163652d313165322d396239322d313164363762363136336234":
>>> [["app_name","50f21d3d",1357785277207001,"d"],
>>> ["client_ip","50f21d3d",1357785277207001,"d"],
>>> ["client_req_id","50f21d3d",1357785277207001,"d"],
>>> ["mysql_call_cnt","50f21d3d",1357785277207001,"d"],
>>> ["mysql_duration_us","50f21d3d",1357785277207001,"d"],
>>> ["mysql_failure_call_cnt","50f21d3d",1357785277207001,"d"],
>>> ["mysql_success_call_cnt","50f21d3d",1357785277207001,"d"],
>>> ["req_duration_us","50f21d3d",1357785277207001,"d"],
>>> ["req_finish_time_us","50f21d3d",1357785277207001,"d"],
>>> ["req_method","50f21d3d",1357785277207001,"d"],
>>> ["req_service","50f21d3d",1357785277207001,"d"],
>>> ["req_start_time_us","50f21d3d",1357785277207001,"d"],
>>> ["success","50f21d3d",1357785277207001,"d"]]****
>>>
>>> }****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> Decoding the column timestamps to shows that the columns were written at
>>> "Thu, 10 Jan 2013 02:34:37 GMT" and that their TTL expired at "Sun, 13 Jan
>>> 2013 02:34:37 GMT". The date of the SSTable shows that it was generated on
>>> Jan 16 which is 3 days after all columns have TTL-ed out.****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> The schema shows that gc_grace is set to 0 since this data is
>>> write-once, read-seldom and is never updated or deleted.****
>>>
>>> ** **
>>>
>>> create column family request_summary****
>>>
>>> with column_type = 'Standard'****
>>>
>>> and comparator = 'UTF8Type'****
>>>
>>> and default_validation_class = 'UTF8Type'****
>>>
>>> and key_validation_class = 'UTF8Type'****
>>>
>>> and read_repair_chance = 0.1****
>>>
>>> and dclocal_read_repair_chance = 0.0****
>>>
>>> and gc_grace = 0****
>>>
>>> and min_compaction_threshold = 4****
>>>
>>> and max_compaction_threshold = 32****
>>>
>>> and replicate_on_write = true****
>>>
>>> and compaction_strategy =
>>> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'****
>>>
>>> and caching = 'NONE'****
>>>
>>> and bloom_filter_fp_chance = 1.0****
>>>
>>> and compression_options = {'chunk_length_kb' : '64',
>>> 'sstable_compression' :
>>> 'org.apache.cassandra.io.compress.SnappyCompressor'};****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> Thanks in advance for help in understanding why rows such as this are
>>> not removed!****
>>>
>>> ** **
>>>
>>> -Bryan****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> ** **
>>>
>>
>>
>>
>
--
Derek Williams
<<signature-logo29.png>>
<<signature-best-employer-logo4823.png>>
