[
https://issues.apache.org/jira/browse/CASSANDRA-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580498#comment-14580498
]
Antti Nissinen commented on CASSANDRA-9572:
-------------------------------------------
In the DataTieredCompactionStrategy class (method getNextBackgroundSSTables) is
the first call to the getFullyExpiredSSTables. It will return all such SSTables
that are fully expired, but do not overlap with any of the non-fully expired
SSTables. In the next phases the most interesting bucket is composed from
SSTables that are not yet fully expired and are not under any other compaction
thread. These two sets are combined and given as argument to create a new
CompactionTask.
In the CompactionTask (method runMayThrow) is a line where a variable
"actuallyCompact" is assigned by taking a difference between the list of
SSTables used to create the CompactionTask (contains SSTables that fully
expired and do not overlap with any of the non-fully expired SStables AND
SSTables from the most interesting bucket) and the set returned by method call
getFullyExpiredSSTables without any arguments. Since the
getFullyExpiredSSTables is called without arguments "compacting" variable
contains the set of fully-expired files and files in the most interesting
bucket. The set of overlapping files contain files including one of the fully
expired files that was previously overlapping with one of the non-fully expired
SSTables.
Now the second round finds out that the previously discarded fully expired file
overlaps with one of the fully expired files and it will discard one more fully
expired file even if that does not actually overlap with any of the non-fully
expired files. That means that one potentially droppable fully expired SSTable
is missed from deletion.
Now the files for the "actuallyCompact" variable has been selected and they
will be compacted together. All files in the original list will then be deleted
from the disk. The compaction puts together fully expired files and newer files
from the most interesting bucket. All the data from the fully expired file
should be discarded in the compaction and the minimum time stamp for the new
file should be the one belonging to the oldest file in the most interesting
bucket. However, the minimum timestamp for the new file is the minimum
timestamp belonging to the file discarded on the second round
getFullyExpiredSSTables (missed case in the previous paragraph). This happens
on the first compaction round. On later compaction rounds the minimum timestamp
does not follow that rule and I don't know where it takes the minimum timestamp
for the new file. This will lead to SStables that are suppose to have data from
a long time window and it will mess up later compactions because files
containing only new data will be selected to the compacted with older files.
Hopefully this clarifies the original report.
> DateTieredCompactionStrategy fails to combine SSTables correctly when TTL is
> used.
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-9572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Antti Nissinen
> Assignee: Marcus Eriksson
> Fix For: 2.1.5
>
> Attachments: cassandra_sstable_metadata_reader.py,
> cassandra_sstable_timespan_graph.py, compaction_stage_test01_jira.log,
> compaction_stage_test02_jira.log, datagen.py, explanation_jira.txt,
> motivation_jira.txt
>
>
> DateTieredCompaction works correctly when data is dumped for a certain time
> period in short SSTables in time manner and then compacted together. However,
> if TTL is applied to the data columns the DTCS fails to compact files
> correctly in timely manner. In our opinion the problem is caused by two
> issues:
> A) During the DateTieredCompaction process the getFullyExpiredSStables is
> called twice. First from the DateTieredCompactionStrategy class and second
> time from the CompactionTask class. On the first time the target is to find
> out fully expired SStables that are not overlapping with any non-fully
> expired SSTables. That works correctly. When the getFullyExpiredSSTables is
> called second time from CompactionTask class the selection of fully expired
> SSTables is modified compared to the first selection.
> B) The minimum timestamp of the new SSTables created by combining together
> fully expired SSTable and files from the most interesting bucket is not
> correct.
> These two issues together cause problems for the DTCS process when it
> combines together SSTables having overlap in time and TTL for the column.
> This is demonstrated by generating test data first without compactions and
> showing the timely distribution of files. When the compaction is enabled the
> DCTS combines files together, but the end result is not something to be
> expected. This is demonstrated in the file motivation_jira.txt
> Attachments contain following material:
> - Motivation_jira.txt: Practical examples how the DTCS behaves with TTL
> - Explanation_jira.txt: gives more details, explains test cases and
> demonstrates the problems in the compaction process
> - Logfile file for the compactions in the first test case
> (compaction_stage_test01_jira.log)
> - Logfile file for the compactions in the seconnd test case
> (compaction_stage_test02_jira.log)
> - source code zip file for version 2.1.5 with additional comment statements
> (src_2.1.5_with_debug.zip)
> - Python script to generate test data (datagen.py)
> - Python script to read metadata from SStables
> (cassandra_sstable_metadata_reader.py)
> - Python script to generate timeline representation of SSTables
> (cassandra_sstable_timespan_graph.py)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
