This is an automated email from the ASF dual-hosted git repository.
smiklosovic pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 58a3b12508 Remove DateTieredCompactionStrategy
58a3b12508 is described below
commit 58a3b12508f97e44d3812f6c97e5a969dc6b5a1b
Author: Stefan Miklosovic <smikloso...@apache.org>
AuthorDate: Mon Nov 14 15:07:03 2022 +0100
Remove DateTieredCompactionStrategy
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski for
CASSANDRA-18043
---
CHANGES.txt | 1 +
NEWS.txt | 4 +-
doc/cql3/CQL.textile | 5 +-
.../cassandra/pages/cql/cql_singlefile.adoc | 14 +-
doc/modules/cassandra/pages/cql/ddl.adoc | 2 -
.../pages/operating/compaction/index.adoc | 12 +-
pylib/cqlshlib/cql3handling.py | 10 -
pylib/cqlshlib/cqlhandling.py | 1 -
pylib/cqlshlib/test/test_cqlsh_completion.py | 9 -
.../compaction/DateTieredCompactionStrategy.java | 519 ---------------------
.../DateTieredCompactionStrategyOptions.java | 156 -------
.../distributed/test/SSTableIdGenerationTest.java | 2 -
.../compaction/AbstractCompactionStrategyTest.java | 16 -
.../db/compaction/CompactionsCQLTest.java | 32 +-
.../DateTieredCompactionStrategyTest.java | 389 ---------------
15 files changed, 19 insertions(+), 1153 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index ae7420fbbe..344d60b2ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Remove DateTieredCompactionStrategy (CASSANDRA-18043)
* Add system_views.max_sstable_size and system_views.max_sstable_duration
tables (CASSANDRA-18333)
* Extend implicit allow-filtering for virtual tables to clustering columns
(CASSANDRA-18331)
* Upgrade maven-shade-plugin to 3.4.1 to fix shaded dtest JAR build
(CASSANDRA-18136)
diff --git a/NEWS.txt b/NEWS.txt
index 4dc0fe8cc7..f23a518d3d 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -135,7 +135,7 @@ New features
Upgrading
---------
- Ephemeral marker files for snapshots done by repairs are not created
anymore,
- there is a dedicated flag in snapshot manifest instead. On upgrade of a
node to version 4.2, on node's start, in case there
+ there is a dedicated flag in snapshot manifest instead. On upgrade of a
node to this version, on node's start, in case there
are such ephemeral snapshots on disk, they will be deleted (same
behaviour as before) and any new ephemeral snapshots
will stop to create ephemeral marker files as flag in a snapshot
manifest was introduced instead.
- There were new table properties introduced called 'allow_auto_snapshot'
and 'incremental_backups' (see section 'New features'). Hence, upgraded
@@ -146,6 +146,8 @@ Upgrading
4.1.0 or 4.0.0-4.0.7. If that is your case, please use protocol v4 or
higher in your driver. See CASSANDRA-17507
for further details.
- Added API for alternative sstable implementations. For details, see
src/java/org/apache/cassandra/io/sstable/SSTable_API.md
+ - DateTieredCompactionStrategy was removed. Please change the compaction
strategy for the tables using this strategy
+ to TimeWindowCompactionStrategy before upgrading to this version.
Deprecation
-----------
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index fd9cd56963..bd16a43ed4 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -362,7 +362,7 @@ Table creation supports the following other @<property>@:
h4(#compactionOptions). Compaction options
-The @compaction@ property must at least define the @'class'@ sub-option, that
defines the compaction strategy class to use. The default supported class are
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@,
@'DateTieredCompactionStrategy'@ and @'TimeWindowCompactionStrategy'@. Custom
strategy can be provided by specifying the full class name as a "string
constant":#constants. The rest of the sub-options depends on the chosen class.
The sub-options supported by the default c [...]
+The @compaction@ property must at least define the @'class'@ sub-option, that
defines the compaction strategy class to use. The default supported class are
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@ and
@'TimeWindowCompactionStrategy'@. Custom strategy can be provided by specifying
the full class name as a "string constant":#constants. The rest of the
sub-options depends on the chosen class. The sub-options supported by the
default classes are:
|_. option |_. supported compaction strategy |_.
default |_. description |
| @enabled@ | _all_ |
true | A boolean denoting whether compaction should be enabled or not.|
@@ -375,9 +375,6 @@ The @compaction@ property must at least define the
@'class'@ sub-option, that de
| @bucket_low@ | SizeTieredCompactionStrategy |
0.5 | Size tiered consider sstables to be within the same bucket if
their size is within [average_size * @bucket_low@, average_size * @bucket_high@
] (i.e the default groups sstable whose sizes diverges by at most 50%)|
| @bucket_high@ | SizeTieredCompactionStrategy |
1.5 | Size tiered consider sstables to be within the same bucket if
their size is within [average_size * @bucket_low@, average_size * @bucket_high@
] (i.e the default groups sstable whose sizes diverges by at most 50%).|
| @sstable_size_in_mb@ | LeveledCompactionStrategy |
5MB | The target size (in MB) for sstables in the leveled strategy.
Note that while sstable sizes should stay less or equal to
@sstable_size_in_mb@, it is possible to exceptionally have a larger sstable as
during compaction, data for a given partition key are never split into 2
sstables|
-| @timestamp_resolution@ | DateTieredCompactionStrategy |
MICROSECONDS | The timestamp resolution used when inserting data, could be
MILLISECONDS, MICROSECONDS etc (should be understandable by Java TimeUnit) -
don't change this unless you do mutations with USING TIMESTAMP
<non_microsecond_timestamps> (or equivalent directly in the client)|
-| @base_time_seconds@ | DateTieredCompactionStrategy |
60 | The base size of the time windows. |
-| @max_sstable_age_days@ | DateTieredCompactionStrategy |
365 | SSTables only containing data that is older than this will never
be compacted. |
| @timestamp_resolution@ | TimeWindowCompactionStrategy |
MICROSECONDS | The timestamp resolution used when inserting data, could be
MILLISECONDS, MICROSECONDS etc (should be understandable by Java TimeUnit) -
don't change this unless you do mutations with USING TIMESTAMP
<non_microsecond_timestamps> (or equivalent directly in the client)|
| @compaction_window_unit@ | TimeWindowCompactionStrategy |
DAYS | The Java TimeUnit used for the window size, set in conjunction
with @compaction_window_size@. Must be one of DAYS, HOURS, MINUTES |
| @compaction_window_size@ | TimeWindowCompactionStrategy | 1
| The number of @compaction_window_unit@ units that make up a time
window. |
diff --git a/doc/modules/cassandra/pages/cql/cql_singlefile.adoc
b/doc/modules/cassandra/pages/cql/cql_singlefile.adoc
index 3e8c47f209..73bb4626fa 100644
--- a/doc/modules/cassandra/pages/cql/cql_singlefile.adoc
+++ b/doc/modules/cassandra/pages/cql/cql_singlefile.adoc
@@ -562,7 +562,7 @@ link:#cachingOptions[below].
The `compaction` property must at least define the `'class'` sub-option,
that defines the compaction strategy class to use. The default supported
class are `'SizeTieredCompactionStrategy'`,
-`'LeveledCompactionStrategy'`, `'DateTieredCompactionStrategy'` and
+`'LeveledCompactionStrategy'` and
`'TimeWindowCompactionStrategy'`. Custom strategy can be provided by
specifying the full class name as a link:#constants[string constant].
The rest of the sub-options depends on the chosen class. The sub-options
@@ -618,18 +618,6 @@ sizes should stay less or equal to `sstable_size_in_mb`,
it is possible
to exceptionally have a larger sstable as during compaction, data for a
given partition key are never split into 2 sstables
-|`timestamp_resolution` |DateTieredCompactionStrategy |MICROSECONDS |The
-timestamp resolution used when inserting data, could be MILLISECONDS,
-MICROSECONDS etc (should be understandable by Java TimeUnit) - don’t
-change this unless you do mutations with USING TIMESTAMP (or equivalent
-directly in the client)
-
-|`base_time_seconds` |DateTieredCompactionStrategy |60 |The base size of
-the time windows.
-
-|`max_sstable_age_days` |DateTieredCompactionStrategy |365 |SSTables
-only containing data that is older than this will never be compacted.
-
|`timestamp_resolution` |TimeWindowCompactionStrategy |MICROSECONDS |The
timestamp resolution used when inserting data, could be MILLISECONDS,
MICROSECONDS etc (should be understandable by Java TimeUnit) - don’t
diff --git a/doc/modules/cassandra/pages/cql/ddl.adoc
b/doc/modules/cassandra/pages/cql/ddl.adoc
index 09e888808e..cb905f581b 100644
--- a/doc/modules/cassandra/pages/cql/ddl.adoc
+++ b/doc/modules/cassandra/pages/cql/ddl.adoc
@@ -605,8 +605,6 @@ The supported classes are:
* `'LeveledCompactionStrategy'`, xref:operating/compaction/lcs.adoc#lcs[LCS]
* `'TimeWindowCompactionStrategy'`,
xref:operating/compaction/twcs.adoc#twcs[TWCS]
-The `'DateTieredCompactionStrategy'` is also supported but deprecated;
-`'TimeWindowCompactionStrategy'` should be used.
If a custom strategies is required, specify the full class name as a
xref:cql/definitions.adoc#constants[string constant].
All default strategies support a number of
xref:operating/compaction/index.adoc#compaction-options[common options], as
well as options specific to the strategy chosen. See the section corresponding
to your strategy for details:
xref:operating/compaction/stcs.adoc#stcs_options[STCS],
xref:operating/compaction/lcs.adoc#lcs_options[LCS],
xref:operating/compaction/twcs.adoc#twcs_options[TWCS].
diff --git a/doc/modules/cassandra/pages/operating/compaction/index.adoc
b/doc/modules/cassandra/pages/operating/compaction/index.adoc
index be2065690d..1795f03726 100644
--- a/doc/modules/cassandra/pages/operating/compaction/index.adoc
+++ b/doc/modules/cassandra/pages/operating/compaction/index.adoc
@@ -188,16 +188,14 @@ compacted at once.
== Fully expired SSTables
-If an sstable contains only tombstones and it is guaranteed that that
-sstable is not shadowing data in any other sstable compaction can drop
-that sstable. If you see SSTables with only tombstones (note that TTL:ed
-data is considered tombstones once the time to live has expired) but it
+If an SSTable contains only tombstones and it is guaranteed that
+SSTable is not shadowing data in any other SSTable, then the compaction can
drop
+that SSTable. If you see SSTables with only tombstones (note that TTL-ed
+data is considered tombstones once the time-to-live has expired), but it
is not being dropped by compaction, it is likely that other SSTables
contain older data. There is a tool called `sstableexpiredblockers` that
will list which SSTables are droppable and which are blocking them from
-being dropped. This is especially useful for time series compaction with
-`TimeWindowCompactionStrategy` (and the deprecated
-`DateTieredCompactionStrategy`). With `TimeWindowCompactionStrategy` it
+being dropped. With `TimeWindowCompactionStrategy` it
is possible to remove the guarantee (not check for shadowing data) by
enabling `unsafe_aggressive_sstable_expiration`.
diff --git a/pylib/cqlshlib/cql3handling.py b/pylib/cqlshlib/cql3handling.py
index ddf8723863..1bc564849e 100644
--- a/pylib/cqlshlib/cql3handling.py
+++ b/pylib/cqlshlib/cql3handling.py
@@ -97,14 +97,6 @@ class Cql3ParsingRuleSet(CqlParsingRuleSet):
'fanout_size'
)
- date_tiered_compaction_strategy_options = (
- 'base_time_seconds',
- 'max_sstable_age_days',
- 'min_threshold',
- 'max_window_size_seconds',
- 'timestamp_resolution'
- )
-
time_window_compaction_strategy_options = (
'compaction_window_unit',
'compaction_window_size',
@@ -554,8 +546,6 @@ def cf_prop_val_mapkey_completer(ctxt, cass):
opts =
opts.union(set(CqlRuleSet.size_tiered_compaction_strategy_options))
elif csc == 'LeveledCompactionStrategy':
opts =
opts.union(set(CqlRuleSet.leveled_compaction_strategy_options))
- elif csc == 'DateTieredCompactionStrategy':
- opts =
opts.union(set(CqlRuleSet.date_tiered_compaction_strategy_options))
elif csc == 'TimeWindowCompactionStrategy':
opts =
opts.union(set(CqlRuleSet.time_window_compaction_strategy_options))
diff --git a/pylib/cqlshlib/cqlhandling.py b/pylib/cqlshlib/cqlhandling.py
index ca12a251c1..5ea0391a22 100644
--- a/pylib/cqlshlib/cqlhandling.py
+++ b/pylib/cqlshlib/cqlhandling.py
@@ -49,7 +49,6 @@ class CqlParsingRuleSet(pylexotron.ParsingRuleSet):
available_compaction_classes = (
'LeveledCompactionStrategy',
'SizeTieredCompactionStrategy',
- 'DateTieredCompactionStrategy',
'TimeWindowCompactionStrategy'
)
diff --git a/pylib/cqlshlib/test/test_cqlsh_completion.py
b/pylib/cqlshlib/test/test_cqlsh_completion.py
index 416e15ee69..04929a92e9 100644
--- a/pylib/cqlshlib/test/test_cqlsh_completion.py
+++ b/pylib/cqlshlib/test/test_cqlsh_completion.py
@@ -669,7 +669,6 @@ class TestCqlshCompletion(CqlshCompletionCase):
+ "{'class': '",
choices=['SizeTieredCompactionStrategy',
'LeveledCompactionStrategy',
- 'DateTieredCompactionStrategy',
'TimeWindowCompactionStrategy'])
self.trycompletions(prefix + " new_table (col_a int PRIMARY KEY) WITH
compaction = "
+ "{'class': 'S",
@@ -708,14 +707,6 @@ class TestCqlshCompletion(CqlshCompletionCase):
'CLUSTERING',
'COMPACT', 'caching', 'comment',
'min_index_interval',
'speculative_retry', 'additional_write_policy', 'cdc', 'read_repair'])
- self.trycompletions(prefix + " new_table (col_a int PRIMARY KEY) WITH
compaction = "
- + "{'class': 'DateTieredCompactionStrategy', '",
- choices=['base_time_seconds',
'max_sstable_age_days',
- 'timestamp_resolution', 'min_threshold',
'class', 'max_threshold',
- 'tombstone_compaction_interval',
'tombstone_threshold',
- 'enabled',
'unchecked_tombstone_compaction',
- 'max_window_size_seconds',
- 'only_purge_repaired_tombstones',
'provide_overlapping_tombstones'])
self.trycompletions(prefix + " new_table (col_a int PRIMARY KEY) WITH
compaction = "
+ "{'class': 'TimeWindowCompactionStrategy', '",
choices=['compaction_window_unit',
'compaction_window_size',
diff --git
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
deleted file mode 100644
index 1e18bf9000..0000000000
---
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
+++ /dev/null
@@ -1,519 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.db.compaction;
-
-import java.util.*;
-import java.util.concurrent.TimeUnit;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Predicate;
-import com.google.common.collect.*;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.node.JsonNodeFactory;
-import com.fasterxml.jackson.databind.node.ObjectNode;
-import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
-import org.apache.cassandra.db.lifecycle.SSTableSet;
-import org.apache.cassandra.exceptions.ConfigurationException;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
-import org.apache.cassandra.schema.CompactionParams;
-import org.apache.cassandra.utils.Pair;
-
-import static com.google.common.collect.Iterables.filter;
-import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
-
-/**
- * @deprecated in favour of {@link TimeWindowCompactionStrategy}
- */
-@Deprecated
-public class DateTieredCompactionStrategy extends AbstractCompactionStrategy
-{
- private static final Logger logger =
LoggerFactory.getLogger(DateTieredCompactionStrategy.class);
-
- private final DateTieredCompactionStrategyOptions options;
- protected volatile int estimatedRemainingTasks;
- private final Set<SSTableReader> sstables = new HashSet<>();
- private long lastExpiredCheck;
- private final SizeTieredCompactionStrategyOptions stcsOptions;
-
- public DateTieredCompactionStrategy(ColumnFamilyStore cfs, Map<String,
String> options)
- {
- super(cfs, options);
- this.estimatedRemainingTasks = 0;
- this.options = new DateTieredCompactionStrategyOptions(options);
- if
(!options.containsKey(AbstractCompactionStrategy.TOMBSTONE_COMPACTION_INTERVAL_OPTION)
&& !options.containsKey(AbstractCompactionStrategy.TOMBSTONE_THRESHOLD_OPTION))
- {
- disableTombstoneCompactions = true;
- logger.trace("Disabling tombstone compactions for DTCS");
- }
- else
- logger.trace("Enabling tombstone compactions for DTCS");
-
- this.stcsOptions = new SizeTieredCompactionStrategyOptions(options);
- }
-
- @Override
- @SuppressWarnings("resource")
- public AbstractCompactionTask getNextBackgroundTask(int gcBefore)
- {
- List<SSTableReader> previousCandidate = null;
- while (true)
- {
- List<SSTableReader> latestBucket =
getNextBackgroundSSTables(gcBefore);
-
- if (latestBucket.isEmpty())
- return null;
-
- // Already tried acquiring references without success. It means
there is a race with
- // the tracker but candidate SSTables were not yet replaced in the
compaction strategy manager
- if (latestBucket.equals(previousCandidate))
- {
- logger.warn("Could not acquire references for compacting
SSTables {} which is not a problem per se," +
- "unless it happens frequently, in which case it
must be reported. Will retry later.",
- latestBucket);
- return null;
- }
-
- LifecycleTransaction modifier =
cfs.getTracker().tryModify(latestBucket, OperationType.COMPACTION);
- if (modifier != null)
- return new CompactionTask(cfs, modifier, gcBefore);
- previousCandidate = latestBucket;
- }
- }
-
- /**
- *
- * @param gcBefore
- * @return
- */
- private synchronized List<SSTableReader> getNextBackgroundSSTables(final
int gcBefore)
- {
- Set<SSTableReader> uncompacting;
- synchronized (sstables)
- {
- if (sstables.isEmpty())
- return Collections.emptyList();
-
- uncompacting =
ImmutableSet.copyOf(filter(cfs.getUncompactingSSTables(), sstables::contains));
- }
-
- Set<SSTableReader> expired = Collections.emptySet();
- // we only check for expired sstables every 10 minutes (by default)
due to it being an expensive operation
- if (currentTimeMillis() - lastExpiredCheck >
options.expiredSSTableCheckFrequency)
- {
- // Find fully expired SSTables. Those will be included no matter
what.
- expired = CompactionController.getFullyExpiredSSTables(cfs,
uncompacting, cfs.getOverlappingLiveSSTables(uncompacting), gcBefore);
- lastExpiredCheck = currentTimeMillis();
- }
- Set<SSTableReader> candidates =
Sets.newHashSet(filterSuspectSSTables(uncompacting));
-
- List<SSTableReader> compactionCandidates = new
ArrayList<>(getNextNonExpiredSSTables(Sets.difference(candidates, expired),
gcBefore));
- if (!expired.isEmpty())
- {
- logger.trace("Including expired sstables: {}", expired);
- compactionCandidates.addAll(expired);
- }
- return compactionCandidates;
- }
-
- private List<SSTableReader>
getNextNonExpiredSSTables(Iterable<SSTableReader> nonExpiringSSTables, final
int gcBefore)
- {
- int base = cfs.getMinimumCompactionThreshold();
- long now = getNow();
- List<SSTableReader> mostInteresting =
getCompactionCandidates(nonExpiringSSTables, now, base);
- if (mostInteresting != null)
- {
- return mostInteresting;
- }
-
- // if there is no sstable to compact in standard way, try compacting
single sstable whose droppable tombstone
- // ratio is greater than threshold.
- List<SSTableReader> sstablesWithTombstones = Lists.newArrayList();
- for (SSTableReader sstable : nonExpiringSSTables)
- {
- if (worthDroppingTombstones(sstable, gcBefore))
- sstablesWithTombstones.add(sstable);
- }
- if (sstablesWithTombstones.isEmpty())
- return Collections.emptyList();
-
- return
Collections.singletonList(Collections.min(sstablesWithTombstones,
SSTableReader.sizeComparator));
- }
-
- private List<SSTableReader>
getCompactionCandidates(Iterable<SSTableReader> candidateSSTables, long now,
int base)
- {
- Iterable<SSTableReader> candidates =
filterOldSSTables(Lists.newArrayList(candidateSSTables), options.maxSSTableAge,
now);
-
- List<List<SSTableReader>> buckets =
getBuckets(createSSTableAndMinTimestampPairs(candidates), options.baseTime,
base, now, options.maxWindowSize);
- logger.debug("Compaction buckets are {}", buckets);
- updateEstimatedCompactionsByTasks(buckets);
- List<SSTableReader> mostInteresting = newestBucket(buckets,
-
cfs.getMinimumCompactionThreshold(),
-
cfs.getMaximumCompactionThreshold(),
- now,
- options.baseTime,
-
options.maxWindowSize,
- stcsOptions);
- if (!mostInteresting.isEmpty())
- return mostInteresting;
- return null;
- }
-
- /**
- * Gets the timestamp that DateTieredCompactionStrategy considers to be
the "current time".
- * @return the maximum timestamp across all SSTables.
- * @throws java.util.NoSuchElementException if there are no SSTables.
- */
- private long getNow()
- {
- // no need to convert to collection if had an Iterables.max(), but not
present in standard toolkit, and not worth adding
- List<SSTableReader> list = new ArrayList<>();
- Iterables.addAll(list, cfs.getSSTables(SSTableSet.LIVE));
- if (list.isEmpty())
- return 0;
- return Collections.max(list, (o1, o2) ->
Long.compare(o1.getMaxTimestamp(), o2.getMaxTimestamp()))
- .getMaxTimestamp();
- }
-
- /**
- * Removes all sstables with max timestamp older than maxSSTableAge.
- * @param sstables all sstables to consider
- * @param maxSSTableAge the age in milliseconds when an SSTable stops
participating in compactions
- * @param now current time. SSTables with max timestamp less than (now -
maxSSTableAge) are filtered.
- * @return a list of sstables with the oldest sstables excluded
- */
- @VisibleForTesting
- static Iterable<SSTableReader> filterOldSSTables(List<SSTableReader>
sstables, long maxSSTableAge, long now)
- {
- if (maxSSTableAge == 0)
- return sstables;
- final long cutoff = now - maxSSTableAge;
- return filter(sstables, new Predicate<SSTableReader>()
- {
- @Override
- public boolean apply(SSTableReader sstable)
- {
- return sstable.getMaxTimestamp() >= cutoff;
- }
- });
- }
-
- public static List<Pair<SSTableReader, Long>>
createSSTableAndMinTimestampPairs(Iterable<SSTableReader> sstables)
- {
- List<Pair<SSTableReader, Long>> sstableMinTimestampPairs =
Lists.newArrayListWithCapacity(Iterables.size(sstables));
- for (SSTableReader sstable : sstables)
- sstableMinTimestampPairs.add(Pair.create(sstable,
sstable.getMinTimestamp()));
- return sstableMinTimestampPairs;
- }
-
- @Override
- public synchronized void addSSTable(SSTableReader sstable)
- {
- sstables.add(sstable);
- }
-
- @Override
- public synchronized void removeSSTable(SSTableReader sstable)
- {
- sstables.remove(sstable);
- }
-
- @Override
- protected synchronized Set<SSTableReader> getSSTables()
- {
- return ImmutableSet.copyOf(sstables);
- }
-
- /**
- * A target time span used for bucketing SSTables based on timestamps.
- */
- private static class Target
- {
- // How big a range of timestamps fit inside the target.
- public final long size;
- // A timestamp t hits the target iff t / size == divPosition.
- public final long divPosition;
-
- public final long maxWindowSize;
-
- public Target(long size, long divPosition, long maxWindowSize)
- {
- this.size = size;
- this.divPosition = divPosition;
- this.maxWindowSize = maxWindowSize;
- }
-
- /**
- * Compares the target to a timestamp.
- * @param timestamp the timestamp to compare.
- * @return a negative integer, zero, or a positive integer as the
target lies before, covering, or after than the timestamp.
- */
- public int compareToTimestamp(long timestamp)
- {
- return Long.compare(divPosition, timestamp / size);
- }
-
- /**
- * Tells if the timestamp hits the target.
- * @param timestamp the timestamp to test.
- * @return <code>true</code> iff timestamp / size == divPosition.
- */
- public boolean onTarget(long timestamp)
- {
- return compareToTimestamp(timestamp) == 0;
- }
-
- /**
- * Gets the next target, which represents an earlier time span.
- * @param base The number of contiguous targets that will have the
same size. Targets following those will be <code>base</code> times as big.
- * @return
- */
- public Target nextTarget(int base)
- {
- if (divPosition % base > 0 || size * base > maxWindowSize)
- return new Target(size, divPosition - 1, maxWindowSize);
- else
- return new Target(size * base, divPosition / base - 1,
maxWindowSize);
- }
- }
-
-
- /**
- * Group files with similar min timestamp into buckets. Files with recent
min timestamps are grouped together into
- * buckets designated to short timespans while files with older timestamps
are grouped into buckets representing
- * longer timespans.
- * @param files pairs consisting of a file and its min timestamp
- * @param timeUnit
- * @param base
- * @param now
- * @return a list of buckets of files. The list is ordered such that the
files with newest timestamps come first.
- * Each bucket is also a list of files ordered from newest to
oldest.
- */
- @VisibleForTesting
- static <T> List<List<T>> getBuckets(Collection<Pair<T, Long>> files, long
timeUnit, int base, long now, long maxWindowSize)
- {
- // Sort files by age. Newest first.
- final List<Pair<T, Long>> sortedFiles = Lists.newArrayList(files);
- Collections.sort(sortedFiles, Collections.reverseOrder(new
Comparator<Pair<T, Long>>()
- {
- public int compare(Pair<T, Long> p1, Pair<T, Long> p2)
- {
- return p1.right.compareTo(p2.right);
- }
- }));
-
- List<List<T>> buckets = Lists.newArrayList();
- Target target = getInitialTarget(now, timeUnit, maxWindowSize);
- PeekingIterator<Pair<T, Long>> it =
Iterators.peekingIterator(sortedFiles.iterator());
-
- outerLoop:
- while (it.hasNext())
- {
- while (!target.onTarget(it.peek().right))
- {
- // If the file is too new for the target, skip it.
- if (target.compareToTimestamp(it.peek().right) < 0)
- {
- it.next();
-
- if (!it.hasNext())
- break outerLoop;
- }
- else // If the file is too old for the target, switch targets.
- target = target.nextTarget(base);
- }
- List<T> bucket = Lists.newArrayList();
- while (target.onTarget(it.peek().right))
- {
- bucket.add(it.next().left);
-
- if (!it.hasNext())
- break;
- }
- buckets.add(bucket);
- }
-
- return buckets;
- }
-
- @VisibleForTesting
- static Target getInitialTarget(long now, long timeUnit, long maxWindowSize)
- {
- return new Target(timeUnit, now / timeUnit, maxWindowSize);
- }
-
-
- private void updateEstimatedCompactionsByTasks(List<List<SSTableReader>>
tasks)
- {
- int n = 0;
- for (List<SSTableReader> bucket : tasks)
- {
- for (List<SSTableReader> stcsBucket : getSTCSBuckets(bucket,
stcsOptions))
- if (stcsBucket.size() >= cfs.getMinimumCompactionThreshold())
- n += Math.ceil((double)stcsBucket.size() /
cfs.getMaximumCompactionThreshold());
- }
- estimatedRemainingTasks = n;
- cfs.getCompactionStrategyManager().compactionLogger.pending(this, n);
- }
-
-
- /**
- * @param buckets list of buckets, sorted from newest to oldest, from
which to return the newest bucket within thresholds.
- * @param minThreshold minimum number of sstables in a bucket to qualify.
- * @param maxThreshold maximum number of sstables to compact at once (the
returned bucket will be trimmed down to this).
- * @return a bucket (list) of sstables to compact.
- */
- @VisibleForTesting
- static List<SSTableReader> newestBucket(List<List<SSTableReader>> buckets,
int minThreshold, int maxThreshold, long now, long baseTime, long
maxWindowSize, SizeTieredCompactionStrategyOptions stcsOptions)
- {
- // If the "incoming window" has at least minThreshold SSTables, choose
that one.
- // For any other bucket, at least 2 SSTables is enough.
- // In any case, limit to maxThreshold SSTables.
- Target incomingWindow = getInitialTarget(now, baseTime, maxWindowSize);
- for (List<SSTableReader> bucket : buckets)
- {
- boolean inFirstWindow =
incomingWindow.onTarget(bucket.get(0).getMinTimestamp());
- if (bucket.size() >= minThreshold || (bucket.size() >= 2 &&
!inFirstWindow))
- {
- List<SSTableReader> stcsSSTables = getSSTablesForSTCS(bucket,
inFirstWindow ? minThreshold : 2, maxThreshold, stcsOptions);
- if (!stcsSSTables.isEmpty())
- return stcsSSTables;
- }
- }
- return Collections.emptyList();
- }
-
- private static List<SSTableReader>
getSSTablesForSTCS(Collection<SSTableReader> sstables, int minThreshold, int
maxThreshold, SizeTieredCompactionStrategyOptions stcsOptions)
- {
- List<SSTableReader> s =
SizeTieredCompactionStrategy.mostInterestingBucket(getSTCSBuckets(sstables,
stcsOptions), minThreshold, maxThreshold);
- logger.debug("Got sstables {} for STCS from {}", s, sstables);
- return s;
- }
-
- private static List<List<SSTableReader>>
getSTCSBuckets(Collection<SSTableReader> sstables,
SizeTieredCompactionStrategyOptions stcsOptions)
- {
- List<Pair<SSTableReader,Long>> pairs =
SizeTieredCompactionStrategy.createSSTableAndLengthPairs(AbstractCompactionStrategy.filterSuspectSSTables(sstables));
- return SizeTieredCompactionStrategy.getBuckets(pairs,
- stcsOptions.bucketHigh,
- stcsOptions.bucketLow,
-
stcsOptions.minSSTableSize);
- }
-
- @Override
- @SuppressWarnings("resource")
- public synchronized Collection<AbstractCompactionTask> getMaximalTask(int
gcBefore, boolean splitOutput)
- {
- Iterable<SSTableReader> filteredSSTables =
filterSuspectSSTables(sstables);
- if (Iterables.isEmpty(filteredSSTables))
- return null;
- LifecycleTransaction txn =
cfs.getTracker().tryModify(filteredSSTables, OperationType.COMPACTION);
- if (txn == null)
- return null;
- return Collections.<AbstractCompactionTask>singleton(new
CompactionTask(cfs, txn, gcBefore));
- }
-
- @Override
- @SuppressWarnings("resource")
- public synchronized AbstractCompactionTask
getUserDefinedTask(Collection<SSTableReader> sstables, int gcBefore)
- {
- assert !sstables.isEmpty(); // checked for by CM.submitUserDefined
-
- LifecycleTransaction modifier = cfs.getTracker().tryModify(sstables,
OperationType.COMPACTION);
- if (modifier == null)
- {
- logger.trace("Unable to mark {} for compaction; probably a
background compaction got to it first. You can disable background compactions
temporarily if this is a problem", sstables);
- return null;
- }
-
- return new CompactionTask(cfs, modifier,
gcBefore).setUserDefined(true);
- }
-
- public int getEstimatedRemainingTasks()
- {
- return estimatedRemainingTasks;
- }
-
- public long getMaxSSTableBytes()
- {
- return Long.MAX_VALUE;
- }
-
- /**
- * DTCS should not group sstables for anticompaction - this can mix new
and old data
- */
- @Override
- public Collection<Collection<SSTableReader>>
groupSSTablesForAntiCompaction(Collection<SSTableReader> sstablesToGroup)
- {
- Collection<Collection<SSTableReader>> groups = new
ArrayList<>(sstablesToGroup.size());
- for (SSTableReader sstable : sstablesToGroup)
- {
- groups.add(Collections.singleton(sstable));
- }
- return groups;
- }
-
- public static Map<String, String> validateOptions(Map<String, String>
options) throws ConfigurationException
- {
- Map<String, String> uncheckedOptions =
AbstractCompactionStrategy.validateOptions(options);
- uncheckedOptions =
DateTieredCompactionStrategyOptions.validateOptions(options, uncheckedOptions);
-
-
uncheckedOptions.remove(CompactionParams.Option.MIN_THRESHOLD.toString());
-
uncheckedOptions.remove(CompactionParams.Option.MAX_THRESHOLD.toString());
-
- uncheckedOptions =
SizeTieredCompactionStrategyOptions.validateOptions(options, uncheckedOptions);
-
- return uncheckedOptions;
- }
-
- public CompactionLogger.Strategy strategyLogger()
- {
- return new CompactionLogger.Strategy()
- {
- public JsonNode sstable(SSTableReader sstable)
- {
- ObjectNode node = JsonNodeFactory.instance.objectNode();
- node.put("min_timestamp", sstable.getMinTimestamp());
- node.put("max_timestamp", sstable.getMaxTimestamp());
- return node;
- }
-
- public JsonNode options()
- {
- ObjectNode node = JsonNodeFactory.instance.objectNode();
- TimeUnit resolution =
DateTieredCompactionStrategy.this.options.timestampResolution;
-
node.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
- resolution.toString());
- node.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY,
-
resolution.toSeconds(DateTieredCompactionStrategy.this.options.baseTime));
-
node.put(DateTieredCompactionStrategyOptions.MAX_WINDOW_SIZE_KEY,
-
resolution.toSeconds(DateTieredCompactionStrategy.this.options.maxWindowSize));
- return node;
- }
- };
- }
-
- public String toString()
- {
- return String.format("DateTieredCompactionStrategy[%s/%s]",
- cfs.getMinimumCompactionThreshold(),
- cfs.getMaximumCompactionThreshold());
- }
-}
diff --git
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyOptions.java
b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyOptions.java
deleted file mode 100644
index 7604bbc5cf..0000000000
---
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyOptions.java
+++ /dev/null
@@ -1,156 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.db.compaction;
-
-import java.util.Map;
-import java.util.concurrent.TimeUnit;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.cassandra.exceptions.ConfigurationException;
-
-public final class DateTieredCompactionStrategyOptions
-{
- private static final Logger logger =
LoggerFactory.getLogger(DateTieredCompactionStrategyOptions.class);
- protected static final TimeUnit DEFAULT_TIMESTAMP_RESOLUTION =
TimeUnit.MICROSECONDS;
- @Deprecated
- protected static final double DEFAULT_MAX_SSTABLE_AGE_DAYS = 365*1000;
- protected static final long DEFAULT_BASE_TIME_SECONDS = 60;
- protected static final long DEFAULT_MAX_WINDOW_SIZE_SECONDS =
TimeUnit.SECONDS.convert(1, TimeUnit.DAYS);
-
- protected static final int DEFAULT_EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS
= 60 * 10;
- protected static final String TIMESTAMP_RESOLUTION_KEY =
"timestamp_resolution";
- @Deprecated
- protected static final String MAX_SSTABLE_AGE_KEY = "max_sstable_age_days";
- protected static final String BASE_TIME_KEY = "base_time_seconds";
- protected static final String EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY
= "expired_sstable_check_frequency_seconds";
- protected static final String MAX_WINDOW_SIZE_KEY =
"max_window_size_seconds";
-
- @Deprecated
- protected final long maxSSTableAge;
- protected final TimeUnit timestampResolution;
- protected final long baseTime;
- protected final long expiredSSTableCheckFrequency;
- protected final long maxWindowSize;
-
- public DateTieredCompactionStrategyOptions(Map<String, String> options)
- {
- String optionValue = options.get(TIMESTAMP_RESOLUTION_KEY);
- timestampResolution = optionValue == null ?
DEFAULT_TIMESTAMP_RESOLUTION : TimeUnit.valueOf(optionValue);
- if (timestampResolution != DEFAULT_TIMESTAMP_RESOLUTION)
- logger.warn("Using a non-default timestamp_resolution {} - are you
really doing inserts with USING TIMESTAMP <non_microsecond_timestamp> (or
driver equivalent)?", timestampResolution);
- optionValue = options.get(MAX_SSTABLE_AGE_KEY);
- double fractionalDays = optionValue == null ?
DEFAULT_MAX_SSTABLE_AGE_DAYS : Double.parseDouble(optionValue);
- maxSSTableAge = Math.round(fractionalDays *
timestampResolution.convert(1, TimeUnit.DAYS));
- optionValue = options.get(BASE_TIME_KEY);
- baseTime = timestampResolution.convert(optionValue == null ?
DEFAULT_BASE_TIME_SECONDS : Long.parseLong(optionValue), TimeUnit.SECONDS);
- optionValue = options.get(EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY);
- expiredSSTableCheckFrequency =
TimeUnit.MILLISECONDS.convert(optionValue == null ?
DEFAULT_EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS : Long.parseLong(optionValue),
TimeUnit.SECONDS);
- optionValue = options.get(MAX_WINDOW_SIZE_KEY);
- maxWindowSize = timestampResolution.convert(optionValue == null ?
DEFAULT_MAX_WINDOW_SIZE_SECONDS : Long.parseLong(optionValue),
TimeUnit.SECONDS);
- }
-
- public DateTieredCompactionStrategyOptions()
- {
- maxSSTableAge = Math.round(DEFAULT_MAX_SSTABLE_AGE_DAYS *
DEFAULT_TIMESTAMP_RESOLUTION.convert((long) DEFAULT_MAX_SSTABLE_AGE_DAYS,
TimeUnit.DAYS));
- timestampResolution = DEFAULT_TIMESTAMP_RESOLUTION;
- baseTime = timestampResolution.convert(DEFAULT_BASE_TIME_SECONDS,
TimeUnit.SECONDS);
- expiredSSTableCheckFrequency =
TimeUnit.MILLISECONDS.convert(DEFAULT_EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS,
TimeUnit.SECONDS);
- maxWindowSize = timestampResolution.convert(1, TimeUnit.DAYS);
- }
-
- public static Map<String, String> validateOptions(Map<String, String>
options, Map<String, String> uncheckedOptions) throws ConfigurationException
- {
- String optionValue = options.get(TIMESTAMP_RESOLUTION_KEY);
- try
- {
- if (optionValue != null)
- TimeUnit.valueOf(optionValue);
- }
- catch (IllegalArgumentException e)
- {
- throw new
ConfigurationException(String.format("timestamp_resolution %s is not valid",
optionValue));
- }
-
- optionValue = options.get(MAX_SSTABLE_AGE_KEY);
- try
- {
- double maxSStableAge = optionValue == null ?
DEFAULT_MAX_SSTABLE_AGE_DAYS : Double.parseDouble(optionValue);
- if (maxSStableAge < 0)
- {
- throw new ConfigurationException(String.format("%s must be
non-negative: %.2f", MAX_SSTABLE_AGE_KEY, maxSStableAge));
- }
- }
- catch (NumberFormatException e)
- {
- throw new ConfigurationException(String.format("%s is not a
parsable int (base10) for %s", optionValue, MAX_SSTABLE_AGE_KEY), e);
- }
-
- optionValue = options.get(BASE_TIME_KEY);
- try
- {
- long baseTime = optionValue == null ? DEFAULT_BASE_TIME_SECONDS :
Long.parseLong(optionValue);
- if (baseTime <= 0)
- {
- throw new ConfigurationException(String.format("%s must be
greater than 0, but was %d", BASE_TIME_KEY, baseTime));
- }
- }
- catch (NumberFormatException e)
- {
- throw new ConfigurationException(String.format("%s is not a
parsable int (base10) for %s", optionValue, BASE_TIME_KEY), e);
- }
-
- optionValue = options.get(EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY);
- try
- {
- long expiredCheckFrequency = optionValue == null ?
DEFAULT_EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS : Long.parseLong(optionValue);
- if (expiredCheckFrequency < 0)
- {
- throw new ConfigurationException(String.format("%s must not be
negative, but was %d", EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY,
expiredCheckFrequency));
- }
- }
- catch (NumberFormatException e)
- {
- throw new ConfigurationException(String.format("%s is not a
parsable int (base10) for %s", optionValue,
EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY), e);
- }
-
- optionValue = options.get(MAX_WINDOW_SIZE_KEY);
- try
- {
- long maxWindowSize = optionValue == null ?
DEFAULT_MAX_WINDOW_SIZE_SECONDS : Long.parseLong(optionValue);
- if (maxWindowSize < 0)
- {
- throw new ConfigurationException(String.format("%s must not be
negative, but was %d", MAX_WINDOW_SIZE_KEY, maxWindowSize));
- }
- }
- catch (NumberFormatException e)
- {
- throw new ConfigurationException(String.format("%s is not a
parsable int (base10) for %s", optionValue, MAX_WINDOW_SIZE_KEY), e);
- }
-
-
- uncheckedOptions.remove(MAX_SSTABLE_AGE_KEY);
- uncheckedOptions.remove(BASE_TIME_KEY);
- uncheckedOptions.remove(TIMESTAMP_RESOLUTION_KEY);
- uncheckedOptions.remove(EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY);
- uncheckedOptions.remove(MAX_WINDOW_SIZE_KEY);
-
- return uncheckedOptions;
- }
-}
diff --git
a/test/distributed/org/apache/cassandra/distributed/test/SSTableIdGenerationTest.java
b/test/distributed/org/apache/cassandra/distributed/test/SSTableIdGenerationTest.java
index a752b7326d..5794b75cf3 100644
---
a/test/distributed/org/apache/cassandra/distributed/test/SSTableIdGenerationTest.java
+++
b/test/distributed/org/apache/cassandra/distributed/test/SSTableIdGenerationTest.java
@@ -36,7 +36,6 @@ import org.apache.cassandra.cql3.UntypedResultSet;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.db.compaction.AbstractCompactionStrategy;
-import org.apache.cassandra.db.compaction.DateTieredCompactionStrategy;
import org.apache.cassandra.db.compaction.LeveledCompactionStrategy;
import org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy;
import org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy;
@@ -151,7 +150,6 @@ public class SSTableIdGenerationTest extends TestBaseImpl
public final void testCompactionStrategiesWithMixedSSTables() throws
Exception
{
testCompactionStrategiesWithMixedSSTables(SizeTieredCompactionStrategy.class,
-
DateTieredCompactionStrategy.class,
TimeWindowCompactionStrategy.class,
LeveledCompactionStrategy.class);
}
diff --git
a/test/unit/org/apache/cassandra/db/compaction/AbstractCompactionStrategyTest.java
b/test/unit/org/apache/cassandra/db/compaction/AbstractCompactionStrategyTest.java
index bd4b28fc03..1fc43bdcc3 100644
---
a/test/unit/org/apache/cassandra/db/compaction/AbstractCompactionStrategyTest.java
+++
b/test/unit/org/apache/cassandra/db/compaction/AbstractCompactionStrategyTest.java
@@ -19,8 +19,6 @@
package org.apache.cassandra.db.compaction;
import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
import org.junit.After;
import org.junit.BeforeClass;
@@ -44,15 +42,11 @@ public class AbstractCompactionStrategyTest
private static final String KEYSPACE1 = "Keyspace1";
private static final String LCS_TABLE = "LCS_TABLE";
private static final String STCS_TABLE = "STCS_TABLE";
- private static final String DTCS_TABLE = "DTCS_TABLE";
private static final String TWCS_TABLE = "TWCS_TABLE";
@BeforeClass
public static void loadData() throws ConfigurationException
{
- Map<String, String> stcsOptions = new HashMap<>();
- stcsOptions.put("tombstone_compaction_interval", "1");
-
SchemaLoader.prepareServer();
SchemaLoader.createKeyspace(KEYSPACE1,
KeyspaceParams.simple(1),
@@ -60,13 +54,10 @@ public class AbstractCompactionStrategyTest
.compaction(CompactionParams.lcs(Collections.emptyMap())),
SchemaLoader.standardCFMD(KEYSPACE1,
STCS_TABLE)
.compaction(CompactionParams.stcs(Collections.emptyMap())),
- SchemaLoader.standardCFMD(KEYSPACE1,
DTCS_TABLE)
-
.compaction(CompactionParams.create(DateTieredCompactionStrategy.class,
Collections.emptyMap())),
SchemaLoader.standardCFMD(KEYSPACE1,
TWCS_TABLE)
.compaction(CompactionParams.create(TimeWindowCompactionStrategy.class,
Collections.emptyMap())));
Keyspace.open(KEYSPACE1).getColumnFamilyStore(LCS_TABLE).disableAutoCompaction();
Keyspace.open(KEYSPACE1).getColumnFamilyStore(STCS_TABLE).disableAutoCompaction();
-
Keyspace.open(KEYSPACE1).getColumnFamilyStore(DTCS_TABLE).disableAutoCompaction();
Keyspace.open(KEYSPACE1).getColumnFamilyStore(TWCS_TABLE).disableAutoCompaction();
}
@@ -76,7 +67,6 @@ public class AbstractCompactionStrategyTest
Keyspace.open(KEYSPACE1).getColumnFamilyStore(LCS_TABLE).truncateBlocking();
Keyspace.open(KEYSPACE1).getColumnFamilyStore(STCS_TABLE).truncateBlocking();
-
Keyspace.open(KEYSPACE1).getColumnFamilyStore(DTCS_TABLE).truncateBlocking();
Keyspace.open(KEYSPACE1).getColumnFamilyStore(TWCS_TABLE).truncateBlocking();
}
@@ -92,12 +82,6 @@ public class AbstractCompactionStrategyTest
testGetNextBackgroundTaskDoesNotBlock(STCS_TABLE);
}
- @Test(timeout=30000)
- public void testGetNextBackgroundTaskDoesNotBlockDTCS()
- {
- testGetNextBackgroundTaskDoesNotBlock(DTCS_TABLE);
- }
-
@Test(timeout=30000)
public void testGetNextBackgroundTaskDoesNotBlockTWCS()
{
diff --git
a/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
b/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
index 4608285988..b889f94b44 100644
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
@@ -112,19 +112,6 @@ public class CompactionsCQLTest extends CQLTester
waitForMinor(KEYSPACE, currentTable(), SLEEP_TIME, true);
}
-
- @Test
- public void testTriggerMinorCompactionDTCS() throws Throwable
- {
- createTable("CREATE TABLE %s (id text PRIMARY KEY) WITH compaction =
{'class':'DateTieredCompactionStrategy', 'min_threshold':2};");
-
assertTrue(getCurrentColumnFamilyStore().getCompactionStrategyManager().isEnabled());
- execute("insert into %s (id) values ('1') using timestamp 1000"); //
same timestamp = same window = minor compaction triggered
- flush();
- execute("insert into %s (id) values ('1') using timestamp 1000");
- flush();
- waitForMinor(KEYSPACE, currentTable(), SLEEP_TIME, true);
- }
-
@Test
public void testTriggerMinorCompactionTWCS() throws Throwable
{
@@ -217,21 +204,21 @@ public class CompactionsCQLTest extends CQLTester
{
createTable("CREATE TABLE %s (id text PRIMARY KEY)");
Map<String, String> localOptions = new HashMap<>();
- localOptions.put("class", "DateTieredCompactionStrategy");
+ localOptions.put("class", "SizeTieredCompactionStrategy");
getCurrentColumnFamilyStore().setCompactionParameters(localOptions);
-
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
DateTieredCompactionStrategy.class));
+
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
SizeTieredCompactionStrategy.class));
// Invalidate disk boundaries to ensure that boundary invalidation
will not cause the old strategy to be reloaded
getCurrentColumnFamilyStore().invalidateLocalRanges();
// altering something non-compaction related
execute("ALTER TABLE %s WITH gc_grace_seconds = 1000");
// should keep the local compaction strat
-
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
DateTieredCompactionStrategy.class));
+
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
SizeTieredCompactionStrategy.class));
// Alter keyspace replication settings to force compaction strategy
reload
execute("alter keyspace "+keyspace()+" with replication = { 'class' :
'SimpleStrategy', 'replication_factor' : 3 }");
// should keep the local compaction strat
-
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
DateTieredCompactionStrategy.class));
+
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
SizeTieredCompactionStrategy.class));
// altering a compaction option
- execute("ALTER TABLE %s WITH compaction =
{'class':'SizeTieredCompactionStrategy', 'min_threshold':3}");
+ execute("ALTER TABLE %s WITH compaction =
{'class':'SizeTieredCompactionStrategy', 'min_threshold': 3}");
// will use the new option
assertTrue(verifyStrategies(getCurrentColumnFamilyStore().getCompactionStrategyManager(),
SizeTieredCompactionStrategy.class));
}
@@ -241,24 +228,23 @@ public class CompactionsCQLTest extends CQLTester
{
createTable("CREATE TABLE %s (id text PRIMARY KEY)");
Map<String, String> localOptions = new HashMap<>();
- localOptions.put("class", "DateTieredCompactionStrategy");
+ localOptions.put("class", "SizeTieredCompactionStrategy");
localOptions.put("enabled", "false");
getCurrentColumnFamilyStore().setCompactionParameters(localOptions);
assertFalse(getCurrentColumnFamilyStore().getCompactionStrategyManager().isEnabled());
localOptions.clear();
- localOptions.put("class", "DateTieredCompactionStrategy");
+ localOptions.put("class", "SizeTieredCompactionStrategy");
// localOptions.put("enabled", "true"); - this is default!
getCurrentColumnFamilyStore().setCompactionParameters(localOptions);
assertTrue(getCurrentColumnFamilyStore().getCompactionStrategyManager().isEnabled());
}
-
@Test
public void testSetLocalCompactionStrategyEnable() throws Throwable
{
createTable("CREATE TABLE %s (id text PRIMARY KEY)");
Map<String, String> localOptions = new HashMap<>();
- localOptions.put("class", "DateTieredCompactionStrategy");
+ localOptions.put("class", "LeveledCompactionStrategy");
getCurrentColumnFamilyStore().disableAutoCompaction();
assertFalse(getCurrentColumnFamilyStore().getCompactionStrategyManager().isEnabled());
@@ -267,8 +253,6 @@ public class CompactionsCQLTest extends CQLTester
assertTrue(getCurrentColumnFamilyStore().getCompactionStrategyManager().isEnabled());
}
-
-
@Test(expected = IllegalArgumentException.class)
public void testBadLocalCompactionStrategyOptions()
{
diff --git
a/test/unit/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyTest.java
b/test/unit/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyTest.java
deleted file mode 100644
index 156a2a328c..0000000000
---
a/test/unit/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyTest.java
+++ /dev/null
@@ -1,389 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.db.compaction;
-
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.TimeUnit;
-
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-import org.apache.cassandra.SchemaLoader;
-import org.apache.cassandra.Util;
-import org.apache.cassandra.config.CassandraRelevantProperties;
-import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.DecoratedKey;
-import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.db.RowUpdateBuilder;
-import org.apache.cassandra.db.lifecycle.SSTableSet;
-import org.apache.cassandra.exceptions.ConfigurationException;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
-import org.apache.cassandra.schema.KeyspaceParams;
-import org.apache.cassandra.utils.Pair;
-
-import static
org.apache.cassandra.db.compaction.DateTieredCompactionStrategy.filterOldSSTables;
-import static
org.apache.cassandra.db.compaction.DateTieredCompactionStrategy.getBuckets;
-import static
org.apache.cassandra.db.compaction.DateTieredCompactionStrategy.newestBucket;
-import static
org.apache.cassandra.db.compaction.DateTieredCompactionStrategy.validateOptions;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-public class DateTieredCompactionStrategyTest extends SchemaLoader
-{
- public static final String KEYSPACE1 = "DateTieredCompactionStrategyTest";
- private static final String CF_STANDARD1 = "Standard1";
- @BeforeClass
- public static void defineSchema() throws ConfigurationException
- {
- // Disable tombstone histogram rounding for tests
-
CassandraRelevantProperties.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS.setInt(1);
-
- SchemaLoader.prepareServer();
-
- SchemaLoader.createKeyspace(KEYSPACE1,
- KeyspaceParams.simple(1),
- SchemaLoader.standardCFMD(KEYSPACE1, CF_STANDARD1));
- }
-
- @Test
- public void testOptionsValidation() throws ConfigurationException
- {
- Map<String, String> options = new HashMap<>();
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY, "30");
- options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY,
"1825");
- Map<String, String> unvalidated = validateOptions(options);
- assertTrue(unvalidated.isEmpty());
-
- try
- {
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY,
"0");
- validateOptions(options);
- fail(String.format("%s == 0 should be rejected",
DateTieredCompactionStrategyOptions.BASE_TIME_KEY));
- }
- catch (ConfigurationException e) {}
-
- try
- {
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY,
"-1337");
- validateOptions(options);
- fail(String.format("Negative %s should be rejected",
DateTieredCompactionStrategyOptions.BASE_TIME_KEY));
- }
- catch (ConfigurationException e)
- {
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY,
"1");
- }
-
- try
- {
-
options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY, "-1337");
- validateOptions(options);
- fail(String.format("Negative %s should be rejected",
DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY));
- }
- catch (ConfigurationException e)
- {
-
options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY, "0");
- }
-
- try
- {
-
options.put(DateTieredCompactionStrategyOptions.MAX_WINDOW_SIZE_KEY, "-1");
- validateOptions(options);
- fail(String.format("Negative %s should be rejected",
DateTieredCompactionStrategyOptions.MAX_WINDOW_SIZE_KEY));
- }
- catch (ConfigurationException e)
- {
-
options.put(DateTieredCompactionStrategyOptions.MAX_WINDOW_SIZE_KEY, "0");
- }
-
- options.put("bad_option", "1.0");
- unvalidated = validateOptions(options);
- assertTrue(unvalidated.containsKey("bad_option"));
- }
-
- @Test
- public void testTimeConversions()
- {
- Map<String, String> options = new HashMap<>();
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY, "30");
-
options.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
"SECONDS");
-
- DateTieredCompactionStrategyOptions opts = new
DateTieredCompactionStrategyOptions(options);
- assertEquals(opts.maxSSTableAge, TimeUnit.SECONDS.convert(365*1000,
TimeUnit.DAYS));
-
-
options.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
"MILLISECONDS");
- opts = new DateTieredCompactionStrategyOptions(options);
- assertEquals(opts.maxSSTableAge,
TimeUnit.MILLISECONDS.convert(365*1000, TimeUnit.DAYS));
-
-
options.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
"MICROSECONDS");
- options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY,
"10");
- opts = new DateTieredCompactionStrategyOptions(options);
- assertEquals(opts.maxSSTableAge, TimeUnit.MICROSECONDS.convert(10,
TimeUnit.DAYS));
-
- options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY,
"0.5");
- opts = new DateTieredCompactionStrategyOptions(options);
- assertEquals(opts.maxSSTableAge, TimeUnit.MICROSECONDS.convert(1,
TimeUnit.DAYS) / 2);
-
-
options.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
"HOURS");
- options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY,
"0.5");
- opts = new DateTieredCompactionStrategyOptions(options);
- assertEquals(opts.maxSSTableAge, 12);
-
- }
-
- @Test
- public void testGetBuckets()
- {
- List<Pair<String, Long>> pairs = Lists.newArrayList(
- Pair.create("a", 199L),
- Pair.create("b", 299L),
- Pair.create("a", 1L),
- Pair.create("b", 201L)
- );
- List<List<String>> buckets = getBuckets(pairs, 100L, 2, 200L,
Long.MAX_VALUE);
- assertEquals(2, buckets.size());
-
- for (List<String> bucket : buckets)
- {
- assertEquals(2, bucket.size());
- assertEquals(bucket.get(0), bucket.get(1));
- }
-
-
- pairs = Lists.newArrayList(
- Pair.create("a", 2000L),
- Pair.create("b", 3600L),
- Pair.create("a", 200L),
- Pair.create("c", 3950L),
- Pair.create("too new", 4125L),
- Pair.create("b", 3899L),
- Pair.create("c", 3900L)
- );
- buckets = getBuckets(pairs, 100L, 3, 4050L, Long.MAX_VALUE);
- // targets (divPosition, size): (40, 100), (39, 100), (12, 300), (3,
900), (0, 2700)
- // in other words: 0 - 2699, 2700 - 3599, 3600 - 3899, 3900 - 3999,
4000 - 4099
- assertEquals(3, buckets.size());
-
- for (List<String> bucket : buckets)
- {
- assertEquals(2, bucket.size());
- assertEquals(bucket.get(0), bucket.get(1));
- }
-
-
- // Test base 1.
- pairs = Lists.newArrayList(
- Pair.create("a", 200L),
- Pair.create("a", 299L),
- Pair.create("b", 2000L),
- Pair.create("b", 2014L),
- Pair.create("c", 3610L),
- Pair.create("c", 3690L),
- Pair.create("d", 3898L),
- Pair.create("d", 3899L),
- Pair.create("e", 3900L),
- Pair.create("e", 3950L),
- Pair.create("too new", 4125L)
- );
- buckets = getBuckets(pairs, 100L, 1, 4050L, Long.MAX_VALUE);
-
- assertEquals(5, buckets.size());
-
- for (List<String> bucket : buckets)
- {
- assertEquals(2, bucket.size());
- assertEquals(bucket.get(0), bucket.get(1));
- }
- }
-
- @Test
- public void testPrepBucket()
- {
- Keyspace keyspace = Keyspace.open(KEYSPACE1);
- ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD1);
- cfs.disableAutoCompaction();
-
- ByteBuffer value = ByteBuffer.wrap(new byte[100]);
-
- // create 3 sstables
- int numSSTables = 3;
- for (int r = 0; r < numSSTables; r++)
- {
- DecoratedKey key = Util.dk(String.valueOf(r));
- new RowUpdateBuilder(cfs.metadata(), r, key.getKey())
- .clustering("column")
- .add("val", value).build().applyUnsafe();
-
- Util.flush(cfs);
- }
- Util.flush(cfs);
-
- List<SSTableReader> sstrs = new ArrayList<>(cfs.getLiveSSTables());
-
- List<SSTableReader> newBucket =
newestBucket(Collections.singletonList(sstrs.subList(0, 2)), 4, 32, 9, 10,
Long.MAX_VALUE, new SizeTieredCompactionStrategyOptions());
- assertTrue("incoming bucket should not be accepted when it has below
the min threshold SSTables", newBucket.isEmpty());
-
- newBucket = newestBucket(Collections.singletonList(sstrs.subList(0,
2)), 4, 32, 10, 10, Long.MAX_VALUE, new SizeTieredCompactionStrategyOptions());
- assertFalse("non-incoming bucket should be accepted when it has at
least 2 SSTables", newBucket.isEmpty());
-
- assertEquals("an sstable with a single value should have equal min/max
timestamps", sstrs.get(0).getMinTimestamp(), sstrs.get(0).getMaxTimestamp());
- assertEquals("an sstable with a single value should have equal min/max
timestamps", sstrs.get(1).getMinTimestamp(), sstrs.get(1).getMaxTimestamp());
- assertEquals("an sstable with a single value should have equal min/max
timestamps", sstrs.get(2).getMinTimestamp(), sstrs.get(2).getMaxTimestamp());
- cfs.truncateBlocking();
- }
-
- @Test
- public void testFilterOldSSTables()
- {
- Keyspace keyspace = Keyspace.open(KEYSPACE1);
- ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD1);
- cfs.disableAutoCompaction();
-
- ByteBuffer value = ByteBuffer.wrap(new byte[100]);
-
- // create 3 sstables
- int numSSTables = 3;
- for (int r = 0; r < numSSTables; r++)
- {
- DecoratedKey key = Util.dk(String.valueOf(r));
- new RowUpdateBuilder(cfs.metadata(), r, key.getKey())
- .clustering("column")
- .add("val", value).build().applyUnsafe();
-
- Util.flush(cfs);
- }
- Util.flush(cfs);
-
- Iterable<SSTableReader> filtered;
- List<SSTableReader> sstrs = new ArrayList<>(cfs.getLiveSSTables());
-
- filtered = filterOldSSTables(sstrs, 0, 2);
- assertEquals("when maxSSTableAge is zero, no sstables should be
filtered", sstrs.size(), Iterables.size(filtered));
-
- filtered = filterOldSSTables(sstrs, 1, 2);
- assertEquals("only the newest 2 sstables should remain", 2,
Iterables.size(filtered));
-
- filtered = filterOldSSTables(sstrs, 1, 3);
- assertEquals("only the newest sstable should remain", 1,
Iterables.size(filtered));
-
- filtered = filterOldSSTables(sstrs, 1, 4);
- assertEquals("no sstables should remain when all are too old", 0,
Iterables.size(filtered));
- cfs.truncateBlocking();
- }
-
-
- @Test
- public void testDropExpiredSSTables() throws InterruptedException
- {
- Keyspace keyspace = Keyspace.open(KEYSPACE1);
- ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD1);
- cfs.disableAutoCompaction();
-
- ByteBuffer value = ByteBuffer.wrap(new byte[100]);
-
- // create 2 sstables
- DecoratedKey key = Util.dk(String.valueOf("expired"));
- new RowUpdateBuilder(cfs.metadata(), System.currentTimeMillis(), 1,
key.getKey())
- .clustering("column")
- .add("val", value).build().applyUnsafe();
-
- Util.flush(cfs);
- SSTableReader expiredSSTable = cfs.getLiveSSTables().iterator().next();
- Thread.sleep(10);
-
- key = Util.dk(String.valueOf("nonexpired"));
- new RowUpdateBuilder(cfs.metadata(), System.currentTimeMillis(),
key.getKey())
- .clustering("column")
- .add("val", value).build().applyUnsafe();
-
- Util.flush(cfs);
- assertEquals(cfs.getLiveSSTables().size(), 2);
-
- Map<String, String> options = new HashMap<>();
-
- options.put(DateTieredCompactionStrategyOptions.BASE_TIME_KEY, "30");
-
options.put(DateTieredCompactionStrategyOptions.TIMESTAMP_RESOLUTION_KEY,
"MILLISECONDS");
- options.put(DateTieredCompactionStrategyOptions.MAX_SSTABLE_AGE_KEY,
Double.toString((1d / (24 * 60 * 60))));
-
options.put(DateTieredCompactionStrategyOptions.EXPIRED_SSTABLE_CHECK_FREQUENCY_SECONDS_KEY,
"0");
- DateTieredCompactionStrategy dtcs = new
DateTieredCompactionStrategy(cfs, options);
- for (SSTableReader sstable : cfs.getLiveSSTables())
- dtcs.addSSTable(sstable);
- dtcs.startup();
- assertNull(dtcs.getNextBackgroundTask((int)
(System.currentTimeMillis() / 1000)));
- Thread.sleep(2000);
- AbstractCompactionTask t = dtcs.getNextBackgroundTask((int)
(System.currentTimeMillis()/1000));
- assertNotNull(t);
- assertEquals(1, Iterables.size(t.transaction.originals()));
- SSTableReader sstable = t.transaction.originals().iterator().next();
- assertEquals(sstable, expiredSSTable);
- t.transaction.abort();
- cfs.truncateBlocking();
- }
-
- @Test
- public void testSTCSBigWindow()
- {
- Keyspace keyspace = Keyspace.open(KEYSPACE1);
- ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD1);
- cfs.disableAutoCompaction();
- ByteBuffer bigValue = ByteBuffer.wrap(new byte[10000]);
- ByteBuffer value = ByteBuffer.wrap(new byte[100]);
- int numSSTables = 40;
- // create big sstabels out of half:
- long timestamp = System.currentTimeMillis();
- for (int r = 0; r < numSSTables / 2; r++)
- {
- for (int i = 0; i < 10; i++)
- {
- DecoratedKey key = Util.dk(String.valueOf(r));
- new RowUpdateBuilder(cfs.metadata(), timestamp, key.getKey())
- .clustering("column")
- .add("val", bigValue).build().applyUnsafe();
- }
- Util.flush(cfs);
- }
- // and small ones:
- for (int r = 0; r < numSSTables / 2; r++)
- {
- DecoratedKey key = Util.dk(String.valueOf(r));
- new RowUpdateBuilder(cfs.metadata(), timestamp, key.getKey())
- .clustering("column")
- .add("val", value).build().applyUnsafe();
- Util.flush(cfs);
- }
- Map<String, String> options = new HashMap<>();
- options.put(SizeTieredCompactionStrategyOptions.MIN_SSTABLE_SIZE_KEY,
"1");
- DateTieredCompactionStrategy dtcs = new
DateTieredCompactionStrategy(cfs, options);
- for (SSTableReader sstable : cfs.getSSTables(SSTableSet.CANONICAL))
- dtcs.addSSTable(sstable);
- AbstractCompactionTask task = dtcs.getNextBackgroundTask(0);
- assertEquals(20, task.transaction.originals().size());
- task.transaction.abort();
- cfs.truncateBlocking();
- }
-}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
