RE: TWCS Log Warning
A typo, even though they are debug message I am interested in knowing their
meaning since they happen very often.
Thanks Jon for the video, that is helpful, our data is partitioned by other
metrics and not time, its clustered by a timestamp, our biggest partition is
~8MB.
Since we are using TWCS with 7 days window each week data should be in its own
bucket sstable, dare I ask, is there a way to manually remove \‘archive’ very
old time buckets at one point by removing those sstables or that could break
things?
From: Jon Haddad
Sent: Thursday, May 23, 2024 5:43 PM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: Re: TWCS Log Warning
As an aside, if you're not putting a TTL on your data, it's a good idea to be
proactive and use multiple tables. For example, one per month or year. This
allows you the flexibility to delete your data by dropping old tables.
Storing old data in Cassandra is expensive. Once you get to a certain point it
becomes far more cost effective to offload your old data to an object store and
keep your Cassandra cluster to a minimum size.
I gave a talk on this topic on my YT channel:
https://www.youtube.com/live/Ysfi3V2KQtU
Jon
On Thu, May 23, 2024 at 7:35 AM Bowen Song via user
mailto:user@cassandra.apache.org>> wrote:
As the log level name "DEBUG" suggested, these are debug messages, not warnings.
Is there any reason made you believe that these messages are warnings?
On 23/05/2024 11:10, Isaeed Mohanna wrote:
Hi
I have a big table (~220GB reported by used space live by tablestats) with time
series data that uses TWCS with the following settings
compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '7', 'compaction_window_unit': 'DAYS',
'max_threshold': '32', 'min_threshold': '4'}
The table does not have a TTL configured since we need the data, it now has
~450 sstables, I have had this setup for several years and so far I am
satisfied with the performance, we mostly read\write data from the previous
several months. Requests for earlier data occur but not in the quantities and
performance is less critical then.
I have recently noticed reoccurring warning in the Cassandra log file and I
wanted to ask about their meaning and wither I need to do something about it
DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,655
TimeWindowCompactionStrategy.java:122 - TWCS expired check sufficiently far in
the past, checking for fully expired SSTables
DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,658
TimeWindowCompactionStrategy.java:122 - TWCS expired check sufficiently far in
the past, checking for fully expired SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,656
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
DEBUG [CompactionExecutor:356244] 2024-05-23 09:06:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired
SSTables
The debug messages above appear in one of my Cassandra nodes every several
minutes, I have a 4 node cluster with RF=3.
Is there anything I need to do about those messages or its safe to ignore them
Thank you for the help
Re: TWCS Log Warning
As an aside, if you're not putting a TTL on your data, it's a good idea to
be proactive and use multiple tables. For example, one per month or year.
This allows you the flexibility to delete your data by dropping old tables.
Storing old data in Cassandra is expensive. Once you get to a certain
point it becomes far more cost effective to offload your old data to an
object store and keep your Cassandra cluster to a minimum size.
I gave a talk on this topic on my YT channel:
https://www.youtube.com/live/Ysfi3V2KQtU
Jon
On Thu, May 23, 2024 at 7:35 AM Bowen Song via user <
user@cassandra.apache.org> wrote:
> As the log level name "DEBUG" suggested, these are debug messages, not
> warnings.
>
> Is there any reason made you believe that these messages are warnings?
>
>
> On 23/05/2024 11:10, Isaeed Mohanna wrote:
>
> Hi
>
> I have a big table (~220GB reported by used space live by tablestats) with
> time series data that uses TWCS with the following settings
> compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '7', 'compaction_window_unit': 'DAYS',
> 'max_threshold': '32', 'min_threshold': '4'}
>
> The table does not have a TTL configured since we need the data, it now
> has ~450 sstables, I have had this setup for several years and so far I am
> satisfied with the performance, we mostly read\write data from the previous
> several months. Requests for earlier data occur but not in the quantities
> and performance is less critical then.
>
> I have recently noticed reoccurring warning in the Cassandra log file and
> I wanted to ask about their meaning and wither I need to do something about
> it
>
> DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,655
> TimeWindowCompactionStrategy.java:122 - TWCS expired check sufficiently far
> in the past, checking for fully expired SSTables
> DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,658
> TimeWindowCompactionStrategy.java:122 - TWCS expired check sufficiently far
> in the past, checking for fully expired SSTables
> DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,655
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,656
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> DEBUG [CompactionExecutor:356244] 2024-05-23 09:06:00,490
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
>
>
>
> The debug messages above appear in one of my Cassandra nodes every several
> minutes, I have a 4 node cluster with RF=3.
>
> Is there anything I need to do about those messages or its safe to ignore
> them
>
> Thank you for the help
>
>
Re: TWCS Log Warning
As the log level name "DEBUG" suggested, these are debug messages, not
warnings.
Is there any reason made you believe that these messages are warnings?
On 23/05/2024 11:10, Isaeed Mohanna wrote:
Hi
I have a big table (~220GB reported by used space live by tablestats)
with time series data that uses TWCS with the following settings
compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '7', 'compaction_window_unit': 'DAYS',
'max_threshold': '32', 'min_threshold': '4'}
The table does not have a TTL configured since we need the data, it
now has ~450 sstables, I have had this setup for several years and so
far I am satisfied with the performance, we mostly read\write data
from the previous several months. Requests for earlier data occur but
not in the quantities and performance is less critical then.
I have recently noticed reoccurring warning in the Cassandra log file
and I wanted to ask about their meaning and wither I need to do
something about it
DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:01:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,655
TimeWindowCompactionStrategy.java:122 - TWCS expired check
sufficiently far in the past, checking for fully expired SSTables
DEBUG [CompactionExecutor:356243] 2024-05-23 09:02:59,658
TimeWindowCompactionStrategy.java:122 - TWCS expired check
sufficiently far in the past, checking for fully expired SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,655
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356242] 2024-05-23 09:03:59,656
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356245] 2024-05-23 09:05:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
DEBUG [CompactionExecutor:356244] 2024-05-23 09:06:00,490
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
The debug messages above appear in one of my Cassandra nodes every
several minutes, I have a 4 node cluster with RF=3.
Is there anything I need to do about those messages or its safe to
ignore them
Thank you for the help
