Re: how to immediately delete tombstones

[Alain RODRIGUEZ]

Hello,

When you don't have any disk space anymore (or not much), there are things
you can do :



*Make some space:*
- Remove snapshots (nodetool clearsnapshot)
- Remove any heap dump that might be stored there
- Remove *old* -tmp- SSTables that could still be around.
- Truncate unused table / data: Truncate does not create tombstone, but
removes the files, after creating a snapshot (default behavior). Thus
truncating/dropping a table and removing the snapshot could produce
immediate disk space availability.
- Anything else than SSTable in this disk that can be removed?

OR

*Add some space:*

- Add some disk space temporary (EBS, physical disk)
- Make sure to clean tombstones ('uncheck_tombstone_compaction: true' often
helped)
- Wait for tombstones to be compacted with all the data it shadows and disk
space to reduce
- Remove extra disk

OR

*Play around, at the edge*:

- Look at the biggest SSTables that you can actually compact (be aware of
the compressed vs uncompressed sizes when monitoring - I think the thing
was that 'nodetool compactionstats -H' shows the values uncompressed)
- Use sstablemetadata to determine the ratio of the data that is droppable
- Run user defined compaction on these sstables specifically
- If it works and there is more disk space available, reproduce with bigger
sstables.

In some complex cases removing commit logs helped as well, but this is
riskier already as it would be playing with consistency/durability
considerations.

I'm using cassandra on a single node.
>

I would not play with commit logs with a single-node setup. But I imagine
it is not a production 'cluster' either.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-06-02 8:29 GMT+01:00 Nitan Kainth :

> You can compact selective sstables using jmx Call.
>
> Sent from my iPhone
>
> On Jun 2, 2018, at 12:04 AM, onmstester onmstester 
> wrote:
>
> Thanks for your replies
> But my current situation is that i do not have enough free disk for my
> biggest sstable, so i could not run major compaction or nodetool
> garbagecollect
>
> Sent using Zoho Mail 
>
>
>  On Thu, 31 May 2018 22:32:32 +0430 *Alain RODRIGUEZ
> >* wrote 
>
>
>

Re: how to immediately delete tombstones

[Nitan Kainth]

You can compact selective sstables using jmx Call.

Sent from my iPhone

> On Jun 2, 2018, at 12:04 AM, onmstester onmstester  
> wrote:
> 
> Thanks for your replies
> But my current situation is that i do not have enough free disk for my 
> biggest sstable, so i could not run major compaction or nodetool 
> garbagecollect
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Thu, 31 May 2018 22:32:32 +0430 Alain RODRIGUEZ  
> wrote 
> 
> 

Re: how to immediately delete tombstones

[onmstester onmstester]

Thanks for your replies

But my current situation is that i do not have enough free disk for my biggest 
sstable, so i could not run major compaction or nodetool garbagecollect


Sent using Zoho Mail






 On Thu, 31 May 2018 22:32:32 +0430 Alain RODRIGUEZ 
 wrote 

Re: how to immediately delete tombstones

[Alain RODRIGUEZ]

Hello,

It's a very common but somewhat complex topic. We wrote about it 2 years
ago and I really think this post might have answers you are looking for:
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html

Something that you could try (if you do care ending up with one big sstable
:)) is to enable 'unchecked_tombstone_compaction'. This removes a pre-check
supposed to only trigger tombstone compactions when it is worth it (ie
there is no sstable overlaps) but that over time proved to be inefficient
from a tombstone eviction perspective.

If data can actually be deleted (no overlaps, gc_grace_seconds lowered or
reached, ...) then changing option 'unchecked_tombstone_compaction' to
'true' might do a lot of good in terms of disk space. Be aware that a bunch
of compactions might be triggered and that disk space will start by
increasing before (possibly) reducing after the compactions.

The gc_grace of table was default (10 days), now i set that to 0, although
> many compactions finished but no space reclaimed so far.
>

Be aware that if some deletes did not reach all the replicas, the data will
eventually come back as lowering the gc_grace_seconds, you don't allow
repairs to process the data before tombstones are actually evicted.

Also, by setting the gc_grace_seconds to 0, you also disabled the hints
altogether. gc_grace_seconds should always be equivalent to
'max_hint_windows_in_ms'. My colleague Radovan wrote a post with
more information on this:
http://thelastpickle.com/blog/2018/03/21/hinted-handoff-gc-grace-demystified.html

Good luck with your tombstones, again, those are a bit tricky to handle
sometimes ;-)

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-05-31 16:19 GMT+01:00 Nicolas Guyomar :

> Hi,
>
> You need to manually force compaction if you do not care ending up with
> one big sstable (nodetool compact)
>
> On 31 May 2018 at 11:07, onmstester onmstester 
> wrote:
>
>> Hi,
>> I've deleted 50% of my data row by row now disk usage of cassandra data
>> is more than 80%.
>> The gc_grace of table was default (10 days), now i set that to 0,
>> although many compactions finished but no space reclaimed so far.
>> How could i force deletion of tombstones in sstables and reclaim the disk
>> used by deleted rows?
>> I'm using cassandra on a single node.
>>
>> Sent using Zoho Mail 
>>
>>
>>
>

Re: how to immediately delete tombstones

[Nicolas Guyomar]

Hi,

You need to manually force compaction if you do not care ending up with one
big sstable (nodetool compact)

On 31 May 2018 at 11:07, onmstester onmstester  wrote:

> Hi,
> I've deleted 50% of my data row by row now disk usage of cassandra data is
> more than 80%.
> The gc_grace of table was default (10 days), now i set that to 0, although
> many compactions finished but no space reclaimed so far.
> How could i force deletion of tombstones in sstables and reclaim the disk
> used by deleted rows?
> I'm using cassandra on a single node.
>
> Sent using Zoho Mail 
>
>
>