Re: Sudden disk usage

[Branton Davis]

Yes, however, for anyone dealing with this in the future, adding a second
EBS volume is much quicker during a crisis.  Granted, I haven't yet gone
through the process of merging the two volumes, but it sounds like it
should be fairly straight forward.  The tradeoff, of course, is spending
the time moving data later versus crisis mode (when we had 3 nodes, all
running out of space during cleanup/compaction).

On Wed, Feb 17, 2016 at 11:42 AM, Ben Bromhead  wrote:

> +1 to checking for snapshots. Cassandra by default will automatically
> snapshot tables before destructive actions like drop or truncate.
>
> Some general advice regarding cleanup. Cleanup will result in a temporary
> increase in both disk I/O load and disk space usage (especially with STCS).
> It should only be used as part of a planned increase in capacity when you
> still have plenty of disk space left on your existing nodes.
>
> If you are running Cassandra in the cloud (AWS, Azure etc) you can add an
> EBS volume, copy your sstables to it then bind mount it to the troubled CF
> directory. This will give you some emergency disk space to let compaction
> and cleanup do its thing safely.
>
> On Tue, 16 Feb 2016 at 10:57 Robert Coli  wrote:
>
>> On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis <
>> branton.da...@spanning.com> wrote:
>>
>>> We use SizeTieredCompaction.  The nodes were about 67% full and we were
>>> planning on adding new nodes (doubling the cluster to 6) soon.
>>>
>>
>> Be sure to add those new nodes one at a time.
>>
>> Have you checked for, and cleared, old snapshots? Snapshots are
>> automatically taken at various times and have the unusual property of
>> growing larger over time. This is because they are hard links of data files
>> and do not take up disk space of their own until the files they link to are
>> compacted into new files.
>>
>> =Rob
>>
>>
> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>

Re: Sudden disk usage

[Branton Davis]

Yep, nodes were added one at a time and I ran clearsnapshots (there weren't
any).  The way I finally got past this was adding a second volume via
data_file_directories (JBOD).

On Tue, Feb 16, 2016 at 12:57 PM, Robert Coli  wrote:

> On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis  > wrote:
>
>> We use SizeTieredCompaction.  The nodes were about 67% full and we were
>> planning on adding new nodes (doubling the cluster to 6) soon.
>>
>
> Be sure to add those new nodes one at a time.
>
> Have you checked for, and cleared, old snapshots? Snapshots are
> automatically taken at various times and have the unusual property of
> growing larger over time. This is because they are hard links of data files
> and do not take up disk space of their own until the files they link to are
> compacted into new files.
>
> =Rob
>
>

Re: Sudden disk usage

[Ben Bromhead]

+1 to checking for snapshots. Cassandra by default will automatically
snapshot tables before destructive actions like drop or truncate.

Some general advice regarding cleanup. Cleanup will result in a temporary
increase in both disk I/O load and disk space usage (especially with STCS).
It should only be used as part of a planned increase in capacity when you
still have plenty of disk space left on your existing nodes.

If you are running Cassandra in the cloud (AWS, Azure etc) you can add an
EBS volume, copy your sstables to it then bind mount it to the troubled CF
directory. This will give you some emergency disk space to let compaction
and cleanup do its thing safely.

On Tue, 16 Feb 2016 at 10:57 Robert Coli  wrote:

> On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis  > wrote:
>
>> We use SizeTieredCompaction.  The nodes were about 67% full and we were
>> planning on adding new nodes (doubling the cluster to 6) soon.
>>
>
> Be sure to add those new nodes one at a time.
>
> Have you checked for, and cleared, old snapshots? Snapshots are
> automatically taken at various times and have the unusual property of
> growing larger over time. This is because they are hard links of data files
> and do not take up disk space of their own until the files they link to are
> compacted into new files.
>
> =Rob
>
>
-- 
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Sudden disk usage

[Robert Coli]

On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis 
wrote:

> We use SizeTieredCompaction.  The nodes were about 67% full and we were
> planning on adding new nodes (doubling the cluster to 6) soon.
>

Be sure to add those new nodes one at a time.

Have you checked for, and cleared, old snapshots? Snapshots are
automatically taken at various times and have the unusual property of
growing larger over time. This is because they are hard links of data files
and do not take up disk space of their own until the files they link to are
compacted into new files.

=Rob

Re: Sudden disk usage

[Branton Davis]

We use SizeTieredCompaction.  The nodes were about 67% full and we were
planning on adding new nodes (doubling the cluster to 6) soon.  I've been
watching the disk space used, and the nodes were taking about 100GB during
compaction, so I thought we were going to be okay for another week.  The
other nodes are still like that.  It's just this one node that's now taking
a lot more and I'm worried about running out of disk space.  I've gone
ahead and added 2 new nodes and was hoping cleanup would buy some space,
but it looks like compaction still has to complete, and is just continuing
to eat up space.

I guess, worst case scenario, I can remove that node and replace it, but
it's just really strange that this is happening with just the one node, and
apparently adding the new nodes hasn't helped in the short term.

On Sat, Feb 13, 2016 at 4:37 PM, Jan Kesten  wrote:

> Hi,
>
> what kind of compaction strategy do you use? What you are about to see is
> a compaction likely - think of 4 sstables of 50gb each, compacting those
> can take up 200g while rewriting the new sstable. After that the old ones
> are deleted and space will be freed again.
>
> If using SizeTieredCompaction you can end up with very huge sstables as I
> do (>250gb each). In the worst case you could possibly need twice the space
> - a reason why I set up my monitoring for disk to 45% usage.
>
> Just my 2 cents.
> Jan
>
> Von meinem iPhone gesendet
>
> > Am 13.02.2016 um 08:48 schrieb Branton Davis  >:
> >
> > One of our clusters had a strange thing happen tonight.  It's a 3 node
> cluster, running 2.1.10.  The primary keyspace has RF 3, vnodes with 256
> tokens.
> >
> > This evening, over the course of about 6 hours, disk usage increased
> from around 700GB to around 900GB on only one node.  I was at a loss as to
> what was happening and, on a whim, decided to run nodetool cleanup on the
> instance.  I had no reason to believe that it was necessary, as no nodes
> were added or tokens moved (not intentionally, anyhow).  But it immediately
> cleared up that extra space.
> >
> > I'm pretty lost as to what would have happened here.  Any ideas where to
> look?
> >
> > Thanks!
> >
>

Re: Sudden disk usage

[Jan Kesten]

Hi,

what kind of compaction strategy do you use? What you are about to see is a 
compaction likely - think of 4 sstables of 50gb each, compacting those can take 
up 200g while rewriting the new sstable. After that the old ones are deleted 
and space will be freed again. 

If using SizeTieredCompaction you can end up with very huge sstables as I do 
(>250gb each). In the worst case you could possibly need twice the space - a 
reason why I set up my monitoring for disk to 45% usage.

Just my 2 cents.
Jan

Von meinem iPhone gesendet

> Am 13.02.2016 um 08:48 schrieb Branton Davis :
> 
> One of our clusters had a strange thing happen tonight.  It's a 3 node 
> cluster, running 2.1.10.  The primary keyspace has RF 3, vnodes with 256 
> tokens.
> 
> This evening, over the course of about 6 hours, disk usage increased from 
> around 700GB to around 900GB on only one node.  I was at a loss as to what 
> was happening and, on a whim, decided to run nodetool cleanup on the 
> instance.  I had no reason to believe that it was necessary, as no nodes were 
> added or tokens moved (not intentionally, anyhow).  But it immediately 
> cleared up that extra space.
> 
> I'm pretty lost as to what would have happened here.  Any ideas where to look?
> 
> Thanks!
> 

Sudden disk usage

[Branton Davis]

One of our clusters had a strange thing happen tonight.  It's a 3 node
cluster, running 2.1.10.  The primary keyspace has RF 3, vnodes with 256
tokens.

This evening, over the course of about 6 hours, disk usage increased from
around 700GB to around 900GB on only one node.  I was at a loss as to what
was happening and, on a whim, decided to run nodetool cleanup on the
instance.  I had no reason to believe that it was necessary, as no nodes
were added or tokens moved (not intentionally, anyhow).  But it immediately
cleared up that extra space.

I'm pretty lost as to what would have happened here.  Any ideas where to
look?

Thanks!