Re: Deleting old archives is slow

[Graham Percival]

On Tue, Sep 29, 2015 at 05:48:00PM -0700, Colin Percival wrote:
> On 09/29/15 11:31, Tom Limoncelli wrote:
> > On Mon, Sep 28, 2015 at 10:17 AM, James Turner  
> > wrote:
> >> tarsnap -d -f archivename-2014010101 -f archivename-2014010102
> >> -f archivename-2014010103
> > 
> > Also, I noticed that it fails the first time it hits a
> > non-existent archive, so restarting after a failure requires a binary
> > search or just reverting to doing them one at a time.
> > 
> > tarsnap: Archive does not exist: archivename-2014010103
> > tarsnap: Error deleting archive
> 
> Hmm, I never thought of that.  Maybe we should have a --keep-going option?
> https://github.com/Tarsnap/tarsnap/issues/77

To clarify for the list archives: --keep-going was added in Tarsnap 1.0.37.

Cheers,
- Graham Percival

Re: Deleting old archives is slow

[Colin Percival]

On 09/29/15 11:31, Tom Limoncelli wrote:
> On Mon, Sep 28, 2015 at 10:17 AM, James Turner  wrote:
>> tarsnap -d -f archivename-2014010101 -f archivename-2014010102
>> -f archivename-2014010103
> 
> It is good to know that this works.  However it doesn't seem much
> faster.

Whether it's faster depends on the amount of shared data between the archives.

> Also, I noticed that it fails the first time it hits a
> non-existent archive, so restarting after a failure requires a binary
> search or just reverting to doing them one at a time.
> 
> tarsnap: Archive does not exist: archivename-2014010103
> tarsnap: Error deleting archive

Hmm, I never thought of that.  Maybe we should have a --keep-going option?
https://github.com/Tarsnap/tarsnap/issues/77

As a workaround you could run `tarsnap --list-archives` and use that to filter
your deletion list.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

Re: Deleting old archives is slow

[Tom Limoncelli]

On Mon, Sep 28, 2015 at 10:17 AM, James Turner  wrote:
> tarsnap -d -f archivename-2014010101 -f archivename-2014010102
> -f archivename-2014010103

It is good to know that this works.  However it doesn't seem much
faster.  Also, I noticed that it fails the first time it hits a
non-existent archive, so restarting after a failure requires a binary
search or just reverting to doing them one at a time.

tarsnap: Archive does not exist: archivename-2014010103
tarsnap: Error deleting archive

Tom

-- 
Email: t...@whatexit.orgWork: tlimonce...@stackoverflow.com
Skype: YesThatTom
Blog:  http://EverythingSysadmin.com

Re: Deleting old archives is slow

[Daniel Staal]

--As of September 29, 2015 3:06:39 PM -0400, Garance AE Drosehn is alleged 
to have said:



On Sep 29, 2015, at 2:31 PM, Tom Limoncelli  wrote:


On Mon, Sep 28, 2015 at 10:17 AM, James Turner 
wrote:

tarsnap -d -f archivename-2014010101 -f archivename-2014010102
-f archivename-2014010103


It is good to know that this works.  However it doesn't seem much
faster.  Also, I noticed that it fails the first time it hits a
non-existent archive, so restarting after a failure requires a binary
search or just reverting to doing them one at a time.

tarsnap: Archive does not exist: archivename-2014010103
tarsnap: Error deleting archive


I assume there isn't much you can do about how long it takes to delete
each archive.  And you probably can't delete multiple archives at once
(in parallel), because they all have to work on the same database of
what data is still referenced by at least one of the remaining archives.

So I suspect the best you can do is stack up the delete requests so
you don't have to wait around for each one to finish before typing
in the next one.  I spend so much time typing in command-lines that
I would just type in a 'for' loop:


--As for the rest, it is mine.

It is slightly faster to have it all in one commandline - there's some 
repetitive overhead involved in doing it archive-by-archive, that tarsnap 
will avoid doing if it has the whole list.  (It also saves some data 
transfer - both on your network and on tarsnap.)


Daniel T. Staal

---
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---

Re: Deleting old archives is slow

[Tom Limoncelli]

Thanks, Garance. That's all good advice.  A similar thing that I
recommend to new sysadmins is that their scripts halt on some
threshold. For example, if the number of archives to delete is more
than something reasonable (in my case... 10) then halt or ask for
confirmation.

In this case I'm writing something that is fully automated so doing
all those kind of checks is very important.

Best,
Tom Limoncelli

-- 
Email: t...@whatexit.orgWork: tlimonce...@stackoverflow.com
Skype: YesThatTom
Blog:  http://EverythingSysadmin.com

Deleting old archives is slow

[Tom Limoncelli]

Every couple months I expire a bunch of old archives with commands like...

tarsnap -d -f archivename-2014010101
tarsnap -d -f archivename-2014010102
tarsnap -d -f archivename-2014010103
...

Each deletion takes a minute or so.   Is there any way to speed this up?

I'd be willing to forego the statistics if I could just submit the
deletion request and not wait around for the completion.

Thanks!
Tom

-- 
Email: t...@whatexit.orgWork: tlimonce...@stackoverflow.com
Skype: YesThatTom
Blog:  http://EverythingSysadmin.com

Re: Deleting old archives is slow

[James Turner]

On Mon, Sep 28, 2015 at 10:14:38AM -0400, Tom Limoncelli wrote:
> Every couple months I expire a bunch of old archives with commands like...
> 
> tarsnap -d -f archivename-2014010101
> tarsnap -d -f archivename-2014010102
> tarsnap -d -f archivename-2014010103
> ...
> 
> Each deletion takes a minute or so.   Is there any way to speed this up?
> 

You can send one delete command for all the archives.

tarsnap -d -f archivename-2014010101 -f archivename-2014010102
-f archivename-2014010103

-- 
James Turner