Re: vtape, end of tape waste

2006-05-24 Thread Geert Uytterhoeven
On Tue, 23 May 2006, Ian Turner wrote:
 On Tuesday 23 May 2006 16:28, Jon LaBadie wrote:
  But running out of disk space caused me to look more
  closely at the situation and I realized that the failed
  taping is left on the disk.  This of course mimics what
  happens on physical tape.  However with the file:driver
  if this failed, and useless tape file were deleted,
  it would free up space for other data.
 
  Has anyone addressed this situation?

 There is no good short-term solution to this problem. Sorry. :-( Tape 
 spanning 
 helps, but is not a panacea.
 
 This is one of the limitations of the vtape API that I was talking about -- 
 it 
 tries to reimplement tape semantics on a filesystem, even when that doesn't 
 make sense.

[ Disclaimer: I haven't looked at the code yet ]

But I guess it can't be that difficult to call remove() on write error?
When close() is called later, it will be deleted.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say programmer or something like that.
-- Linus Torvalds


Re: vtape, end of tape waste

2006-05-23 Thread Ian Turner
Jon,

There is no good short-term solution to this problem. Sorry. :-( Tape spanning 
helps, but is not a panacea.

This is one of the limitations of the vtape API that I was talking about -- it 
tries to reimplement tape semantics on a filesystem, even when that doesn't 
make sense.

When the Device API is done, this problem will go away, though the combination 
of two new features: Partial device recycling means that you can delete only 
certain files from a virtual tape: This would include failed dumps, but might 
also include old dumps. The second new feature is appending to volumes, which 
I hope is self-explanatory.

With the combination of these two features, the future of disk-based storage 
is that you will have only one virtual tape (a VFS Volume), to which you 
constantly append data at the end and recycle data from the beginning. This 
makes a lot more sense in terms of provisioning space, as well as making use 
of provisioned space. Also, with only one volume, you no longer need the 
chg-disk changer, which simplifies setup quite a bit.

Besides the Device API bits, which are mostly only interesting to developers, 
there will need to be discussion in the community about the right way to 
do the user side of these features: In particular, partial volume recycling 
will require rethinking Amanda's retention policy scheme. One possibility 
(but not the only one) might be to talk about keeping a particular dumptype 
for a particular number of days. Now might be too soon, but at some point I'd 
like to discuss this with you in greater detail.

Cheers,

--Ian

On Tuesday 23 May 2006 16:28, Jon LaBadie wrote:
 I'm running vtapes on a new server.  The vtapes
 are split across two external disk drives.

 Realizing that some tapes would not fill completely,
 I decided that rather than define the tape size to be
 exactly disk/N, I would add a fudge factor to the size.

 Things worked exactly as anticipated.  The install has
 now reached beyond the last tape which ran out of disk
 space just before it ran our of the last tapes' length.
 And as usual, the taping restarted on the next vtape
 on the disk with remaining space.

 But running out of disk space caused me to look more
 closely at the situation and I realized that the failed
 taping is left on the disk.  This of course mimics what
 happens on physical tape.  However with the file:driver
 if this failed, and useless tape file were deleted,
 it would free up space for other data.

 Curious, I added up all the sizes of the failed, partially
 taped dumps.  They totalled 46+ GB.  That is substantially
 more than I dump daily.

 Has anyone addressed this situation?

 Before you ask, no I've not gone to tape-spanning (yet).

-- 
Forums for Amanda discussion: http://forums.zmanda.com/


Re: vtape, end of tape waste

2006-05-23 Thread Ross Vandegrift
On Tue, May 23, 2006 at 04:28:31PM -0400, Jon LaBadie wrote:
 But running out of disk space caused me to look more
 closely at the situation and I realized that the failed
 taping is left on the disk.  This of course mimics what
 happens on physical tape.  However with the file:driver
 if this failed, and useless tape file were deleted,
 it would free up space for other data.

Our setups avoid this situation by having a dedicated chunk of holding
space.  I cut out 150-200GiB on each Amanda server just for holding
space.  That way, no dump ever fails because the data in holding was
using space in the vtape filesystem.

I know throwing more hardware/disk space at the problem isn't a
particularly interesting or clever solution, but I can vouch for the
fact that it works!

-- 
Ross Vandegrift
[EMAIL PROTECTED]

The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell.
--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37