Re: Best size for vtapes

[Olivier]

Jon LaBadie  writes:

> Do you really expect all (or most) of your vtapes to be 100% full?  If so,
> I do not think you have allocated enough total space.
>
> Amanda has one provision for dealing with such situations, the holding
> disk.  Mine is dedicated, and about the size of four vtapes.
>
> Another is "runtapes".  Oh, or do you plan to run exactly the number of
> vtapes that you need for your chosen dumpcycle?

Of course not. runtapes can be as big as needed, provided that vtapes
are small enough.

>> So I prefer to stick with the amount of vtapes equal to the real amount
>> of disk space.
>
> Then, from my experience, you will be leaving about 1/3 of your disk empty.

/dev/ada3p1  2.6T2.2T234G91%/automnt/ada3
/dev/ada4p1  2.6T2.3T121G95%/automnt/ada4
/dev/ada2p1  2.6T2.4T8.5G   100%/automnt/ada2
/dev/ada5p1  2.6T2.4T 60G98%/automnt/ada5
/dev/ada1p1  2.6T2.0T458G82%/automnt/ada1
/dev/ada6p1  2.6T2.3T 86G97%/automnt/ada6

Disks are 3TB with 27 vtapes in each disk an 10GB per chunk. I am
surprised at the 100% utilisation reported for ada2, even if it is
"system utilisation" with some decent amount of space left.

> Is your backup size really even pseudo random?  Mine, over 40+ years,
> at many sites, have never been.

As I said, this is a theoretical exercise. The largest chuck of data
that I backup every day is the users files, that can vary a lot when you
have 200+ users.

> That is based on the assumption that your tapes match the available space
> and your runtapes is 1.  Neither of which ?we?/I recommend.  BTW I just
> peeked, my disks dedicated to vtapes, even though substantially over-
> subscribed are between 79% and 89% full.

Runtapes could be bigger than needed, Amanda will use only what it needs.

> First, though an unused inode would be allocated, no inodes would be
> wasted.  When you create your file system (assuming extX, ???) space
> to a set number of inodes is created.

OK, my mistake for mentioning inodes. I would have to review how
filesystems (ufs, FreeBSD) work. If creating a directpry does not
consume any disk space, then there is no penalty for having a multitude
of small vtapes.

> Second, disks have many millions, even billions of data blocks.  Are
> you really worried about using another 1 or 3 for a directory?  You
> must have more important thing with which to be concerned.
>
> One last thing, when you create your file system(s) for vtapes you may
> be able to control how many inodes are created.  Remember each file
> takes only one inode.  A 3TB disk of vtapes on my system only has a
> total of 947 files.

I have the same number of files per disk.

> Yet there were 350,000 inodes created even though
> I changed the mkfs options to greatly reduce them.  Another disk where
> I forgot to reduce the number of inodes created has 190,000,000 inodes.
>
> So I'm "wasting" about 20,000,000 data blocks as inodes.  Not enough
> for a 100GB vtape, but enough for four 5GB chunks.

I will have to remember twicking the filesystem the next time I change
the disks, because with the default, I am wasting over 350 million
inodes per disk.

Thank you,

Olivier

-- 

Re: Best size for vtapes

[Jon LaBadie]

On Mon, Nov 18, 2019 at 02:48:40PM +0700, Olivier wrote:
> Charles Curley  writes:
> 
> > So in theory you could allocate more vtape space that there is room on
> > the partition. Just make sure you never use more vtape space than you
> > actually have.
> 
> In theory. But you are accepting the risk that from time to time all
> your vtapes will be 100% full and you will not have enough space on your
> disk. And I don't think there is any provision in Amanda to prevent
> that.

Do you really expect all (or most) of your vtapes to be 100% full?  If so,
I do not think you have allocated enough total space.

Amanda has one provision for dealing with such situations, the holding
disk.  Mine is dedicated, and about the size of four vtapes.

Another is "runtapes".  Oh, or do you plan to run exactly the number of
vtapes that you need for your chosen dumpcycle?

> 
> So I prefer to stick with the amount of vtapes equal to the real amount
> of disk space.

Then, from my experience, you will be leaving about 1/3 of your disk empty.

> > The answer to that is, "that depends". I have tried to have a vtape
> > size a bit larger than my typical daily backup, and then allow amanda
> > to use enough extra tapes to cover the largest likely backup. So most
> > days I use one vtape, 40-90% filled. Some days I use three or four
> > vtapes. All but the last are almost 100% filled. You can also play
> > with your split size.

My runs mirror Charles' exactly.  I also configure my DLEs' dumpcycles
according to the data.  Unchanging data (music, photos) get 1 month
cycles, other stuff 1 or 2 weeks.

> I don't think that depends at all unless you have a very deterministic
> usage patern. When the size of the daily backup is truly random, it
> becomes a purely mathematical problem:

Is your backup size really even pseudo random?  Mine, over 40+ years,
at many sites, have never been.

> Each day, you are wasting on average 1/2 vtape amount of disk. So you
> could have vtape being half the size of what you are using, wasting 1/4
> of the initial amount, ...

That is based on the assumption that your tapes match the available space
and your runtapes is 1.  Neither of which ?we?/I recommend.  BTW I just
peeked, my disks dedicated to vtapes, even though substantially over-
subscribed are between 79% and 89% full.

> ... but then you are wasting x blocks overheard for declaring new
> directories and using more inodes. When do both fucntions cross?

First, though an unused inode would be allocated, no inodes would be
wasted.  When you create your file system (assuming extX, ???) space
to a set number of inodes is created.

Second, disks have many millions, even billions of data blocks.  Are
you really worried about using another 1 or 3 for a directory?  You
must have more important thing with which to be concerned.

One last thing, when you create your file system(s) for vtapes you may
be able to control how many inodes are created.  Remember each file
takes only one inode.  A 3TB disk of vtapes on my system only has a
total of 947 files.  Yet there were 350,000 inodes created even though
I changed the mkfs options to greatly reduce them.  Another disk where
I forgot to reduce the number of inodes created has 190,000,000 inodes.

So I'm "wasting" about 20,000,000 data blocks as inodes.  Not enough
for a 100GB vtape, but enough for four 5GB chunks.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Best size for vtapes

[Gene Heskett]

On Monday 18 November 2019 02:11:07 Charles Curley wrote:

> On Mon, 18 Nov 2019 13:13:00 +0700
>
> Olivier  wrote:
> > I am wondering what is the best size to declare for the lenght of
> > the vtapes. Too long a vtape could lead to disk space being unused,
>
> You may have a misconception here. Vtapes emulate physical tapes, but
> not in all respects. If you don't fill up a tape, that unused space is
> gone. But if you don't fill up a vtape, it simply means there is that
> much space available for other files, such as other vtapes.
>
> So in theory you could allocate more vtape space that there is room on
> the partition. Just make sure you never use more vtape space than you
> actually have.
>
Track that usage with an occasional df. And I'm glad you asked as I see I 
am about to have a problem. Using a 2T drive with 60 vtapes on it, I am 
very close to full. As in I've never been this close before.

 /dev/sdd1  1921802432 1802889848   21220528  99% /amandatapes

So I need to add 2 days to the cycle, making the average backup a little 
smaller. Done. So far, amanda has not given me any media full warnings. 
Trouble till now has been avoided because one of the 5 machines being 
backed up here has been changed out, and the new one has not been 
re-added to the schedule.  So I'll look in 2 or 3 weeks or so to see if 
theres room for that 5th machine. Its an rpi4, but it has around 400 GB 
of SSD's attached too. Otherwise it may be time to go get a 4TB drive.

> > but
> > having too many small vtapes one or two directories created for each
> > new vtape, and if many vtapes, the main directory containing them
> > would not fit in a single inode; but the space left unused would be
> > smaller.
> >
> > My understanding is that splitting a backup does not consume more
> > space, so it is not the backup data that gets bigger, only what is
> > surrounding it.
> >
> > If wee are not considering the overhead of computing the tape
> > spliting, nor the possible extra I/O, if we only focus on the disk
> > space, what would be the optimum size for the vtapes?
>
> The answer to that is, "that depends". I have tried to have a vtape
> size a bit larger than my typical daily backup, and then allow amanda
> to use enough extra tapes to cover the largest likely backup. So most
> days I use one vtape, 40-90% filled. Some days I use three or four
> vtapes. All but the last are almost 100% filled. You can also play
> with your split size.
>
> I suggest you do the rough calculations to get your initial setup,
> then fiddle with it as you gain experience and as amanda settles its
> calculations down over time.

And adding 2 days to the cycle might not be enough, but I am not going to 
further disturb amanda until I have recovered from a aortic heart valve 
replacement in mid December. I don't recommend gettong old, its a PITA. 
But its also fun since the reaper has come for me twice now, and blinked 
both times. :)


Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Best size for vtapes

[Olivier]

Charles Curley  writes:

> So in theory you could allocate more vtape space that there is room on
> the partition. Just make sure you never use more vtape space than you
> actually have.

In theory. But you are accepting the risk that from time to time all
your vtapes will be 100% full and you will not have enough space on your
disk. And I don't think there is any provision in Amanda to prevent
that.

So I prefer to stick with the amount of vtapes equal to the real amount
of disk space.

> The answer to that is, "that depends". I have tried to have a vtape
> size a bit larger than my typical daily backup, and then allow amanda
> to use enough extra tapes to cover the largest likely backup. So most
> days I use one vtape, 40-90% filled. Some days I use three or four
> vtapes. All but the last are almost 100% filled. You can also play
> with your split size.

I don't think that depends at all unless you have a very deterministic
usage patern. When the size of the daily backup is truly random, it
becomes a purely mathematical problem:

Each day, you are wasting on average 1/2 vtape amount of disk. So you
could have vtape being half the size of what you are using, wasting 1/4
of the initial amount, but then you are wasting x blocks overheard for
declaring new directories and using more inodes. When do both fucntions
cross?

Best regards,

Olivier
-- 

Re: Best size for vtapes

[Charles Curley]

On Mon, 18 Nov 2019 13:13:00 +0700
Olivier  wrote:

> I am wondering what is the best size to declare for the lenght of the
> vtapes. Too long a vtape could lead to disk space being unused,

You may have a misconception here. Vtapes emulate physical tapes, but
not in all respects. If you don't fill up a tape, that unused space is
gone. But if you don't fill up a vtape, it simply means there is that
much space available for other files, such as other vtapes.

So in theory you could allocate more vtape space that there is room on
the partition. Just make sure you never use more vtape space than you
actually have.

> but
> having too many small vtapes one or two directories created for each
> new vtape, and if many vtapes, the main directory containing them
> would not fit in a single inode; but the space left unused would be
> smaller.
> 
> My understanding is that splitting a backup does not consume more
> space, so it is not the backup data that gets bigger, only what is
> surrounding it.
> 
> If wee are not considering the overhead of computing the tape
> spliting, nor the possible extra I/O, if we only focus on the disk
> space, what would be the optimum size for the vtapes?

The answer to that is, "that depends". I have tried to have a vtape
size a bit larger than my typical daily backup, and then allow amanda
to use enough extra tapes to cover the largest likely backup. So most
days I use one vtape, 40-90% filled. Some days I use three or four
vtapes. All but the last are almost 100% filled. You can also play
with your split size.

I suggest you do the rough calculations to get your initial setup, then
fiddle with it as you gain experience and as amanda settles its
calculations down over time.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Best size for vtapes

[Olivier]

Hello,

I apologize in advance if the question has already been answered or has
become obsolete.

I am wondering what is the best size to declare for the lenght of the
vtapes. Too long a vtape could lead to disk space being unused, but
having too many small vtapes one or two directories created for each new
vtape, and if many vtapes, the main directory containing them would not
fit in a single inode; but the space left unused would be smaller.

My understanding is that splitting a backup does not consume more space,
so it is not the backup data that gets bigger, only what is surrounding
it.

If wee are not considering the overhead of computing the tape spliting,
nor the possible extra I/O, if we only focus on the disk space, what
would be the optimum size for the vtapes?

Best regards,

Olivier
--