Re: holding disk question
On Mon, 19 Jun 2006 at 11:49am, Jon LaBadie wrote On Mon, Jun 19, 2006 at 04:05:05PM +0200, Cyrille Bollu wrote: I would like a confirmation: I'm using amanda to backup only local (SCSI) hard drives (about 1TB). In such a configuration are there any benefits from using a holding disk? I hope the answer is as easy as the question but :-/ parallelism of multiple DLE's dumping at same time. more consistant feeding of tape drive from a "single" disk file rather than at dump program rate. All of those benefits assume the I/O subsystem can handle the demands being placed on it. A 6-disk RAID5 is going to be *very* hard pressed to handle 2 read and 1 write thread all on the same volume simultaneously. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: holding disk question
On Mon, Jun 19, 2006 at 04:05:05PM +0200, Cyrille Bollu wrote: > Hi again, > > I would like a confirmation: > > I'm using amanda to backup only local (SCSI) hard drives (about 1TB). > > In such a configuration are there any benefits from using a holding disk? > > I hope the answer is as easy as the question but :-/ parallelism of multiple DLE's dumping at same time. more consistant feeding of tape drive from a "single" disk file rather than at dump program rate. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
holding disk question
Hi again, I would like a confirmation: I'm using amanda to backup only local (SCSI) hard drives (about 1TB). In such a configuration are there any benefits from using a holding disk? I hope the answer is as easy as the question but :-/ Cyrille Bollu Responsable système Fedasil - ICT tel: +32.2.213.43.49
Re: Holding Disk Question
On Thu, 15 Aug 2002, Gene Heskett wrote: > On Thursday 15 August 2002 14:49, John Koenig wrote: > >So does the chunksize parameter affect performance in any way? > > Only in that it prevents troubles with a filesystem that can't > handle large files. There was, at the time amanda was first > deployed, a 2gig limit to the filesize in many of its various > platforms filesystems. Many of those limits are now historical, > but you'd hate to find it out by doing a recovery and having it > blow up because a tape of 20gigs was all one big file. A 20 GB dump _is_ all one big file. Chunking is only implemented on the holding disk during dumping. The chunks are merged back into a single file when spooled from holding disk to tape. When you do a recovery what comes back is a single large file. If you want to recover the whole dump from tape to your holding disk before pulling out the files that need to be restored, then you'll have to split it up by hand in the process. -Mitch
Re: Holding Disk Question
On Thursday 15 August 2002 14:49, John Koenig wrote: >>My motto: Disk is cheap, don't skimp on holding disk. > >Yup... after I deployed a 181 GB drive for our holding disk, I saw >somewhat different behavior in amstatus... though I would have >expected the total time of the backups to drop (compared to a >relatively small 25 GB holding space I used before) they did > not... I conclude this was because the entire run is tape i/o > bound... Holding disk usage on the runs with this disk were about > 90% capacity... so the clients were not taking so long to > complete their data transfers... > >This would seem to indicate I need to double this amount of space > to hold 2 days of backups, currently... I'd like to have a week's > worth, actually in case the tape goes South and replacement > is not easy and quick. > >So does the chunksize parameter affect performance in any way? Only in that it prevents troubles with a filesystem that can't handle large files. There was, at the time amanda was first deployed, a 2gig limit to the filesize in many of its various platforms filesystems. Many of those limits are now historical, but you'd hate to find it out by doing a recovery and having it blow up because a tape of 20gigs was all one big file. >thx -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.11% setiathome rank, not too shabby for a WV hillbilly
Re: Holding Disk Question
On Thu, 15 Aug 2002, John Koenig wrote: > Yup... after I deployed a 181 GB drive for our holding disk, I saw > somewhat different behavior in amstatus... though I would have > expected the total time of the backups to drop (compared to a > relatively small 25 GB holding space I used before) they did not... I > conclude this was because the entire run is tape i/o bound... Holding > disk usage on the runs with this disk were about 90% capacity... so > the clients were not taking so long to complete their data > transfers... > > This would seem to indicate I need to double this amount of space to > hold 2 days of backups, currently... I'd like to have a week's worth, > actually in case the tape goes South and replacement is not easy > and quick. > > So does the chunksize parameter affect performance in any way? No. Taping of a dump spooled to holding disk does not begin until all chunks are received. The best way to figure out where your bottleneck and what configuration changes will give you the greatest return is is to run amplot. It will give you a graphical representation of what's happening when. Very revealing. -Mitch
Re: Holding Disk Question
On Thu, Aug 15, 2002 at 11:49:15AM -0700, John Koenig wrote: > > So does the chunksize parameter affect performance in any way? > Not to my knowledge. It is a patch for holding disks with limited file size maximums, often 2GB. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Holding Disk Question
>My motto: Disk is cheap, don't skimp on holding disk. > Yup... after I deployed a 181 GB drive for our holding disk, I saw somewhat different behavior in amstatus... though I would have expected the total time of the backups to drop (compared to a relatively small 25 GB holding space I used before) they did not... I conclude this was because the entire run is tape i/o bound... Holding disk usage on the runs with this disk were about 90% capacity... so the clients were not taking so long to complete their data transfers... This would seem to indicate I need to double this amount of space to hold 2 days of backups, currently... I'd like to have a week's worth, actually in case the tape goes South and replacement is not easy and quick. So does the chunksize parameter affect performance in any way? thx --
Re: Holding Disk Question
On Thu, Aug 15, 2002 at 10:35:18AM -0600, Scott Sanders wrote: > I'm sure there are several things to consider here but one thing that comes to > my mind is that your holding disk should be at least large enough to hold one > nights worth of backups and maybe a little more. That way in the event that > for some reason amanda can't dump everything to tape at least you have a copy > on disk that you can flush to tape after you resolve the problem with the tape > drive. > > Kevin Passey wrote: > > > What is the best size for a holding disk - is it a "how long is a piece > > string" or are there some optimal values. > > > > I have 2 servers I want to back up - the Amanda server 40gb RH Linux box - > > an WIN NT machine 16gb. > > Ditto's on Scott's comments plus with "reserve" set low enough to allow the scheduled level 0's to be dumped to holding disk. I'm in the enviable position of having over a week's worth of holding disk space. It was nice once when I forgot to change tape cartridges before a business trip. Had 6 amflush's to do to resyncronize. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Holding Disk Question
On Thursday 15 August 2002 12:19, Kevin Passey wrote: >What is the best size for a holding disk - is it a "how long is a > piece string" or are there some optimal values. > >I have 2 servers I want to back up - the Amanda server 40gb RH > Linux box - an WIN NT machine 16gb. Holding disk sizes are rather hard to pin down, and most of us seem to solve that problem by specifying a /path/to which puts it on a large partition, and then specify a reserve of a couple of gigs. amanda can backup to it while simultainously dumping from it, essentially useing it as a giant circular buffer with certain size limits stated pretty much up front. Those are: 1. No disklist entry should ever exceed the size of a single tape since amanda can't span to the next tape with a single operation. 2. If there are filesize limitations such as 2gigs on the server, then the chunksize will need to be sized less than that, otherwise its a never mind. 3. The reserved amount should be able to hold at least the incrementals of the whole system, a rather unpredictable value at best... I started out with about 27 gigs when I first set this machine up, but have eaten up several of those with source code, pictures from my new digital camera, and music from 'ogg'ing my cd collection, so I'm down to about 12 gigs which is still overkill on a 2 machine system with about 90 gigs worth of disk, not all of which is full of course. I *might* have 40 gigs worth of Junque or data, depending on how one defines the words. Amanda uses a 7 day dumpcycle here, with 20 tapes, and it normally fits a given nights activity onto one 4g tape. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.11% setiathome rank, not too shabby for a WV hillbilly
Re: Holding Disk Question
On Thu, 15 Aug 2002, Kevin Passey wrote: > What is the best size for a holding disk - is it a "how long is a piece > string" or are there some optimal values. It mostly a tuning thing that's up to you to decide what's best for you. That said, I prefer my setups to have as a minimum, enough to hold any single night's backup in its entirety. Then if something bad happens with the tape drive/media, the backups can run to completion spooled to the holding disk(s) and in the morning I can solve the tape problem and flush the holding disk(s) to tape then. Also, having enough to hold all of a single run allows you to take the greatest advantage of dumping parallelism, thus getting your client backups finished as quickly as possible. Further improvements on this theme are, 1) enough holding disk to hold an entire weekend's backups, and 2) enough holding disk to hold an entire . My motto: Disk is cheap, don't skimp on holding disk. -Mitch
Re: Holding Disk Question
I'm sure there are several things to consider here but one thing that comes to my mind is that your holding disk should be at least large enough to hold one nights worth of backups and maybe a little more. That way in the event that for some reason amanda can't dump everything to tape at least you have a copy on disk that you can flush to tape after you resolve the problem with the tape drive. Kevin Passey wrote: > What is the best size for a holding disk - is it a "how long is a piece > string" or are there some optimal values. > > I have 2 servers I want to back up - the Amanda server 40gb RH Linux box - > an WIN NT machine 16gb. > > TIA > > Kevin Passey
Holding Disk Question
What is the best size for a holding disk - is it a "how long is a piece string" or are there some optimal values. I have 2 servers I want to back up - the Amanda server 40gb RH Linux box - an WIN NT machine 16gb. TIA Kevin Passey
Re: Holding Disk Question
* John R. Jackson <[EMAIL PROTECTED]> (Fri, Apr 13, 2001 at 07:44:09AM -0500) >>Or is it more a convenience (as in this case where picking the right >>chunksize will have the load indeed split over multiple disks) > The original intent was to support file systems that cannot handle files > larger than 32 KBytes (e.g. ext2 on Linux). But, as with all good ideas larger than 2G I assume ;) > :-), it turns out to have some other nice benefits, such as what you're > trying to do. Well, it's all in the amanda.conf now, We'll see tonight how it goes Currently listening to: The Radio (http://www.arrow.nl/specials/GVGVplaylist.htm) Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- __O If your watch is wound, wound to run, it will =`\<, If your time is due, due to come, it will (=)/(=) Living this life, is like trying to learn latin in a chines firedrill
Re: Holding Disk Question
>does the chunksize have any performance benefiot/penalty ? No penalties I can think of at the 1 GByte level. If you went silly and told it 100 KBytes, there is an additional 32 KByte header on each chunk so you'd waste a tremendous amount of space, and I suppose at that small a size there might be some speed loss as Amanda was constantly doing the protocol and search for the next chunk. >Or is it more a convenience (as in this case where picking the right >chunksize will have the load indeed split over multiple disks) The original intent was to support file systems that cannot handle files larger than 32 KBytes (e.g. ext2 on Linux). But, as with all good ideas :-), it turns out to have some other nice benefits, such as what you're trying to do. When tape chunking is done, it will also allow disk chunks that have made it to tape to be released early (before either the dump or the tape is completely done), which will reduce the amount of holding disk space needed and also allow everything to flow through the holding disk, not just the complete images that will (were estimated to) fit, which will increase parallelism. > Gerhard John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Holding Disk Question
* John R. Jackson <[EMAIL PROTECTED]> (Fri, Apr 13, 2001 at 07:06:20AM -0500) >>Will that work ? > No. I'm planning on getting a job in sales, so I'm just making all > this up. :-) Good ;) >>If I specify a chunksize of 1G, and dump a 9G backup, will it stick 3 >>chunks on each of the 3 hilding disks ? > Not necessarily. Note that it takes both the number of active dumpers > (how busy) each holding disk has and the amount of free space. So there > are dynamics involved that cannot be known until runtime. But based on > that one comment and a quick glance at the code, I would expect it to > do more or less "the right thing". Good. BTW, does the chunksize have any performance benefiot/penalty ? Or is it more a convenience (as in this case where picking the right chunksize will have the load indeed split over multiple disks) >>That would be really impressive ... > It's Amanda. What did you expect? :-) Miracles Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- __O Some say the end is near. =`\<, Some say we'll see armageddon soon (=)/(=) I certainly hope we will I could use a vacation
Re: Holding Disk Question
>It's in the comments in the example amanda.conf. OK, I'll get those updated. >Will that work ? No. I'm planning on getting a job in sales, so I'm just making all this up. :-) >If I specify a chunksize of 1G, and dump a 9G backup, will it stick 3 >chunks on each of the 3 hilding disks ? Not necessarily. Note that it takes both the number of active dumpers (how busy) each holding disk has and the amount of free space. So there are dynamics involved that cannot be known until runtime. But based on that one comment and a quick glance at the code, I would expect it to do more or less "the right thing". >That would be really impressive ... It's Amanda. What did you expect? :-) > Gerhard John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Holding Disk Question
* John R. Jackson <[EMAIL PROTECTED]> (Thu, Apr 12, 2001 at 11:28:03AM -0500) >>Im currently using a single (big) holding disk. >>I have 2 smaller disks, that I'd like to use as holding disks. >> >>But the samller disks will not be able to hold the big 0 dumps. >> >>According to the docs, the holding disks are used in a round-robin way >>(dump 1 to HD1, dump 2 -> HD 2, dump3 -> HD3, dump4 -> HD1 &c &c ) > According to the code (you actually read and believed the docs? :-): Oddly enough, yes ;) It's in the comments in the example amanda.conf. > /* find the holdingdisk with the fewest active dumpers and among >* those the one with the biggest free space > So you should not have any trouble using all three as a "pool". > It might work even better if you set the holding disk chunksize down > so they can be split up into pieces and scattered around. Will that work ? If I specify a chunksize of 1G, and dump a 9G backup, will it stick 3 chunks on each of the 3 hilding disks ? That would be really impressive ... > What "docs" did you find the round-robin described? I seem to remember > seeing that, too, but can't find it now and would like to get it updated. Line 82 and on in example/amanda.conf in the source tree # Specify holding disks. These are used as a temporary staging area for # dumps before they are written to tape and are recommended for most sites. # The advantages include: tape drive is more likely to operate in streaming # mode (which reduces tape and drive wear, reduces total dump time); multiple # dumps can be done in parallel (which can dramatically reduce total dump time. # The main disadvantage is that dumps on the holding disk need to be flushed # (with amflush) to tape after an operating system crash or a tape failure. # If no holding disks are specified then all dumps will be written directly # to tape. If a dump is too big to fit on the holding disk than it will be # written directly to tape. If more than one holding disk is specified then # they will all be used round-robin. ^ Thanks, > >> Gerhard > > John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED] > Currently listening to: Pearl Jam - 11 Off He Goes (VA Beach 8_3_00) Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- __O i hurt myself today to see if i still feel =`\<, i focus on the pain the only thing that's real (=)/(=) the needle tears a hole the old familiar sting try to kill it all away but i remember everything
Re: Holding Disk Question
>Im currently using a single (big) holding disk. >I have 2 smaller disks, that I'd like to use as holding disks. > >But the samller disks will not be able to hold the big 0 dumps. > >According to the docs, the holding disks are used in a round-robin way >(dump 1 to HD1, dump 2 -> HD 2, dump3 -> HD3, dump4 -> HD1 &c &c ) According to the code (you actually read and believed the docs? :-): /* find the holdingdisk with the fewest active dumpers and among * those the one with the biggest free space So you should not have any trouble using all three as a "pool". It might work even better if you set the holding disk chunksize down so they can be split up into pieces and scattered around. What "docs" did you find the round-robin described? I seem to remember seeing that, too, but can't find it now and would like to get it updated. > Gerhard John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Holding Disk Question
* Alexandre Oliva <[EMAIL PROTECTED]> (Thu, Apr 12, 2001 at 07:45:31AM -0300) >> What happens if my dump is supposed to go to HD1, but is way too big for >> HD1, but would easily fit on HD3 ? >> Will amanda use the big Holding disk ? > I suppose so. Now that's what I call a honest and fair answer ;) Kind regards, -- Gerhard den Hollander Phone +31-10.280.1515 Technical Support Jason Geosystems BV Fax +31-10.280.1511 (When calling please note: we are in GMT+1) [EMAIL PROTECTED] POBox 1573 visit us at http://www.jasongeo.com 3000 BN Rotterdam JASON...#1 in Reservoir CharacterizationThe Netherlands This e-mail and any attachment is/are intended solely for the named addressee(s) and may contain information that is confidential and privileged. If you are not the intended recipient, we request that you do not disseminate, forward, distribute or copy this e-mail message. If you have received this e-mail message in error, please notify us immediately by telephone and destroy the original message.
Re: Holding Disk Question
On Apr 12, 2001, Gerhard den Hollander <[EMAIL PROTECTED]> wrote:
> What happens if my dump is supposed to go to HD1, but is way too big for
> HD1, but would easily fit on HD3 ?
> Will amanda use the big Holding disk ?
I suppose so.
--
Alexandre Oliva Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicampoliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist*Please* write to mailing lists, not to me
Re: Holding Disk Question
raid 0 them? Steven Gerhard den Hollander wrote: > Im currently using a single (big) holding disk. > I have 2 smaller disks, that I'd like to use as holding disks. > > But the samller disks will not be able to hold the big 0 dumps. > > According to the docs, the holding disks are used in a round-robin way > (dump 1 to HD1, dump 2 -> HD 2, dump3 -> HD3, dump4 -> HD1 &c &c ) > > What happens if my dump is supposed to go to HD1, but is way too big for > HD1, but would easily fit on HD3 ? > > Will amanda use the big Holding disk ? > > Currently listening to: Flying Frog Brigade - Here's to the Man (August 29 - New >York City) > > Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == > -- >__O Some say the end is near. > =`\<, Some say we'll see armageddon soon > (=)/(=) I certainly hope we will > I could use a vacation
Holding Disk Question
Im currently using a single (big) holding disk. I have 2 smaller disks, that I'd like to use as holding disks. But the samller disks will not be able to hold the big 0 dumps. According to the docs, the holding disks are used in a round-robin way (dump 1 to HD1, dump 2 -> HD 2, dump3 -> HD3, dump4 -> HD1 &c &c ) What happens if my dump is supposed to go to HD1, but is way too big for HD1, but would easily fit on HD3 ? Will amanda use the big Holding disk ? Currently listening to: Flying Frog Brigade - Here's to the Man (August 29 - New York City) Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- __O Some say the end is near. =`\<, Some say we'll see armageddon soon (=)/(=) I certainly hope we will I could use a vacation
