Re: suggestion for a disk-to-disk backup server
On 2007-07-06 04:28, Olivier Nicole wrote: An idea for the next release of Amanda: Add one config parameter that says use the device povided by the tape changer and that default to YES. So it would not break any existing customized changer script. When set to NO, use the device explicitely defined by tapedev. I frequently use chg-multi, with 2 physical drives, or with several vtape-devices. Which the the device in that case? -- Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: suggestion for a disk-to-disk backup server
Dustin J. Mitchell wrote: Folks -- I looked into this '-o' question, and have some answers for you. On Mon, Jul 02, 2007 at 03:44:55PM -0400, Chris Hoogendyk wrote: Jon LaBadie wrote: On Mon, Jul 02, 2007 at 01:48:59PM -0400, Chris Hoogendyk wrote: I checked the documentation man pages and the documentation of the -o option is pretty sparse. All references say see the configuration override section in amanda(8). That section is only a few lines long and makes no reference to the syntax with a comma separating multiple parameter=value pairs. So, possibilities ... 1. The syntax is wrong? And I need to use amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n? (In either case, this should be documented, since one would typically want to override more than one parameter if any at all.) The documentation could use some help there, for sure. FWIW, it doesn't seem that comma-separated options are supported, from my look at the source. The basic problem is that, since you're using a changer, tapedev is actually supplied by the *changer*, which looks directly in amanda.conf. There's no provision to hand configuration overrides (-o) to changers. Furthermore, many changers (e.g., chg-multi) supply their own tapedev. The best solution, in this case, is to completely disable the changer, too: amdump -o tapedev=/no/such/dev -o tpchanger= I hope that helps. I'll work on the documentation. Thanks, Dustin. I set that up in crontab yesterday and it ran this morning. Worked. 45 0 * * 2-6 /usr/local/sbin/amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n -o tpchanger= Now, I'll change that entry back so it doesn't do it on weekdays. Anyway, it gave me a tape error at the top of the report and then fell back to incrementals on the holding disk -- mostly level 1, but some level 2. The one thing that is annoying is that on /var/mail an incremental is essentially the same as a full. Those mail files (one per account) are just like a database file -- add or remove one message and you have to back up the whole file. It would be interesting to write a wrapper to do incremental mail dumps sort of like the sql database programs that do incremental dumps. I wonder if there would be any demand for such a thing. It could get a little messy, and would probably require the Application API. In our case, we have about 1200 accounts/files. It would require reading headers and doing proper locking so as not to collide with the mail programs. - Comment Question on the need for the changer script to tell amdump what is in amanda.conf (thus overriding the override): This behavior seems rather contrary. I can see that it is reasonable to override tpchanger as well as tapedev to provide a sort of clean slate. I can also see the possibility of some thrashing if amanda tries to read a tape, fails, and asks the changer to bring up the next tape. But reversing an override is just weird behavior. I'm guessing that for those changers where it's all tied up in one scsi address, and/or where there are multiple drives in a library, it might be desirable to have changer handle it. Then my case becomes the special case -- the changer is set up in sgen.conf and has a scsi device address, and the tape drive is set up in st.conf and is /dev/rmt/1 -- so the changer does not know how the drive got set up. This special case might require the changer to look somewhere else, e.g. amanda.conf, to find out. But, then, couldn't it also tell somehow if there had been an override? Or does that get too messy? Ok, maybe we just need to document that when someone overrides tapedev they should also override tpchanger if they have one. --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst [EMAIL PROTECTED] --- Erdös 4
Re: suggestion for a disk-to-disk backup server
On Thu, Jul 05, 2007 at 10:35:21AM -0400, Chris Hoogendyk wrote: The one thing that is annoying is that on /var/mail an incremental is essentially the same as a full. Those mail files (one per account) are just like a database file -- add or remove one message and you have to back up the whole file. It would be interesting to write a wrapper to do incremental mail dumps sort of like the sql database programs that do incremental dumps. I wonder if there would be any demand for such a thing. It could get a little messy, and would probably require the Application API. In our case, we have about 1200 accounts/files. It would require reading headers and doing proper locking so as not to collide with the mail programs. Yes, I think there would be demand for that (there are a lot of application-specific things that could use better backups). And yes, it will be a lot easier after the Application API is in place :) I'm guessing that for those changers where it's all tied up in one scsi address, and/or where there are multiple drives in a library, it might be desirable to have changer handle it. Then my case becomes the special case -- the changer is set up in sgen.conf and has a scsi device address, and the tape drive is set up in st.conf and is /dev/rmt/1 -- so the changer does not know how the drive got set up. This special case might require the changer to look somewhere else, e.g. amanda.conf, to find out. But, then, couldn't it also tell somehow if there had been an override? Or does that get too messy? That's basically the answer -- the current changer interface has no provision for passing along overrides, and adding one would potentially break a lot of custom changer scripts. You've hit the nail on the head as to why changers provide devices to Amanda, rather than leaving her to use tapedev. Ok, maybe we just need to document that when someone overrides tapedev they should also override tpchanger if they have one. See the attached patch. Dustin -- Dustin J. Mitchell Storage Software Engineer, Zmanda, Inc. http://www.zmanda.com/ Index: man/xml-source/amanda.8.xml === --- man/xml-source/amanda.8.xml (revision 423) +++ man/xml-source/amanda.8.xml (working copy) @@ -1221,15 +1221,36 @@ /refsect1 refsect1titleCONFIGURATION OVERRIDE/title -paraMost command allow to overwrite any configuration parameter on -the command line with the -o option./para -para-o NAME=value/para -paraeg. -o runtapes=2/para -paraeg. -o DUMPTYPE:no-compress:compress=server fast/para -paraeg. -o TAPETYPE:HP-DAT:length=2000m/para -paraeg. -o INTERFACE:local:use=2000 kbps/para +paraMost commands allow the override of specific +configuration options on the command line, using the arg +choice=plain-o/arg option. This option has the form arg +choice=plain-o/argreplaceablename/replaceable=replaceablevalue/replaceable. +An optional space is allowed after the arg choice=plain-o/arg. +Each configuration option should be specified in a separate +command-line option./para +paraFor global options, replaceablename/replaceable is simply the name of the option, e.g., +programlisting +amdump -oruntapes=2 +/programlisting +For options in a named section of the configuration, replaceablename/replaceable has the +form replaceableSECTION/replaceable:replaceablesection_name/replaceable:replaceablename/replaceable, where replaceableSECTION/replaceable is one of TAPETYPE, DUMPTYPE, HOLDINGDISK, or INTERFACE, and replaceablesection_name/replaceable is the name of the tapetype, dumptype, holdingdisk, or interface. Examples: +programlisting +amdump -o TAPETYPE:HP-DAT:length=2000m +amdump -o DUMPTYPE:no-compress:compress=server fast +amdump -o HOLDINGDISK:hd1:use=-100 mb +amdump -o INTERFACE:local:use=2000 kbps +/programlisting +/para +paraNote that configuration overrides are not effective for tape +changers, which supply a tapedev based on their own configuration. In order to +override emphasis remap=Itapedev/emphasis, you must also disable any changer: +programlisting +amdump -otapedev=/dev/nst1 -otpchanger='' +/programlisting +/para + /refsect1 refsect1titleAUTHOR/title
Re: suggestion for a disk-to-disk backup server
The one thing that is annoying is that on /var/mail an incremental is essentially the same as a full. Those mail files (one per account) are just like a database file -- add or remove one message and you have to back up the whole file. It would be interesting to write a wrapper to do incremental mail dumps sort of like the sql database programs that do incremental dumps. I wonder if there would be any demand for such a thing. It could get a little messy, and would probably require the Application API. In our case, we have about 1200 accounts/files. It would require reading headers and doing proper locking so as not to collide with the mail programs. Another advantage of maildir format over mailbox :) Only one message per file and no need for locking :) OK, I am still using mailbox, but I will change that :) I'm guessing that for those changers where it's all tied up in one scsi address, and/or where there are multiple drives in a library, it might be desirable to have changer handle it. Then my case becomes the special case -- the changer is set up in sgen.conf and has a scsi device address, and the tape drive is set up in st.conf and is /dev/rmt/1 -- so the changer does not know how the drive got set up. This special case might require the changer to look somewhere else, e.g. amanda.conf, to find out. But, then, couldn't it also tell somehow if there had been an override? Or does that get too messy? That's basically the answer -- the current changer interface has no provision for passing along overrides, and adding one would potentially break a lot of custom changer scripts. You've hit the nail on the head as to why changers provide devices to Amanda, rather than leaving her to use tapedev. An idea for the next release of Amanda: Add one config parameter that says use the device povided by the tape changer and that default to YES. So it would not break any existing customized changer script. When set to NO, use the device explicitely defined by tapedev. No need to change any script, all the changes are inside Amanda only. Best regards, Olivier
Re: suggestion for a disk-to-disk backup server
Folks -- I looked into this '-o' question, and have some answers for you. On Mon, Jul 02, 2007 at 03:44:55PM -0400, Chris Hoogendyk wrote: Jon LaBadie wrote: On Mon, Jul 02, 2007 at 01:48:59PM -0400, Chris Hoogendyk wrote: I checked the documentation man pages and the documentation of the -o option is pretty sparse. All references say see the configuration override section in amanda(8). That section is only a few lines long and makes no reference to the syntax with a comma separating multiple parameter=value pairs. So, possibilities ... 1. The syntax is wrong? And I need to use amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n? (In either case, this should be documented, since one would typically want to override more than one parameter if any at all.) The documentation could use some help there, for sure. FWIW, it doesn't seem that comma-separated options are supported, from my look at the source. The basic problem is that, since you're using a changer, tapedev is actually supplied by the *changer*, which looks directly in amanda.conf. There's no provision to hand configuration overrides (-o) to changers. Furthermore, many changers (e.g., chg-multi) supply their own tapedev. The best solution, in this case, is to completely disable the changer, too: amdump -o tapedev=/no/such/dev -o tpchanger= I hope that helps. I'll work on the documentation. Dustin -- Dustin J. Mitchell Storage Software Engineer, Zmanda, Inc. http://www.zmanda.com/
Re: suggestion for a disk-to-disk backup server
Paul Bijnens wrote: Instead of making it a different configuration, and running into trouble on which full to base an incrmeental, why don't you run an additional amdump of the normal config in the weekend, but which has options to override the config: amdump daily -o reserver=100,tapedev=/no/such/tape With the option tapedev /no/such/tape we force Amanda to fall back to degraded mode, making backup to holdingdisk only, and with the reserve 100, we avoid wasting holdingdisk space for full backups in the degraded mode. ok, tried this over the preceding weekend. the amanda user's crontab has these entries: 0 15 * * 1-5 /usr/local/sbin/amcheck -m daily 45 0 * * 2-6 /usr/local/sbin/amdump daily 45 0 * * 0-1 /usr/local/sbin/amdump daily -o reserve=100,tapedev=/dev/rmt/0n however, the weekend runs looked exactly like the weekday runs. They used an AIT tape off /dev/rmt/1n and ran fulls and incrementals intermixed. I searched the report, the log files, the debug files, etc. for any error messages about the command line options or any references to /dev/rmt/0n. none. I even tried # find /tmp/amanda -type f | xargs grep '/dev/rmt/0' and got nothing except the alternative weekend configuration I had played with the previous weekend. I checked the documentation man pages and the documentation of the -o option is pretty sparse. All references say see the configuration override section in amanda(8). That section is only a few lines long and makes no reference to the syntax with a comma separating multiple parameter=value pairs. So, possibilities ... 1. The syntax is wrong? And I need to use amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n? (In either case, this should be documented, since one would typically want to override more than one parameter if any at all.) 2. There is a bug in amanda, and it doesn't work? 3. Amanda is being too smart and reverting to /dev/rmt/1n when my override to /dev/rmt/0n fails? Doesn't seem likely. 4. There are additional parameters in my configuration that end up overriding my attempt to override tapedev? I am using a tpchanger, but there is no reference to tapedev there. If I search the daily configuration directory for reference to /dev/rmt, the only reference I find is the 'tapedev /dev/rmt/1n' in amanda.conf. In case it matters got lost in the beginnings of this thread, I'm running 2.5.1p3 on Solaris 9 on SPARC (E250). --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst [EMAIL PROTECTED] --- Erdös 4
Re: suggestion for a disk-to-disk backup server
On Mon, Jul 02, 2007 at 01:48:59PM -0400, Chris Hoogendyk wrote: Paul Bijnens wrote: Instead of making it a different configuration, and running into trouble on which full to base an incrmeental, why don't you run an additional amdump of the normal config in the weekend, but which has options to override the config: amdump daily -o reserver=100,tapedev=/no/such/tape With the option tapedev /no/such/tape we force Amanda to fall back to degraded mode, making backup to holdingdisk only, and with the reserve 100, we avoid wasting holdingdisk space for full backups in the degraded mode. ok, tried this over the preceding weekend. the amanda user's crontab has these entries: 0 15 * * 1-5 /usr/local/sbin/amcheck -m daily 45 0 * * 2-6 /usr/local/sbin/amdump daily 45 0 * * 0-1 /usr/local/sbin/amdump daily -o reserve=100,tapedev=/dev/rmt/0n however, the weekend runs looked exactly like the weekday runs. They used an AIT tape off /dev/rmt/1n and ran fulls and incrementals intermixed. I searched the report, the log files, the debug files, etc. for any error messages about the command line options or any references to /dev/rmt/0n. none. I even tried # find /tmp/amanda -type f | xargs grep '/dev/rmt/0' and got nothing except the alternative weekend configuration I had played with the previous weekend. I checked the documentation man pages and the documentation of the -o option is pretty sparse. All references say see the configuration override section in amanda(8). That section is only a few lines long and makes no reference to the syntax with a comma separating multiple parameter=value pairs. So, possibilities ... 1. The syntax is wrong? And I need to use amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n? (In either case, this should be documented, since one would typically want to override more than one parameter if any at all.) 2. There is a bug in amanda, and it doesn't work? 3. Amanda is being too smart and reverting to /dev/rmt/1n when my override to /dev/rmt/0n fails? Doesn't seem likely. 4. There are additional parameters in my configuration that end up overriding my attempt to override tapedev? Maybe an additional possibility: 5. /dev/rmt/0n == /dev/rmt/1n I think by tapedev=/no/such/tape Paul meant that literally, or at least some string that could not be a legit tape device. Two ways I can see the 0 and 1 devices being equivalent. My HP-DDS changer shows up in Solaris 9 x86 as two /dev/rmt devices, 0 and 1 for me. They are the same SCSI ID but different LUNs (the d# in c5t3d0 scsi dev nomenclature). One is for the changer mechanism and the other for the tape drive. At least that is what it is supposed to be. Only one can be used for the changer, but either seems to work for the tape drive. I don't think that is common, but who knows. The other way is if you installed the tape drive multiple times. /dev/rmt/# are not cleared at boot to allow for persistant names for the same device. For example, you could have two tape drives and only one powered up at boot time. Which was rmt/0 would vary. This is handled by recording the /devices data in /etc/path_to_inst and is only updated by a reconfigure reboot or options to the /usr/sbin/devfsadm command. Maybe there is a stale entry in there that makes rmt/0 and rmt/1 the same. You might also check what the symlinks in /dev/rmt point to with an ls -l and what the targets in /devices have for maj/min device numbers with ls -lL of /dev/rmt. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: suggestion for a disk-to-disk backup server
Jon LaBadie wrote: On Mon, Jul 02, 2007 at 01:48:59PM -0400, Chris Hoogendyk wrote: Paul Bijnens wrote: Instead of making it a different configuration, and running into trouble on which full to base an incrmeental, why don't you run an additional amdump of the normal config in the weekend, but which has options to override the config: amdump daily -o reserver=100,tapedev=/no/such/tape With the option tapedev /no/such/tape we force Amanda to fall back to degraded mode, making backup to holdingdisk only, and with the reserve 100, we avoid wasting holdingdisk space for full backups in the degraded mode. ok, tried this over the preceding weekend. the amanda user's crontab has these entries: 0 15 * * 1-5 /usr/local/sbin/amcheck -m daily 45 0 * * 2-6 /usr/local/sbin/amdump daily 45 0 * * 0-1 /usr/local/sbin/amdump daily -o reserve=100,tapedev=/dev/rmt/0n however, the weekend runs looked exactly like the weekday runs. They used an AIT tape off /dev/rmt/1n and ran fulls and incrementals intermixed. I searched the report, the log files, the debug files, etc. for any error messages about the command line options or any references to /dev/rmt/0n. none. I even tried # find /tmp/amanda -type f | xargs grep '/dev/rmt/0' and got nothing except the alternative weekend configuration I had played with the previous weekend. I checked the documentation man pages and the documentation of the -o option is pretty sparse. All references say see the configuration override section in amanda(8). That section is only a few lines long and makes no reference to the syntax with a comma separating multiple parameter=value pairs. So, possibilities ... 1. The syntax is wrong? And I need to use amdump daily -o reserve=100 -o tapedev=/dev/rmt/0n? (In either case, this should be documented, since one would typically want to override more than one parameter if any at all.) 2. There is a bug in amanda, and it doesn't work? 3. Amanda is being too smart and reverting to /dev/rmt/1n when my override to /dev/rmt/0n fails? Doesn't seem likely. 4. There are additional parameters in my configuration that end up overriding my attempt to override tapedev? Maybe an additional possibility: 5. /dev/rmt/0n == /dev/rmt/1n I think by tapedev=/no/such/tape Paul meant that literally, or at least some string that could not be a legit tape device. Two ways I can see the 0 and 1 devices being equivalent. My HP-DDS changer shows up in Solaris 9 x86 as two /dev/rmt devices, 0 and 1 for me. They are the same SCSI ID but different LUNs (the d# in c5t3d0 scsi dev nomenclature). One is for the changer mechanism and the other for the tape drive. At least that is what it is supposed to be. Only one can be used for the changer, but either seems to work for the tape drive. I don't think that is common, but who knows. The other way is if you installed the tape drive multiple times. /dev/rmt/# are not cleared at boot to allow for persistant names for the same device. For example, you could have two tape drives and only one powered up at boot time. Which was rmt/0 would vary. This is handled by recording the /devices data in /etc/path_to_inst and is only updated by a reconfigure reboot or options to the /usr/sbin/devfsadm command. Maybe there is a stale entry in there that makes rmt/0 and rmt/1 the same. You might also check what the symlinks in /dev/rmt point to with an ls -l and what the targets in /devices have for maj/min device numbers with ls -lL of /dev/rmt. /dev/rmt/0n is actually my internal dds/3 drive. It exists and works, but does not have a tape in it. I've used it recently to do some tests of the behavior of ufsdump for incremental recoveries (checking how ufsdump deals with deleted, moved, etc. files and directories -- the answer being quite well :-) ). When I tried a separate weekend configuration the previous weekend, I used tapedev /dev/rmt/0n in amanda.conf, and it worked as expected -- found the drive but no tape and just spooled to holding disk. My AIT drive is /dev/rmt/1n, and the changer for it is addressed as /dev/scsi/changer/c7t0d0 in amanda.conf. They all work just fine from mt, mtx, ufsdump, various scripts and amanda (amtape, amdump, etc) -- except for the above -o command line override failure. --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst [EMAIL PROTECTED] --- Erdös 4
Re: suggestion for a disk-to-disk backup server
I think by tapedev=/no/such/tape Paul meant that literally, or at least some string that could not be a legit tape device. Not speaking for Paul, but I guess he did. # find /tmp/amanda -type f | xargs grep '/dev/rmt/0' and got nothing except the alternative weekend configuration I had played with the previous weekend. I was looking the same direction, in fact I could not see any place where Amanda logs the tape device she is going to use. The only bits that would log the name of the tape device used are amcheck -t and amrecover. I checked the config logs too /var/amanda/daily but nothing there either. 4. There are additional parameters in my configuration that end up overriding my attempt to override tapedev? My AIT drive is /dev/rmt/1n, and the changer for it is addressed as /dev/scsi/changer/c7t0d0 in amanda.conf. Can't there be an interaction between the changer and the tape drive? That is the changer will anyway load the requested tape (changer names has not been overwritten) and let Amanda know that the requested tape is available at that known tape device, then Amanda using the device provided by the changer rather than the one configured/overwritten? You could try overwritting the changer too. Bests, Olivier
Re: suggestion for a disk-to-disk backup server
On 2007-06-25 22:43, Chris Hoogendyk wrote: Jon LaBadie wrote: On Mon, Jun 25, 2007 at 12:43:50PM -0400, Chris Hoogendyk wrote: ... then I created a dumptype that was the same as my regular dumps but with record no strategy incronly Instead of making it a different configuration, and running into trouble on which full to base an incrmeental, why don't you run an additional amdump of the normal config in the weekend, but which has options to override the config: amdump daily -o reserver=100,tapedev=/no/such/tape With the option tapedev /no/such/tape we force Amanda to fall back to degraded mode, making backup to holdingdisk only, and with the reserve 100, we avoid wasting holdingdisk space for full backups in the degraded mode. On monday morning, you can choose: - flush the backup images to tape manually - autoflush together with the normal backup of the next night I'm not 100% certain, but I believe you may even erase (some of) the holdingdisk files, without flushing. I think that Amanda will take into account that the holdingdisk file disappeared, and schedule the same incremental level again (or a level 0 if due). Because there is no equivalent to amrmtape for a holdingdisk file, and, there are at least some things implemented that detect the loss of an holdingdisk file. Would be nice if someone could confirm this. I thought this would work, because the incremental would be based on ufsdump, and it would know there had been a full, even though it was a different configuration. Apparently, I've got this wrong. I'll check through the documentation, but I thought someone on the list might have done something like this or know either that it can't be done or how to do it as well as why. (Possibly, amanda interprets incronly as iff there has already been a full at some time under this configuration.) ufsdump might have been able to figure out from which date to do the incremental, `IF' it had been asked to do an incremental. However, as this was a new config, amanda knew there had never been a level 0 for this config. And of course it must start from a level 0 to do the incrementals. Even on a config meant to do incronly strategy you must do an initial full dump. -- Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: suggestion for a disk-to-disk backup server
On Tue, Jun 26, 2007 at 09:29:52AM +0200, Paul Bijnens wrote: I'm not 100% certain, but I believe you may even erase (some of) the holdingdisk files, without flushing. I think that Amanda will take into account that the holdingdisk file disappeared, and schedule the same incremental level again (or a level 0 if due). Because there is no equivalent to amrmtape for a holdingdisk file, and, there are at least some things implemented that detect the loss of an holdingdisk file. Would be nice if someone could confirm this. You're half-right -- Amanda will not try to flush a missing file, but it will not take missing files into account in the planner. Amanda-2.5.2 and later has 'amadmin conf holding delete' which can delete holding files and adjust the 'curinfo' database accordingly, such that the appropriate level of dump will take place next time. Also, note that 'amdump' takes a [ hostname [ diskname ..] ] argument, if you only want to dump certain DLEs. Dustin -- Dustin J. Mitchell Storage Software Engineer, Zmanda, Inc. http://www.zmanda.com/
Re: suggestion for a disk-to-disk backup server
RAID0 - 750GB x 2 (SATA) Hosts to backup: about 20 hosts with roughly 100GB each host. So you have 20*100= 2TB to backup on 750GB My mistake, you have 1.5 TG of backup space. What will be the complications and the limitations? Limitation is clear, you deon't have enough backup space. And also what are the recommendations? Increase your backup space. You still have less backup space than disks to backup. Oli vier
Re: suggestion for a disk-to-disk backup server
Just taking into consideration of compression effect? In fact compression depends on your data, do you know how much they compress? Full/incr depends on your data, how your disks are full, how much data you need to do incr every day? Try to use the reverse approach: what data you cannot afford to loose? Then you know what you need to backup. Olivier
Re: suggestion for a disk-to-disk backup server
[EMAIL PROTECTED] a écrit sur 25/06/2007 08:49:59 : RAID0 - 750GB x 2 (SATA) Hosts to backup: about 20 hosts with roughly 100GB each host. So you have 20*100= 2TB to backup on 750GB In RAID0 750GB x 2 = 1,5TB But still that isn't enough. I would suggest you to buy something that could gives you up to 5TB at least. Take into account your required retention period, size of incremental backups (by estimating the amount of data modified everyday), and (last but not least) data growth rate.
Re: suggestion for a disk-to-disk backup server
Yes I do understand that and I didnt mention the number of backup servers. Is it a recommendation of 1.5TB (raid-0 750x2) to backup the maximum of 10 to 12 hosts? or will that be too much of a backup server to do all the 10-12 hosts at once (including incremental and full)? Should it be recommended at maybe 6 to 7 hosts per backup server? Just taking into consideration of compression effect? Thank you. Regards, Rudy On 6/25/07, Olivier Nicole [EMAIL PROTECTED] wrote: RAID0 - 750GB x 2 (SATA) Hosts to backup: about 20 hosts with roughly 100GB each host. So you have 20*100= 2TB to backup on 750GB My mistake, you have 1.5 TG of backup space. What will be the complications and the limitations? Limitation is clear, you deon't have enough backup space. And also what are the recommendations? Increase your backup space. You still have less backup space than disks to backup. Oli vier -- + http://foodblog.rudal.com
Re: suggestion for a disk-to-disk backup server
On Sun, Jun 24, 2007 at 11:43:26PM -0700, Rudy Setiawan enlightened us: I am trying to create backup servers out of the following specs: P4 3.0Ghz 1GB RAM RAID0 - 750GB x 2 (SATA) Hosts to backup: about 20 hosts with roughly 100GB each host. What will be the complications and the limitations? And also what are the recommendations? You realize that by using RAID 0, you are doubling the chance of losing all of your backups, because there is NO redundancy - so if either of your disks has a problem, you lose the entire array. If it is only temporary and you are then moving the data to something else, that might be all right, but I wouldn't feel comfortable with my backups (or anything other than scratch space, really) being on RAID 0. I would use RAID 1, or if you have disks to spare RAID 10. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263
Re: suggestion for a disk-to-disk backup server
Cyrille Bollu schrieb: RAID0 - 750GB x 2 (SATA) Sure you want to do a backup to a Raid0? When one disk crash, you ll loos the data. Here I use a Raid6 Array out of 15 320 GB Sata2 disks + 1 hotspare. So 4 disks can fail until i lose data. Having a recoverable backup is the most important thing here. -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
Re: suggestion for a disk-to-disk backup server
In addition to doing everything to make sure a backup disk failure does not destroy all your backups you need to take into consideration the length of your cycle. My experience so far has been with tape and I am just beginning in the disk-to-disk with virtual tape. How long before Amanda can reuse one of your virtual tapes? The initial backup is going to be all level 0. After that the planner is going to try for a balanced mix of fulls and incrementals. The size of the incrementals depends on the amount of activity on a filesystem. Some will change a lot; some won't change. I would count on 30% of your backup size being backed up every day. For disk-to-disk your backup server is definitely going to contain multiples of your full backup size rather than a fraction of your full backup size. Can we get to the point of making a formula? I'll go out on a limb and put some figures down. So much of it depends on amount of compression, activity of users, and growth: (days per cycle) * (size of full backup) * (average daily fraction of full) let's say your 2 TB compresses to 1.5TB: 30 days * 1.5 TB * 1/3 = 15 TB for a full cycle. If you've been backing up the same servers to tape already you can plug in more accurate numbers for your site. Quoting Matt Hyclak [EMAIL PROTECTED]: On Sun, Jun 24, 2007 at 11:43:26PM -0700, Rudy Setiawan enlightened us: I am trying to create backup servers out of the following specs: P4 3.0Ghz 1GB RAM RAID0 - 750GB x 2 (SATA) Hosts to backup: about 20 hosts with roughly 100GB each host. What will be the complications and the limitations? And also what are the recommendations? You realize that by using RAID 0, you are doubling the chance of losing all of your backups, because there is NO redundancy - so if either of your disks has a problem, you lose the entire array. If it is only temporary and you are then moving the data to something else, that might be all right, but I wouldn't feel comfortable with my backups (or anything other than scratch space, really) being on RAID 0. I would use RAID 1, or if you have disks to spare RAID 10. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 Charlie Reitsma x6642
Re: suggestion for a disk-to-disk backup server
Charlie Reitsma wrote: The initial backup is going to be all level 0. After that the planner is going to try for a balanced mix of fulls and incrementals. Ok, I'm hijacking a thread. Sorry about that. But, really, it does fit in. I just tried something this weekend, and the above comment may be the root of why what I tried didn't work the way I had hoped. I've got a successful daily backup scheme running with an AIT5 tape library. It runs 5 days a week with a dumpcycle of 1 week, runspercycle of 5, and a tapecycle of 30. This fits virtually all of our requirements. However, after some discussion about the possibility of people working on weekends and losing something they did over the weekend before we hit the Monday night backup, I decided to try running incrementals only over the weekend and just stashing them on a holding disk. Without getting into virtual tapes, the quick and dirty seemed to be a backup to DAT with the tape missing and a holding disk available. I set that up in just the hour before going home Friday, figuring I would take a look at virtual tapes this week. I had a 14G partition that I made available for holding space, figuring that would work for incrementals only. This was set up in a separate weekend configuration. Well, the report shows amanda trying to do full (level 0) backups on everything and failing because 14G was just not even close to adequate for that. What I tried was: dumpcycle 1 week runspercycle 2 tapecycle 2 tapedev /dev/rmt/0n #DAT so as not to interfere with my AIT5 I know, I know, the tapecycle is going to get all screwed up not having tapes, but it was only temporary. then I created a dumptype that was the same as my regular dumps but with record no strategy incronly I thought this would work, because the incremental would be based on ufsdump, and it would know there had been a full, even though it was a different configuration. Apparently, I've got this wrong. I'll check through the documentation, but I thought someone on the list might have done something like this or know either that it can't be done or how to do it as well as why. (Possibly, amanda interprets incronly as iff there has already been a full at some time under this configuration.) TIA --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst [EMAIL PROTECTED] --- Erdös 4
Re: suggestion for a disk-to-disk backup server
On Mon, Jun 25, 2007 at 12:43:50PM -0400, Chris Hoogendyk wrote: ... then I created a dumptype that was the same as my regular dumps but with record no strategy incronly I thought this would work, because the incremental would be based on ufsdump, and it would know there had been a full, even though it was a different configuration. Apparently, I've got this wrong. I'll check through the documentation, but I thought someone on the list might have done something like this or know either that it can't be done or how to do it as well as why. (Possibly, amanda interprets incronly as iff there has already been a full at some time under this configuration.) ufsdump might have been able to figure out from which date to do the incremental, `IF' it had been asked to do an incremental. However, as this was a new config, amanda knew there had never been a level 0 for this config. And of course it must start from a level 0 to do the incrementals. Even on a config meant to do incronly strategy you must do an initial full dump. jl -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: suggestion for a disk-to-disk backup server
I apologize that I didnt address my question correctly. On 6/25/07, Olivier Nicole [EMAIL PROTECTED] wrote: Just taking into consideration of compression effect? In fact compression depends on your data, do you know how much they compress? Full/incr depends on your data, how your disks are full, how much data you need to do incr every day? Try to use the reverse approach: what data you cannot afford to loose? Then you know what you need to backup. Olivier -- + http://foodblog.rudal.com
Re: suggestion for a disk-to-disk backup server
Rudy Setiawan schrieb: With the number of hosts that I have, what are the best configuration such as, should I separate each file for each host or I can dump them into one huge file? Amanda backup every DLE into a single file, except you specify tape_splitsize. This could be usefull, if your filesystem has a better performance with smaller files (e.g. 3-5 GB) instead of huge files (e. g. 100 GB). Also you should have a look to the maximum filesize of your filesystem. Will it make a difference in the speed of either backup,restoration or cataloging? I think the speed of writing/reading a huge file or some smaller pieces depents on the filesystem you are using. You can look here for some comparisations: http://linuxgazette.net/102/piszcz.html Only the restore of a DLE could be interrupted when it`s spanned across multiple tapes, because you have to insert the new tape. Other system/amanda configuration that can be suggested to improve the speed of the restoration and backup? You can improve the speed in thinking about the raidlevel (e. g. Raid5), the filesystem, a separate Raid-Controller for the OS and the disks you store the backup on, Regards Marc -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
Re: suggestion for a disk-to-disk backup server
On Mon, Jun 25, 2007 at 11:10:21AM -0700, Rudy Setiawan wrote: ah sorry i pressed the sent button accidentally. thank you all for the inputs. With the number of hosts that I have, what are the best configuration such as, should I separate each file for each host or I can dump them into one huge file? Will it make a difference in the speed of either backup,restoration or cataloging? Other system/amanda configuration that can be suggested to improve the speed of the restoration and backup? Thank you. Amanda uses a unit of a DLE (DiskList Entry) which is either a file system or a directory tree on a single host. So each backup client host may have one or multiple DLEs. Considerations: How much data do all your DLE's represent. You gave numbers of twenty clients and one hundred GB/client. But is that disk space or actual data that will need to be backed up? How frequently do you want a full backup of each DLE? This is called the dumpcycle and can be different for different DLEs but most often the same dumpcycle is used for all DLE. A common dumpcycle is 1 week. How often will you perform backups within each dumpcycle. This is known as the runs per cycle. Amanda does not do all the full backups on the same day. Each run (of the program amdump) will average 1/runs per cycle fraction of the total just doing full backups. The rest of the DLE will receive incremental backups of just the changed data. We don't have any idea if your data is relatively static or very volatile. Once you estimate those values you can get some idea of the magnitude of your amdump runs. For example, suppose you do have 20x100GB of total data, it changes say 2GB/day per host, you chose to have a dumpcycle of 1 wk and run amdump every day of the week. An average amdump run will be full backups of about 300GB (20x100GB/7) plus incrementals of 20*2GB for a total of 340GB/run (day). Next consider how long you wish to retain your backup data. For my SOHO I keep a little more than a month's worth on-line. If that is good for you, you will be storing about 32 days x 340GB/day or 11 TB of data. Most of that may be compressible. Data varies in its compressibility from 0% (already compressed or encrypted data) to as much as 90%. If your data compresses by 40%, you will need 6-7 TB of disk storage for 1 months worth of backups. How is your network? 340GB on a 100Mb/sec network is more than 10 hours at the never achieved 100% efficiency. Sounds like you better be on a gigabit network. Multiple servers may or may not help here, particularly if some servers and clients are on an isolated subnet or switch. Who is going to do the compression, the server or the clients? Compression is cpu intense, server (3GHz is more than needed) may be slowed down. Having the clients do the compression may help and it will reduce the amount of network traffic as compressed data is sent. But if the clients are providing a service, that service performance may be impacted. Sounds to me that you should set up a test config with just 2 or 3 clients. See how that goes and expand from there to more clients and/or more servers. jl -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: suggestion for a disk-to-disk backup server
Jon LaBadie wrote: On Mon, Jun 25, 2007 at 12:43:50PM -0400, Chris Hoogendyk wrote: ... then I created a dumptype that was the same as my regular dumps but with record no strategy incronly I thought this would work, because the incremental would be based on ufsdump, and it would know there had been a full, even though it was a different configuration. Apparently, I've got this wrong. I'll check through the documentation, but I thought someone on the list might have done something like this or know either that it can't be done or how to do it as well as why. (Possibly, amanda interprets incronly as iff there has already been a full at some time under this configuration.) ufsdump might have been able to figure out from which date to do the incremental, `IF' it had been asked to do an incremental. However, as this was a new config, amanda knew there had never been a level 0 for this config. And of course it must start from a level 0 to do the incrementals. Even on a config meant to do incronly strategy you must do an initial full dump. That's sort of what I was guessing with my parenthetical iff statement. So, I could feed in DLE's one at a time until they had all done a full (kind of begging the question that some of the individual DLE's would be too large for the 14G partition). But, then, with a tapecycle of 2, would amanda not also smartly conclude that the configuration no longer had a full backup for DLEs x,y and z? For which it would then request a full backup? Which would make it impossible for me to just let the fulls cycle off until the configuration became an incremental only configuration. And that kind of brings up the question, why incronly? Under what circumstances would someone find this useful? And how would they use it? And, how is it possible to do what I want to do, which is run weekend incrementals without using additional tapes or taking down my tape library? I'm perfectly willing to have them not be recorded and then have the Monday backups catch up using the typical amanda strategy. I would only use the weekend configuration for recovery if someone called me before the following weekend, at which point they would get overwritten. There seems to be some confusion in the amanda man pages documentation. The man page for amanda.conf.5 says: /incronly/ Only do incremental dumps. *amadmin force* should be used to tell /Amanda/ that a full dump has been performed off-line, so that it resets to level 1. It is similar to skip-full, but with incronly full dumps may be scheduled manually. Unfortunately, it appears that /Amanda/ will perform full backups with this configuration, which is probably a bug. But the man page for amadmin.8 says: force [ hostname [ disks ]* ]+ Force the disks on hostname to do a full (level 0) backup during the next Amanda run. which actually makes more sense for force, although I would like to be able to tell amanda that there was a full backup done offline. I can afford to experiment for a while, since my primary backup sequence is working; but, at this point, I'm not sure where to go with it. Trying to fake out amanda by editing its database seems rather chancy. Is there something somewhere that I'm completely missing? --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst [EMAIL PROTECTED] --- Erdös 4
Re: suggestion for a disk-to-disk backup server
On Mon, Jun 25, 2007 at 04:43:40PM -0400, Chris Hoogendyk wrote: Jon LaBadie wrote: On Mon, Jun 25, 2007 at 12:43:50PM -0400, Chris Hoogendyk wrote: ... then I created a dumptype that was the same as my regular dumps but with record no strategy incronly I thought this would work, because the incremental would be based on ufsdump, and it would know there had been a full, even though it was a different configuration. ... However, as this was a new config, amanda knew there had never been a level 0 for this config. And of course it must start from a level 0 to do the incrementals. Even on a config meant to do incronly strategy you must do an initial full dump. That's sort of what I was guessing with my parenthetical iff statement. ... And that kind of brings up the question, why incronly? Under what circumstances would someone find this useful? And how would they use it? Two possibilities come to mind, one I used in the past. I had a DLE with just CD .iso images. Certainly never changing, but there may be things added to it. So after the initial full dump I switched to incronly and it picked up whatever I added. I abandoned the setup when the full dumps were overwritten. I think I then set it up to have a long dumpcycle, basically matching the tapecycle. The other idea is to mimic with amanda the ?differential? style of backups. I.e. an initial full dump and only changes after that. You could take the initial full dump out of rotation. But amanda still doesn't quite do differentials. As I under- stand them, differential don't have levels that go back to a higher level. Instead each changed file appears in the backup tapes just once, after it has changed. There seems to be some confusion in the amanda man pages documentation. The man page for amanda.conf.5 says: /incronly/ Only do incremental dumps. *amadmin force* should be used to tell /Amanda/ that a full dump has been performed off-line, so that it resets to level 1. It is similar to skip-full, but with incronly full dumps may be scheduled manually. Unfortunately, it appears that /Amanda/ will perform full backups with this configuration, which is probably a bug. But the man page for amadmin.8 says: force [ hostname [ disks ]* ]+ Force the disks on hostname to do a full (level 0) backup during the next Amanda run. which actually makes more sense for force, although I would like to be able to tell amanda that there was a full backup done offline. I've never done it, but I wonder if force-bump would allow you to skip the level 0. I can afford to experiment for a while, since my primary backup sequence is working; but, at this point, I'm not sure where to go with it. Trying to fake out amanda by editing its database seems rather chancy. Worth a try, let us know. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: suggestion for a disk-to-disk backup server
I am trying to create backup servers out of the following specs: P4 3.0Ghz 1GB RAM RAID0 - 750GB x 2 (SATA) As it was said by others, RAID 0 may not be the right choice. Virtual tapes and tapespaning allows Amanda to extend a backup across several physical disks, so you don't need RAID 0. The only improvement that RAID 0 could bring you is speed to access the disk array, but then the limitation could still be network bandwidth rather than disk speed. RAID 0 won't allow you to add more disks in the server (at least without rebuilding the array), while you could just plug them in simply if it was not RAIDed, and from your configuration you need many more disks in the server. One last question, from what you said, you seem you want to have several servers, each backuping a set of clients. Why not havuing a single backup server, with enouhg disk resources to backup all your clients? It would be easier to manage I beleive. Bests, olivier
