Re: newbie questions about setting up amanda
Stefan G. Weichinger wrote: I don't know right now if there are differences between includes and excludes when it comes to permissions, if there are, we should think about how to handle them and if we should remove them. It would seem to me that includes are processed by amanda before running GNUTAR, dependent on what files she sees in your DLE; excludes on the other hand are simply passed on to GNUTAR. Which means indeed, that for includes to work, amanda needs read access to the directory, whereas for excludes, runtar's being SUID root is sufficient. Since amanda is not supposed to be SUID root, I think that for the time being, a note in the docs is all that we can do. (The relevant code seems to be in client-src/client_util.c, functions build_include() and add_include().) Alex -- Alexander Jolk / BUF Compagnie tel +33-1 42 68 18 28 / fax +33-1 42 68 18 29
Re: newbie questions about setting up amanda
Hi, Frank, on Samstag, 14. Mai 2005 at 01:54 you wrote to amanda-users: >> I thought this would be obvious ... every DLE has to be readable by >> the amanda-user or, more detailled, the user, AMANDA has been >> configured with (--with-user). FS> That's not technically correct, I back up several DLEs that are not FS> readable by the amanda user. Using a group such as disk or bin may give FS> it access to the underlying device in order to run dump, but it doesn't give it FS> access to all directories on that device when using tar. That is why many of FS> the amanda binaries are suid root, so it can back it up. You're right, my mistake ... sorry about that misinformation. -- best regards, Stefan Stefan G. Weichinger mailto:[EMAIL PROTECTED]
Re: newbie questions about setting up amanda
--On Friday, May 13, 2005 23:27:20 +0200 "Stefan G. Weichinger" <[EMAIL PROTECTED]> wrote: > Hi, Jon, > > on Freitag, 13. Mai 2005 at 16:14 you wrote to amanda-users: > >>> Let me strongly suggest that a paragraph explaining that the base >>> directory in the DLE *must* be readable by amanda, so that it can >>> build the include file. > > JL> Absolutely. I think it is only needed for include, not for exclude. > > I thought this would be obvious ... every DLE has to be readable by > the amanda-user or, more detailled, the user, AMANDA has been > configured with (--with-user). That's not technically correct, I back up several DLEs that are not readable by the amanda user. Using a group such as disk or bin may give it access to the underlying device in order to run dump, but it doesn't give it access to all directories on that device when using tar. That is why many of the amanda binaries are suid root, so it can back it up. For example, I back up a DLE of /home/oracle using tar. The permissions on the directory are 700, and it is owned by oracle:dba and the amanda user can't see any of it (and proves it on every amcheck run complaining about not being able to read the amanda exclude file in there specified in the dumptype. However, since runtar is suid root, it can successfully read the exclude file and also backup the contents of that directory. Frank > > I quickly scanned this thread, AFAI can see there was no discussion of > the group-membership of the amanda-user (--with-group). A reason to > make the amanda-user member of a group like bin or disk is to provide > this user with the rights to read files it otherwise would not be > allowed to read. Just as a sidenote ... > > I don't know right now if there are differences between includes and > excludes when it comes to permissions, if there are, we should think > about how to handle them and if we should remove them. > > --- > > If you think the current behavior should get explained more > explicitly, please let me know where you would like to have this > information placed. > > -- > best regards, > Stefan > > Stefan G. Weichinger > mailto:[EMAIL PROTECTED] > > > -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: newbie questions about setting up amanda
Hi, Jon, on Freitag, 13. Mai 2005 at 16:14 you wrote to amanda-users: >> Let me strongly suggest that a paragraph explaining that the base >> directory in the DLE *must* be readable by amanda, so that it can >> build the include file. JL> Absolutely. I think it is only needed for include, not for exclude. I thought this would be obvious ... every DLE has to be readable by the amanda-user or, more detailled, the user, AMANDA has been configured with (--with-user). I quickly scanned this thread, AFAI can see there was no discussion of the group-membership of the amanda-user (--with-group). A reason to make the amanda-user member of a group like bin or disk is to provide this user with the rights to read files it otherwise would not be allowed to read. Just as a sidenote ... I don't know right now if there are differences between includes and excludes when it comes to permissions, if there are, we should think about how to handle them and if we should remove them. --- If you think the current behavior should get explained more explicitly, please let me know where you would like to have this information placed. -- best regards, Stefan Stefan G. Weichinger mailto:[EMAIL PROTECTED]
Re: newbie questions about setting up amanda
Stefan G. Weichinger wrote: Hi, Matt, on Freitag, 13. Mai 2005 at 16:03 you wrote to amanda-users: MH> If you are using software to disable compression, and you are using DDS MH> tapes, you will want to look in the archives for Gene Heskett's script that MH> rewrites tapes...DDS stores compression on/off flags on the tape itself, so MH> disabling the compression on your drive may not be enough. I have read several postings regarding that DDS-tape-issue, describing why and how to reset these tapes. I have heard pros and cons and would like to ask all of you how I could sum up this topic for the AMANDA-docs. If this is an issue that comes up for amanda-users when using DDS-drives (and it seems to be so) then we should document this in the docs. I am ready to do that, I just ask for your help to get all the facts straight. Actually it has nothing to do with DDS-tapes. On Solaris, you have different device names that control how to write to tape (low density, medium density, high density, with or without compression -- because most modern devices have no density difference, people believe there is only a difference between with or without compression). The name is important when you write to a tape. But when you read from a tape, the OS (or tapedrive itself, I don't know) can easily see that you are trying to read a tape with or without compressed format, or low or high density, by just checking the builtin error control: when reading in the wrong setting, the result is garbage, full of errors. When you're near the tapedrive, you can even hear it try out the different settings for the first read. On Solaris, when writing on a tape, Solaris does know what to do depending on the device name. However, when you have e.g. a compressed tape, and you skip to end of file, and then append to tape, Solaris also sets the settings to the current settings and keeps these, and ignores the settings implied by tapedevicename. On linux, there is no setting implied by device name (*). So when you read something from a tape, the settings become those of the tape. Everything you write then, are in these settings. Even with when writing from the beginning. The trick is to set whatever you want with "mt", and then write to the tape without any reading of that tape. The trouble is that amcheck, amdump first read the tape to verify its label, and then start writing. Since 2.4.4 amlabel does the same thing. So the only solution is to write some garbage using dd or any program that does not read before writing to tape. (*) In Linux there is a not-well documented file /etc/stinit.def that has potential to accomplish the same thing as Soliaris: different devicenames, having different settings. At least that's what I think. But I never got there to investigate this decently. Any clear explanation here would be welcome. PS very fast typed, because I need to go home... -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
Re: newbie questions about setting up amanda
Hi, Matt, on Freitag, 13. Mai 2005 at 16:03 you wrote to amanda-users: MH> If you are using software to disable compression, and you are using DDS MH> tapes, you will want to look in the archives for Gene Heskett's script that MH> rewrites tapes...DDS stores compression on/off flags on the tape itself, so MH> disabling the compression on your drive may not be enough. I have read several postings regarding that DDS-tape-issue, describing why and how to reset these tapes. I have heard pros and cons and would like to ask all of you how I could sum up this topic for the AMANDA-docs. If this is an issue that comes up for amanda-users when using DDS-drives (and it seems to be so) then we should document this in the docs. I am ready to do that, I just ask for your help to get all the facts straight. -- best regards, Stefan Stefan G. Weichinger mailto:[EMAIL PROTECTED]
Re: newbie questions about setting up amanda
On Fri, May 13, 2005 at 11:23:12AM +0100, Rodrigo Ventura wrote: > > The amdump I'm doing right now is looking good: the sendsize.*.include > files for the var/spool/imap/user is non-zero (and its contents make > sense). I guess the problem was that the directory var/spool/imap/user > was not readable by amanda. Although gnutar is performed as root, and > therefore there are no permission requirements, the include files are > performed as amanda. Good to know. You may have been the first to have an environment that triggered the problem. > Let me strongly suggest that a paragraph explaining that the base > directory in the DLE *must* be readable by amanda, so that it can > build the include file. Absolutely. I think it is only needed for include, not for exclude. > Maybe an alternative way is to let the program that builds the include > files to be setuid root. That has its own set of problems which are to be avoided if possible. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
On Fri, May 13, 2005 at 02:12:57PM +0100, Rodrigo Ventura enlightened us: > My amdump has just finished, and the report looks good: > > DUMPER STATSTAPER STATS > HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s > -- - > omni / 1 619070 619070 --2:513622.0 2:513619.0 > omni /boot 1 10 10 --0:00 117.3 0:14 0.7 > omni /home/ag1 984600 984600 --2:356370.8 6:282539.7 > omni /home/hm2 34970 34970 --0:084298.4 0:122894.6 > omni /home/nt2 45490 45490 --0:391178.2 0:20.9 > omni /home/uz14940 4940 --0:031866.5 0:041241.9 > omni /root 1 330330 --0:001298.6 0:01 256.2 > omni /usr2 42670 42670 --2:32 280.9 0:113761.4 > omni -ap/user/ag 1 38135203813520 -- 19:163299.5 20:073159.3 > omni -ap/user/hm 0 75231407523140 -- 34:023683.5 36:253443.7 > omni -ap/user/nt 0 66056706605670 -- 21:565021.1 35:393088.3 > omni -ap/user/uz 0 15355301535530 -- 15:041699.0 7:113564.5 > > > During the run, I noticed a process called "gzip --best" being used by > amanda, however I have a "compress none" directive. I want to switch > to gzip in the future -- change compress to somethin non-none -- I > suppose that with software compression, the ORIG-KB and the OUT-KB > will differ. > The gzip --best is the index files being created. And yes, when you enable gzip, the ORIG and OUT will change. > And by the way, how can I disable the hardware compression in the scsi > tape device? Is there any way of doing so within amanda? > Depends. There is usually a hardware switch or two on the drive itself. Software-wise, it's handled through your OS. Solaris uses different device names, whereas Linux uses the mt command (look for COMPRESSION, DEFCOMPRESSION, and/or DATCOMPRESSION...varies from drive to drive). If you are using software to disable compression, and you are using DDS tapes, you will want to look in the archives for Gene Heskett's script that rewrites tapes...DDS stores compression on/off flags on the tape itself, so disabling the compression on your drive may not be enough. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 pgpAWIomJgohV.pgp Description: PGP signature
Re: newbie questions about setting up amanda
On Fri, 13 May 2005 at 2:12pm, Rodrigo Ventura wrote > During the run, I noticed a process called "gzip --best" being used by > amanda, however I have a "compress none" directive. I want to switch Amanda compresses the index files -- that's the gzip. > to gzip in the future -- change compress to somethin non-none -- I > suppose that with software compression, the ORIG-KB and the OUT-KB > will differ. Yes, and amanda will keep a history of how much each DLE compresses and use that in future estimates. > And by the way, how can I disable the hardware compression in the scsi > tape device? Is there any way of doing so within amanda? This is OS dependent and independent of amanda. In Linux, look at the 'mt' command. For Solaries, it's often controlled by which device you use. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: newbie questions about setting up amanda
My amdump has just finished, and the report looks good: DUMPER STATSTAPER STATS HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -- - omni / 1 619070 619070 --2:513622.0 2:513619.0 omni /boot 1 10 10 --0:00 117.3 0:14 0.7 omni /home/ag1 984600 984600 --2:356370.8 6:282539.7 omni /home/hm2 34970 34970 --0:084298.4 0:122894.6 omni /home/nt2 45490 45490 --0:391178.2 0:20.9 omni /home/uz14940 4940 --0:031866.5 0:041241.9 omni /root 1 330330 --0:001298.6 0:01 256.2 omni /usr2 42670 42670 --2:32 280.9 0:113761.4 omni -ap/user/ag 1 38135203813520 -- 19:163299.5 20:073159.3 omni -ap/user/hm 0 75231407523140 -- 34:023683.5 36:253443.7 omni -ap/user/nt 0 66056706605670 -- 21:565021.1 35:393088.3 omni -ap/user/uz 0 15355301535530 -- 15:041699.0 7:113564.5 During the run, I noticed a process called "gzip --best" being used by amanda, however I have a "compress none" directive. I want to switch to gzip in the future -- change compress to somethin non-none -- I suppose that with software compression, the ORIG-KB and the OUT-KB will differ. And by the way, how can I disable the hardware compression in the scsi tape device? Is there any way of doing so within amanda? Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
The amdump I'm doing right now is looking good: the sendsize.*.include files for the var/spool/imap/user is non-zero (and its contents make sense). I guess the problem was that the directory var/spool/imap/user was not readable by amanda. Although gnutar is performed as root, and therefore there are no permission requirements, the include files are performed as amanda. Let me strongly suggest that a paragraph explaining that the base directory in the DLE *must* be readable by amanda, so that it can build the include file. Maybe an alternative way is to let the program that builds the include files to be setuid root. Thank you all for all the help provided. Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
> "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: Jon> Is your runtar (and several other amanda programs) set-uid'ed Jon> root? -rwsr-x---1 root backup 50972 Feb 19 16:57 runtar among others. It seems amanda is properly installed, as far as setuid's are concerned. -- I've been examining the /tmp/amanda files, which seem to provile tons of interesting debug info. 20050512 was the latest date level 0 backups were performed for the /var/spool/imap/user. $ grep -l var/spool/imap *20050512* [a subset of files, hopefuly containing relevant information] The file runtar.20050512043718.debug contains: runtar: debug 1 pid 17395 ruid 92 euid 0: start at Thu May 12 04:37:18 2005 /bin/tar: version 2.4.4p4 running: /bin/tar: /bin/tar --create --file /dev/null --directory /var/spool/imap/user --one-file-system --listed-incremental /usr/local/amanda/var/amanda/gnutar-lists/omni_var_spool_imap_user_ag_0.new --sparse --ignore-failed-read --totals --files-from /tmp/amanda/sendsize._var_spool_imap_user_ag.20050512043718.include The file sendsize._var_spool_imap_user_ag.20050512043718.include is empty (zero bytes). This suggests this latter file is not created by runtar (as root), but rather by amanda; besides: -rw---1 amanda backup 0 May 12 04:37 /tmp/amanda/sendsize._var_spool_imap_user_ag.20050512043718.include Meaning that amanda user maybe requires read access to the base directory, in order to be able to build the include file. I'll change the permissions of /var/spool/imap/user (as well as parent directories) for chmod og+xr and try a amdump right now. I'll post the results after it. Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
Hi, Joshua, on Freitag, 13. Mai 2005 at 01:54 you wrote to amanda-users: JBL> On Thu, 12 May 2005 at 9:34pm, Rodrigo Ventura wrote >> Alexander Jolk wrote: >> > Once again, your syntax seems right to all of us, but nobody really >> > seems to be actively using `include' right now. I know for certain that >> > using `exclude' works, for example. >> >> I see. My problem is that I have a /home full of users (34G used) and a >> /var/spool/imap/users full of mail accounts (21G). I'm using DDS 36/72 >> tapes (36G uncompressed), and it does not seem right to me to used it >> without spliting these directories in several DLE's. >> >> But if the include is so seldom used, maybe there better ways of doing >> backup. JBL> I haven't been following this closely, I must admit, but I use include in JBL> several DLEs. Just for the records: I am also using "include", in my own setup here and at several clients with similar needs as yours, Rodrigo. -- best regards, Stefan Stefan G. Weichinger mailto:[EMAIL PROTECTED]
Re: newbie questions about setting up amanda
Rodrigo Ventura wrote: Alexander Jolk wrote: Ahm, sorry, I don't quite follow you there. Your sendbackup.* files are empty, zero bytes? Yeah, zero bytes. Barring an installation problem that others have suggested, could you tell us whether there are any files of nonzero size in /tmp/amanda/? Could you forward the relevant ones to the list? (I think you already sent an amreport e-mail, didn't you?) Alex -- Alexander Jolk / BUF Compagnie tel +33-1 42 68 18 28 / fax +33-1 42 68 18 29
Re: newbie questions about setting up amanda
On Thu, 12 May 2005 at 9:34pm, Rodrigo Ventura wrote > Alexander Jolk wrote: > > Once again, your syntax seems right to all of us, but nobody really > > seems to be actively using `include' right now. I know for certain that > > using `exclude' works, for example. > > I see. My problem is that I have a /home full of users (34G used) and a > /var/spool/imap/users full of mail accounts (21G). I'm using DDS 36/72 > tapes (36G uncompressed), and it does not seem right to me to used it > without spliting these directories in several DLE's. > > But if the include is so seldom used, maybe there better ways of doing > backup. I haven't been following this closely, I must admit, but I use include in several DLEs. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: newbie questions about setting up amanda
--On Thursday, May 12, 2005 21:37:09 +0100 Rodrigo Ventura <[EMAIL PROTECTED]> wrote: > Alexander Jolk wrote: >> Ahm, sorry, I don't quite follow you there. Your sendbackup.* files are >> empty, zero bytes? That shouldn't be the case already. (Don't get >> caught by the fact that they are readable only to your amanda user, and >> root!) And which files are the include files? Could you elaborate a >> little bit? > > Yeah, zero bytes. > > In fact /var/spool/imap is not readable by amanda. But it is absurd that > amanda has to read every file in the system... (mode=600 files are > unbackupable???) Or maybe amanda only needs to be able to read the base > directory of the dump. Did you run 'make install' as root as required? If not, you didn't get the SUID bits set on the appropriate binaries. Frank > > Cheers, > > Rodrigo -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: newbie questions about setting up amanda
On Thu, May 12, 2005 at 09:37:09PM +0100, Rodrigo Ventura wrote: > > In fact /var/spool/imap is not readable by amanda. But it is absurd that > amanda has to read every file in the system... (mode=600 files are > unbackupable???) Or maybe amanda only needs to be able to read the base > directory of the dump. Uhhh, I don't get why you feel it is absurd to need to read a file in order to back it up. How else does one backup, i.e. copy a file, without reading its contents? However, amanda does not do the backup, so the 'amanda user' does not need read permission. Root does the backups if you are using gnutar. Properly installed, the amanda program ".../libexec/runtar" is root owned and set-uid'ed. That is the program that invokes gnutar and thus gnutar runs with root privleges. Is your runtar (and several other amanda programs) set-uid'ed root? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
Alexander Jolk wrote: Ahm, sorry, I don't quite follow you there. Your sendbackup.* files are empty, zero bytes? That shouldn't be the case already. (Don't get caught by the fact that they are readable only to your amanda user, and root!) And which files are the include files? Could you elaborate a little bit? Yeah, zero bytes. In fact /var/spool/imap is not readable by amanda. But it is absurd that amanda has to read every file in the system... (mode=600 files are unbackupable???) Or maybe amanda only needs to be able to read the base directory of the dump. Cheers, Rodrigo
Re: newbie questions about setting up amanda
Alexander Jolk wrote: Once again, your syntax seems right to all of us, but nobody really seems to be actively using `include' right now. I know for certain that using `exclude' works, for example. I see. My problem is that I have a /home full of users (34G used) and a /var/spool/imap/users full of mail accounts (21G). I'm using DDS 36/72 tapes (36G uncompressed), and it does not seem right to me to used it without spliting these directories in several DLE's. But if the include is so seldom used, maybe there better ways of doing backup. Cheers, Rodrigo
Re: newbie questions about setting up amanda
The man page has this suspicious paragraph: diskdevice Default: same as diskname. The name of the disk device to be backed up. It may be a full device name, a device name without the /dev/ prefix, e.g. sd0a, or a mount point such as /usr. They say "mount point" rather than "directory". I could change the diskdevice to the actual mount point (/). But then there is paragraph concerning "include" directive in DLE: All include expression are expanded by amanda and concatenated in one file and passed to gnutar as a --files-from argument. They must start with "./" and containing no other "/". So setting the include file to "./var/spool/imap/user/[a-h]*" would not work. Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
> "Alexander" == Alexander Jolk <[EMAIL PROTECTED]> writes: Alexander> I'm pretty sure there's an error somewhere in your Alexander> disklist config since your index files are empty. I Alexander> don't remember whether you showed us already your Alexander> relevant /tmp/amanda/ debug files; can you try to find Alexander> one sendbackup.* file corresponding to an empty backup? I didn't know they exist! That's great. The sendbackup.* corresponding to the /var/spool/imap/user/* directories in question are empty. These are the include files generated from the DLE and fed into gnutar. The sendsize's are also empty. Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
On Thu, May 12, 2005 at 04:19:16PM +0100, Rodrigo Ventura wrote: > > "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: > > Jon> Was there ever a successful backup of the DLE's in question? > > No, I'm pretty sure there was never one. > > Jon> I presume the DLE's for /home/{ag|hm|nt} are similar to those > Jon> for /var/spool/imap/user/{ag|hm|nt}. Same similarity for the > Jon> "uz" entries. Yet all the /home ones are working, the > Jon> "include" ones for /var... aren't. > > Jon> What is different about the /home and /var/spool DLE's? > > Not much. The latter was based on a copy&paste of the former. > > Jon> What is different about /home itself vs /var/spool/imap/user? > > There is a physical partition mounted on /home, and therefore /home > corresponds to the root directory of the partition in question. The > /var/spool/... are in the root (/) partition. I'm assuming (hoping) > that the "include" directive only works for directories in the root of > a mounted partition!!! I want to split the /var/spool/imap/user > directory in several dumps; do I have to create a physical partition > and mount it at /var/spool/imap/user? (that's unreasonable for my server) > > I guess the central issue lies here: does a DLE have to correspond to > a filesystem? Does the include directive only works for directories at > the root of mounted filesystems? > > If these are the constraints of amanda, how can find a workaround to > solve my problem of backing up subsets of directories deep in the > directory structure of a filesystem? I don't recall there being any restriction like that. But I must admit, the include code and capability is less solid than the exclude stuff. Given that, how about a workaround that changes things to excludes? The includes you have used seem like they should work (to me anyway). Here would be some sample changes: Current: omni /var/spool/imap/user/ag /var/spool/imap/user { comment "omni mail accounts a-g" include file "./[a-g]*" global } Revised (if you know there is only "a-z": omni /var/spool/imap/user/ag /var/spool/imap/user { comment "omni mail accounts a-g" global exclude file "./[h-z]*" } Revised (if you 'might' have things like Rodrigo or 3ware or ~junkfile in /var/spool/imap): omni /var/spool/imap/user/ag /var/spool/imap/user { comment "omni mail accounts a-g" global exclude file "./[ -`h-~]*" } Space (' ') is the first 'non-control' character (I'm assuming no file/dirs in imap start with control chars) and Backquote, aka accent grave ('`') is the last char before lower case 'a'. Tilde ('~') is the last printable ascii character. BTW if I were setting this up I would not have dumptype 'global' specify a backup program (gnutar). Someday I might want to use the global properties with a different backup program. Instead I'd setup a more general 'global' dumptype and two additional dumptypes like gtarglobal and dumpglobal with first include global then specify the dump program. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
> "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: Jon> Does 'amadmin disklist omni /var/spool/... ' show Jon> indexing turned on? Yes. Jon> Are there index files for those DLE's under your index dir? Yes, all gzipped; decompress to an empty file. Jon> Was there ever a successful backup of the DLE's in question? No, I'm pretty sure there was never one. Jon> Does the mailed to you report show anything of interest about Jon> failures or strange results? Only in the root partition, because of log files, sockets, and so on. No strange results in the DLE's in question. Jon> I presume the DLE's for /home/{ag|hm|nt} are similar to those Jon> for /var/spool/imap/user/{ag|hm|nt}. Same similarity for the Jon> "uz" entries. Yet all the /home ones are working, the Jon> "include" ones for /var... aren't. Jon> What is different about the /home and /var/spool DLE's? Not much. The latter was based on a copy&paste of the former. Jon> What is different about /home itself vs /var/spool/imap/user? There is a physical partition mounted on /home, and therefore /home corresponds to the root directory of the partition in question. The /var/spool/... are in the root (/) partition. I'm assuming (hoping) that the "include" directive only works for directories in the root of a mounted partition!!! I want to split the /var/spool/imap/user directory in several dumps; do I have to create a physical partition and mount it at /var/spool/imap/user? (that's unreasonable for my server) I guess the central issue lies here: does a DLE have to correspond to a filesystem? Does the include directive only works for directories at the root of mounted filesystems? If these are the constraints of amanda, how can find a workaround to solve my problem of backing up subsets of directories deep in the directory structure of a filesystem? Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
On Thu, May 12, 2005 at 11:40:21AM +0100, Rodrigo Ventura wrote: > > "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: > > Jon> These are level 1 dumps. Maybe no changes? Did you get the > Jon> same results on the last level 0? If I recall, these are > Jon> mail dirs, is there stuff there? > > Last dump summary: > > DUMPER STATSTAPER STATS > HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s > -- - > omni / 0 13315801331580 --6:163543.9 6:163542.4 > omni /boot 05020 5020 --0:0011510.0 0:041246.0 > omni /home/ag27010 7010 --0:023301.9 0:023041.9 > omni /home/hm1 109930 109930 --0:205490.9 0:382924.1 > omni /home/nt2 25960 25960 --0:045938.1 0:102607.6 > omni /home/uz0 25456402545640 --9:064660.3 14:033019.2 > omni /root 0 393480 393480 --1:224826.4 2:132964.4 > omni /usr0 35510003551000 -- 17:543305.3 13:364353.2 > omni -ap/user/ag 0 10 10 --0:00 182.9 0:04 2.7 > omni -ap/user/hm 0 10 10 --0:00 184.8 0:03 3.2 > omni -ap/user/nt 0 10 10 --0:00 133.9 0:08 1.3 > omni -ap/user/uz 0 15315201531520 -- 13:581828.5 7:523243.4 > > Now they are level 0, right? Still no dump. The latest one seems > dumped ok, but I'm not sure. Check this interaction: > > -- > # cd / > # amrecover ISR > AMRECOVER Version 2.4.4p4. Contacting server on omni ... > 220 omni AMANDA index server (2.4.4p4) ready. > 200 Access OK > Setting restore date to today (2005-05-12) > 200 Working date set to 2005-05-12. > Scanning /var/amanda/dumps... > 200 Config set to ISR. > 200 Dump host set to omni. > Trying disk / ... > $CWD '/' is on disk '/' mounted at '/'. > 200 Disk set to /. > / > amrecover> listdisk > 200- List of disk for host omni > 201- / > 201- /boot > 201- /usr > 201- /root > 201- /home/ag > 201- /home/hm > 201- /home/nt > 201- /home/uz > 201- /var/spool/imap/user/ag > 201- /var/spool/imap/user/hm > 201- /var/spool/imap/user/nt > 201- /var/spool/imap/user/uz > 200 List of disk for host omni > amrecover> setdisk /var/spool/imap/user/uz > 200 Disk set to /var/spool/imap/user/uz. > amrecover> ls > 2005-05-12 . > 2005-05-12 vab/ > 2005-05-12 vale/ > 2005-05-12 vodakura/ > 2005-05-12 vsilva/ > 2005-05-12 vtraver/ > 2005-05-12 yoda/ > 2005-05-12 zlatan/ > amrecover> setdisk /var/spool/imap/user/ag > 200 Disk set to /var/spool/imap/user/ag. > No index records for disk for specified date > If date correct, notify system administrator > amrecover> ls > amrecover> > -- Does 'amadmin disklist omni /var/spool/... ' show indexing turned on? Are there index files for those DLE's under your index dir? Was there ever a successful backup of the DLE's in question? Does the mailed to you report show anything of interest about failures or strange results? I presume the DLE's for /home/{ag|hm|nt} are similar to those for /var/spool/imap/user/{ag|hm|nt}. Same similarity for the "uz" entries. Yet all the /home ones are working, the "include" ones for /var... aren't. What is different about the /home and /var/spool DLE's? What is different about /home itself vs /var/spool/imap/user? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
> "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: Jon> These are level 1 dumps. Maybe no changes? Did you get the Jon> same results on the last level 0? If I recall, these are Jon> mail dirs, is there stuff there? Last dump summary: DUMPER STATSTAPER STATS HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -- - omni / 0 13315801331580 --6:163543.9 6:163542.4 omni /boot 05020 5020 --0:0011510.0 0:041246.0 omni /home/ag27010 7010 --0:023301.9 0:023041.9 omni /home/hm1 109930 109930 --0:205490.9 0:382924.1 omni /home/nt2 25960 25960 --0:045938.1 0:102607.6 omni /home/uz0 25456402545640 --9:064660.3 14:033019.2 omni /root 0 393480 393480 --1:224826.4 2:132964.4 omni /usr0 35510003551000 -- 17:543305.3 13:364353.2 omni -ap/user/ag 0 10 10 --0:00 182.9 0:04 2.7 omni -ap/user/hm 0 10 10 --0:00 184.8 0:03 3.2 omni -ap/user/nt 0 10 10 --0:00 133.9 0:08 1.3 omni -ap/user/uz 0 15315201531520 -- 13:581828.5 7:523243.4 Now they are level 0, right? Still no dump. The latest one seems dumped ok, but I'm not sure. Check this interaction: -- # cd / # amrecover ISR AMRECOVER Version 2.4.4p4. Contacting server on omni ... 220 omni AMANDA index server (2.4.4p4) ready. 200 Access OK Setting restore date to today (2005-05-12) 200 Working date set to 2005-05-12. Scanning /var/amanda/dumps... 200 Config set to ISR. 200 Dump host set to omni. Trying disk / ... $CWD '/' is on disk '/' mounted at '/'. 200 Disk set to /. / amrecover> listdisk 200- List of disk for host omni 201- / 201- /boot 201- /usr 201- /root 201- /home/ag 201- /home/hm 201- /home/nt 201- /home/uz 201- /var/spool/imap/user/ag 201- /var/spool/imap/user/hm 201- /var/spool/imap/user/nt 201- /var/spool/imap/user/uz 200 List of disk for host omni amrecover> setdisk /var/spool/imap/user/uz 200 Disk set to /var/spool/imap/user/uz. amrecover> ls 2005-05-12 . 2005-05-12 vab/ 2005-05-12 vale/ 2005-05-12 vodakura/ 2005-05-12 vsilva/ 2005-05-12 vtraver/ 2005-05-12 yoda/ 2005-05-12 zlatan/ amrecover> setdisk /var/spool/imap/user/ag 200 Disk set to /var/spool/imap/user/ag. No index records for disk for specified date If date correct, notify system administrator amrecover> ls amrecover> -- Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
On Wed, May 11, 2005 at 09:58:09PM +0100, Rodrigo Ventura wrote: > > "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: > > Jon> So far your DLE's seem ok to me. The one part I question is > Jon> "global". What is your global definition? Is it even > Jon> specifying GNUTAR as the dump program? > > My global is: > > define dumptype global { > comment "Global definitions" > index yes > compress none > program "GNUTAR" > dumpcycle 4 > } > > I'm thinking about changing the compress to something different from > none, disable the hardware compression (don't know how yet), and let > amanda manage the tape size in a more informed way. Maybe I'll have to > run tapetype again with the hardware compression disabled to get a > more precise measurement of the tape capacity. > > I get the following dump summary: > DUMPER STATSTAPER STATS > HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s > -- - > omni / 1 685840 685840 --3:223397.6 3:223394.9 > omni /boot 1 10 10 --0:00 130.5 0:03 2.9 > omni /home/ag12120 2120 --0:011588.8 0:011673.3 > omni /home/hm0 74406807440680 -- 17:387029.5 44:382778.5 > omni /home/nt0 1575509015755090 -- 32:398043.1 92:552825.9 > omni /home/uz18120 8120 --0:023384.3 0:023687.6 > omni /root 1 330330 --0:001879.1 0:01 293.9 > omni /usr1 55220 55220 --0:381435.3 0:134100.1 > omni -ap/user/ag 1 10 10 --0:00 206.7 0:03 3.0 > omni -ap/user/hm 1 10 10 --0:00 195.1 0:03 3.1 > omni -ap/user/nt 1 10 10 --0:00 118.8 0:04 2.7 > omni -ap/user/uz 1 119180 119180 --0:313807.1 0:333635.5 > > where the -ap/user/* disks (except the last one) always get about > 10K since ever. These are level 1 dumps. Maybe no changes? Did you get the same results on the last level 0? If I recall, these are mail dirs, is there stuff there? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
> "Jon" == Jon LaBadie <[EMAIL PROTECTED]> writes: Jon> So far your DLE's seem ok to me. The one part I question is Jon> "global". What is your global definition? Is it even Jon> specifying GNUTAR as the dump program? My global is: define dumptype global { comment "Global definitions" index yes compress none program "GNUTAR" dumpcycle 4 } I'm thinking about changing the compress to something different from none, disable the hardware compression (don't know how yet), and let amanda manage the tape size in a more informed way. Maybe I'll have to run tapetype again with the hardware compression disabled to get a more precise measurement of the tape capacity. I get the following dump summary: DUMPER STATSTAPER STATS HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -- - omni / 1 685840 685840 --3:223397.6 3:223394.9 omni /boot 1 10 10 --0:00 130.5 0:03 2.9 omni /home/ag12120 2120 --0:011588.8 0:011673.3 omni /home/hm0 74406807440680 -- 17:387029.5 44:382778.5 omni /home/nt0 1575509015755090 -- 32:398043.1 92:552825.9 omni /home/uz18120 8120 --0:023384.3 0:023687.6 omni /root 1 330330 --0:001879.1 0:01 293.9 omni /usr1 55220 55220 --0:381435.3 0:134100.1 omni -ap/user/ag 1 10 10 --0:00 206.7 0:03 3.0 omni -ap/user/hm 1 10 10 --0:00 195.1 0:03 3.1 omni -ap/user/nt 1 10 10 --0:00 118.8 0:04 2.7 omni -ap/user/uz 1 119180 119180 --0:313807.1 0:333635.5 where the -ap/user/* disks (except the last one) always get about 10K since ever. What's wrong with my "global" and/or DLE? Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
On Wed, May 11, 2005 at 08:12:21PM +0100, Rodrigo Ventura wrote: > > Thank you for the answers. > > I have one more question/problem: I have several partitions that are > not being dumped. The device/directory/include specifications in DLE > is not yet very clear to me. I have a root partition (/) where the OS > is installed, as well as IMAP accounts at /var/spool/imap/user/. I > need to split the accounts, so I did something like: > > omni /var/spool/imap/user/ag /var/spool/imap/user { > comment "omni mail accounts a-g" > include file "./[a-g]*" > global > } > [...] > omni /var/spool/imap/user/uz /var/spool/imap/user { > comment "omni mail accounts u-z and the rest" > exclude file "./[a-t]*" > global > } > > I'm assuming the include file specification is relative to the > /var/spool/imap/user. Is it not? Do I have to specify > "./var/spool/imap/user/[a-t]*" instead? However, there is a disturbing > sentence in the amanda man page: > >"All include expression are expanded by amanda and concatenated in >one file and passed to gnutar as a --files-from argument. They must >start with "./" and containing no other "/"." > > What does it mean exactly? Strings like "./var/spool/imap/user/[a-t]*" > are not accepted in includes? > So far your DLE's seem ok to me. The one part I question is "global". What is your global definition? Is it even specifying GNUTAR as the dump program? BTW it probably won't make a difference, but I like to include all my common stuff in the DLE first, then have the specific stuff work on that. For example, I'm sure it does not, but what if the global dumptype had a specification for an exclude that overrode yours. I like to think of it as "just like the global dumptype, except" Rather than "here are some unique things, and all the the global stuff too. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: newbie questions about setting up amanda
Thank you for the answers. I have one more question/problem: I have several partitions that are not being dumped. The device/directory/include specifications in DLE is not yet very clear to me. I have a root partition (/) where the OS is installed, as well as IMAP accounts at /var/spool/imap/user/. I need to split the accounts, so I did something like: omni /var/spool/imap/user/ag /var/spool/imap/user { comment "omni mail accounts a-g" include file "./[a-g]*" global } [...] omni /var/spool/imap/user/uz /var/spool/imap/user { comment "omni mail accounts u-z and the rest" exclude file "./[a-t]*" global } I'm assuming the include file specification is relative to the /var/spool/imap/user. Is it not? Do I have to specify "./var/spool/imap/user/[a-t]*" instead? However, there is a disturbing sentence in the amanda man page: "All include expression are expanded by amanda and concatenated in one file and passed to gnutar as a --files-from argument. They must start with "./" and containing no other "/"." What does it mean exactly? Strings like "./var/spool/imap/user/[a-t]*" are not accepted in includes? Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: newbie questions about setting up amanda
--On Wednesday, May 11, 2005 00:36:04 +0100 Rodrigo Ventura <[EMAIL PROTECTED]> wrote: > I'm working on setting up a automated backup system using amanda > 2.4.4p4 on a linux slack 8.1 server. The tape system is a HP DAT 72x6 > autoloader (36/72 GB tapes) > > Q1: Using tapetype I got the following entry: > > define tapetype HP-DAT-72x6 { > comment "HP autoloader DAT 72x6" > # data provided by Rodrigo Ventura <[EMAIL PROTECTED]> > length 31255 mbytes > filemark 527 kbytes > speed 1580 kps > } > > How can I setup amanda to be less conservative, i.e. assume it can > record a little bit more than 31GB per tape? Is there a parameter > where I can specify the assumed compression ratio, say, something > between 1:1 and 1:2? You can change the length to be whatever you think will fit on a tape. The tapetype program write random data to the drive, which evidently has hardware compression enabled. Attempting to compress non-compressible data (or already-compressed data) generally makes it larger instead of smaller, thus only 31 GB fitting on a 36 GB (native) tape. You can either turn off hardware compression on the drive and do software compression on selected DLEs (disklist entries), which will enable amanda to very accurately fit dumps on the tape. or you can adjust the length to a size that approximates how much of your data will will fit after the tape drive compresses it. If you guess too low, you may need more tapes than otherwise needed. guess too high and you may unexpectedly hit EOT, causing the taper process to start over on that DLE on the next tape, also causing more tape to be used. If your daily backups are usually smaller than a single tape, or somewhat larger and will need another tape regardless, then it may not matter much either way. Feel free to adjust it anytime as you get more of a feel for your data and tapes. > > Q2: I'm getting "strange" messanges in dump, mostly related with > sockets ignored by gnutar, temporary files (mail queue) being > manipulated, etc. It seems this is the cause of several dumps being > promoted N days ahead... Is there a way to make amanda ignore such > messages? I can live with backups without those files. But I can't > live with amanda promoting dumps several days ahead, all the time > (they never get dumped this way!). If they are being promoted, they ARE being dumped, just the level is being changed sooner than would otherwise occur to fit your dumpcycle. The promotions occur either to balance out daily tape usage (doing more tonight when there's extra room on the tape gives more flexibility tomorrow when a level0 of a large filesystem might be scheduled). It doesn't increase tape usage (I have runtapes set to 3 on a config that normally uses about 3/4 of a tape to allow for autoflushes if necessary, but amanda doesn't promote everything to try and fill 3 tapes, it just uses the 1 it normally uses). The 'strange' messages just let you know about things that didn't get backed up (or possibly were backed up incompletely). There's a file in the source somewhere that lists strings to ignore, but I wouldn't recommend changing it unless you are sure you will never care about it. It is letting you know what will be missing/incomplete when you go to do a restore. > > In other words, what exactly makes amanda promote a dump N days ahead? > How can I know whether the promotion message is safe to ignore or not? You can always ignore the promotion messages, unless you want to tune your bumpsize parameters to change when the promotions take place. > > Q3: I setup amanda in my server, and currently I'm running it > everyday. However, I want to increase the dump cycle, letting amanda > run, say, three times a week. Is it safe to change such parameters as > dumpcycle and runspercycle in mid-cycle? Say, if I change them right > now, will it keep on working without further operations? Yes, although it may take a few runs to balance everything out again. Making the dumpcycle longer or increasing runspercycle shouldn't ever be a problem (assuming your tapecycle is long enough to handle it). Decreasing either drastically may make it impossible to fit everything on the tapes provided, but amanda will warn you if it is unable to meet your schedule or if you are gong to overwrite your last full backup of a DLE. > > Q4: Following the previous question, is it safe to incrementaly add > entries to disklist? Yes, that's the best way to do it. Adding large numbers of DLEs at once may cause some temporary scheduling problems as amanda has to do a full backup of each new one when its added. It will eventually sort itself out (assuming you have enough tapes), but it is simpler to add a few at a time (or one at a time if they are very large as a percentage of your tapesize). Frank > > Thank you, > > Cheers, > > Rodrigo Ventura -- Frank Smith[EMAIL PROTECTED] Sr. Systems Administrator