Re: Questions and Answers and Thanks
* Chris Marble <[EMAIL PROTECTED]> (Wed, Feb 21, 2001 at 09:51:32PM -0800) > Gerhard den Hollander wrote: >> Since it's a linux box , you may want to look into using reiserFS. >> I found that file access with Reiser is faster than ext2 (although nothing >> shocking) esp. when going through a lot of smaller files in one swoop. > I formatted up the holding disk with a large blocksize and small inode > count. It's never going to have over 100 files on it so I doubt > ReiserFS is worth the hassle. I guess not, Reiser comes prepackaged with Suse, so in my case it wasn't a hassle at all . But my suggestion was making the disk to be backed up use reiser iso the holding disk. > I do all my backups with dump, ufsdump > or xfsdump now. I'd just as soon avoid adding in gnu tar too. Well, amanda handles it all transparently for you. Except that getting estimates with gnutar takes forever and then some. > Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- __O You are a fluke of the universe ... =`\<, You have no right to be here. (=)/(=) Whether you can hear it or not, the universe Is laughing behind your back.
Re: Questions and Answers and Thanks
Gerhard den Hollander wrote: > > Since it's a linux box , you may want to look into using reiserFS. > I found that file access with Reiser is faster than ext2 (although nothing > shocking) esp. when going through a lot of smaller files in one swoop. I formatted up the holding disk with a large blocksize and small inode count. It's never going to have over 100 files on it so I doubt ReiserFS is worth the hassle. I do all my backups with dump, ufsdump or xfsdump now. I'd just as soon avoid adding in gnu tar too. Though since I've got one Digital Unix box where the vfsdump doesn't work... -- [EMAIL PROTECTED] - HMC UNIX Systems Manager My opinions are my own and probably don't represent anything anyway.
Re: Questions and Answers and Thanks
* Chris Marble <[EMAIL PROTECTED]> (Sat, Feb 17, 2001 at 07:59:17PM -0800) > Gerhard den Hollander wrote: >> >> I noticed when looking through the logs that the *dumper* performance is >> around the 1Kps (or less) on my system , >> whereas the *taper* gets a performance of about 7 Kps. >> So apparently it's not the speedn of my tapedrive that's the bottleneck, >> but soemthing within amanda. >> I have now turned compression off and im hoping this will imporve matters. >> Are there any other ways to tweak amanda.conf to improve dumper performance > Best of luck finding your current bottleneck. I did some experiments (I still need to write this out and stick it on a proper website or somesuch :) ) Anyway, the bottleneck is (In my case) the mutli pipe that amanda does tar cvf - +---> holdingdisk | +---> | tar tvf - --> index daemon Sticking gzip in there makes things even worse. What I did is: - Make sure I have a big holding disk - Patch/Hack sendsize to use the calcsize executable iso gnutar to calculate estimates (estimate generation went down frm well over 4 hours to 45 minutes) - set maxdumps to 4 (having 4 dumps in paralel to holding disk) - dump everything from holding disk to tape. With this I get upto 11K performance to tape And while the avergae performance of a backup to dumper has dropped to ~ 2.5 Ks (was 3K whitout compression, 1K with) I have 4 of them inparalel giving approx 10Ks > I never got the bandwidth setting corect on my setup so I have a LOT of > different dump-types with different maxdumps and dumpcycle settings. > I've got 36Gb of holding disk and a single 50Gb AIT-2 drive. One of the > 18 machines I'm backing up has a 36Gb drive with 30Gb of already gzipped > data on it. I work to keep the full backups of that drive confined to > weekends. Or use guntar, and split the big disk in smaller gnutarrable slices. That's what I did with the 420G disk, I've made a bunch of slices all of which are < 50G and I have 50+ G of holding disk space (on that same disk). > My backup host is a Dell Precision 610 with 2 SCSI cards. I'm going to > move the tape drive to the internal controlled since my holding disks are > one the external one. Maybe I'll pick up some speed there. > The holding disk is 4 9Gb drives striped with software RAID. I formatted > for large block sizes and few inodes to improve large file performance. > I'm running RedHat Linux 6.2 with kernel 2.2.16-8. Machine's got a single > 500MHz Xeon CPU and 256Mb RAM. I know a faster CPU will help during the > gzip phase. Will more RAM? Depends on what else you are running. I noticed that a 500Mhz can give rather excellent theoughput with gzip -fast. Since it's a linux box , you may want to look into using reiserFS. I found that file access with Reiser is faster than ext2 (although nothing shocking) esp. when going through a lot of smaller files in one swoop. Gerhard, <@jasongeo.com> == The Acoustic Motorbiker == -- and family <| |Oo. __O__0 |__/-=`\<, =`\<, (*) (=)/(=)(=)/(=)
Re: Questions and Answers and Thanks
Gerhard den Hollander wrote: > > I noticed when looking through the logs that the *dumper* performance is > around the 1Kps (or less) on my system , > whereas the *taper* gets a performance of about 7 Kps. > > So apparently it's not the speedn of my tapedrive that's the bottleneck, > but soemthing within amanda. > > I have now turned compression off and im hoping this will imporve matters. > > Are there any other ways to tweak amanda.conf to improve dumper performance Best of luck finding your current bottleneck. I never got the bandwidth setting corect on my setup so I have a LOT of different dump-types with different maxdumps and dumpcycle settings. I've got 36Gb of holding disk and a single 50Gb AIT-2 drive. One of the 18 machines I'm backing up has a 36Gb drive with 30Gb of already gzipped data on it. I work to keep the full backups of that drive confined to weekends. My backup host is a Dell Precision 610 with 2 SCSI cards. I'm going to move the tape drive to the internal controlled since my holding disks are one the external one. Maybe I'll pick up some speed there. The holding disk is 4 9Gb drives striped with software RAID. I formatted for large block sizes and few inodes to improve large file performance. I'm running RedHat Linux 6.2 with kernel 2.2.16-8. Machine's got a single 500MHz Xeon CPU and 256Mb RAM. I know a faster CPU will help during the gzip phase. Will more RAM? -- [EMAIL PROTECTED] - HMC UNIX Systems Manager
RE: Questions and Answers and Thanks
I've found that when gzip is running on my own, and clients systems it's usually using 95+% of one cpu, which isn't a problem for us or them because the machines either have a spare cpu, or aren't doing anything at the time. more of a problem for bigger sites where your server will always be busy... The dumper itself usually takes very little cpu time. --- >I still noticed that the dumper process is the bottleneck, but dumper now >gets a performance of about 3 - 4 Kbps (iso sometimes down to 250bps). Are you sure it's dumper? Or is that just the symptom? For instance, it could be the dump program on the client, disks on the client, SCSI termination (or lack there of), bad cables, bad controller, your network connection, other workload on the system at the same time, disk/controller/bus contention between the backed up disk and holding disk, ditto for the tape drive if the image goes direct to tape, etc. >Gerhard den Hollander John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Questions and Answers and Thanks
>Are there any other ways to tweak amanda.conf to improve dumper performance >(I've set the bandwith parameters to large values since it's on local disks >and I've specified local in my disklist. Bandwidth only affects starting a dump, not anything while the dump is running. >I have lsof running on Sol 2.6 7 and 8 and it rules, >now if only your boss would decide to port it to Irix 6.5 . It may be because he does not have access to such a machine. Have you asked him/volunteered access? Or sent him the explicit errors you're getting to see if he has any ideas? >A normal amanda session with compression takes between 10 and 15 hours, >running whitout compression finished in 3.5 hours . Did you use "fast" or "best" compression? I've been told, and had some experience, that "best" often does not buy you much more space but costs a large amount of time. How much compression were you getting (it's listed in the E-mail)? If you were not getting much (i.e. the data being dumped was already compressed), that might also have an effect. >I still noticed that the dumper process is the bottleneck, but dumper now >gets a performance of about 3 - 4 Kbps (iso sometimes down to 250bps). Are you sure it's dumper? Or is that just the symptom? For instance, it could be the dump program on the client, disks on the client, SCSI termination (or lack there of), bad cables, bad controller, your network connection, other workload on the system at the same time, disk/controller/bus contention between the backed up disk and holding disk, ditto for the tape drive if the image goes direct to tape, etc. >Gerhard den Hollander John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Questions and Answers and Thanks
On Fri, Jan 19, 2001 at 09:59:57AM +0100, Gerhard den Hollander wrote: > Hmm, > > Favor #1: can the reply to adress on the digests be set to @amanda.org ;) > > * [EMAIL PROTECTED] <[EMAIL PROTECTED]> (Fri, Jan 19, 2001 at >06:34:02AM -) unsubscribe from [EMAIL PROTECTED] and subscribe to [EMAIL PROTECTED] The @egroups.com lists should be use only for archiving, no one should subscribe or send mail to them. Jean-Louis -- Jean-Louis Martineau email: [EMAIL PROTECTED] Departement IRO, Universite de Montreal C.P. 6128, Succ. CENTRE-VILLETel: (514) 343-6111 ext. 3529 Montreal, Canada, H3C 3J7Fax: (514) 343-5834
