Evan, On Thu, Dec 07, 2006 at 04:52:52PM -0600, Evan Harris wrote: > > On Thu, 7 Dec 2006, Brian Cuttler wrote: > > >I've assumed that the spindle the amanda work area is on was dedicated > >to the work area and not split, I know nothing about the architecture > > That is correct. That disk is only used by amanda. > > >of the specific system bus. If you are able to get through put with > >amdump than I'd guess its not a bus issue to either the tape nor the > >drive. That leaves perhaps CPU or sharing the bus access to the disk > >drive ? > > It is almost certainly an issue (as I think I stated previously) with the > dumper writes and drive seeks between reading from and writing to the same > spindle degrading the holding disk throughput enough to cause the disk to > be unable to have sufficient bandwidth to keep the tape drive streaming. > > >But I'm going to have to step back at this point, since it sounds like > >a problem specific to a system that I'm not readily able to visualize. > > Here's an attempt to explain the issue better: > > Let's say my holding disk has a throughput of about 20MB/sec. I think > everyone can agree that with 20MB/sec of throuhput from the drive and > 11MB/sec to the tape, there should be no problem keeping the tape drive > streaming, assuming there isn't a CPU or bus bottleneck, and that there is > no additional disk contention for the same spindle. We've got 9MB/sec of > left-over bandwidth, which is plenty. > > Now, the tech specs on drive used for my holding disk say that the average > seek time is 9ms. That means for every seek done (on average), I lose > 184KB of bandwidth. > > Ok, so my disk has a total of 20MB/sec bandwidth. For the purposes of this > discussion, lets assume that the read and write bandwidths are equal. Lets > also say that the dumper (for the purposes of this discussion, we have > limited amanda to only allow one dumper at a time, not several running > concurrently, in order to avoid additional complications) is consuming > about 6MB/sec on average (reasonable assumption for dumps coming from a > remote machine on a 100mbit network backbone). That leaves 14MB/sec left. > To keep the tape drive streaming (from your own specs for uncompressed > writes) requres 11MB/sec. That leaves us with 3MB/sec. > > Now say that interleaving the reads and writes to the disk is causing 25 > seeks per second. 25 seeks * 184KB/sec is 4.6MB/sec of throughput lost to > seek overhead, and you can see that we are now have a deficit of 1.6MB/sec. > From this example, you should be able to see that it is impossible to keep > the tape drive streaming unless either the number of seeks or the amount of > data being written by the dumper (or both) is reduced. > > Does that make the issue clear? The only way to reliably fix the > shoe-shine problem (absent replacement of hardware) is to keep the dumper > from using the disk while taper is using it, or being able to throttle the > dumper's use of the disk drive to leave enough bandwidth to keep the taper > from starving. And it is appears that neither method of limiting is > currently supported by Amanda, which seems very surprising to me.
Are there performance enhancements available by using a different type of file system on the amanda work area drive ? Anything else I can think of involves new hardware, mirrors, multiple amanda work areas... I have no idea how to set (active dumpers + tapers) = 1 > Evan > > >>Evan > >> > >>On Thu, 7 Dec 2006, Brian Cuttler wrote: > >> > >>>On Wed, Dec 06, 2006 at 04:33:16PM -0600, Evan Harris wrote: > >>>> > >>>>Debian testing, SDLT 110/220, Intel P4 2.8Ghz 3gig RAM. Amanda > >>>>2.5.1p1-2. > >>>> > >>>>The holding disk is a dedicated PATA IDE drive (master) on its own > >>>>cable/bus (no slave). The tape drive is on its own SCSI bus (no other > >>>>devices). Bonnie tests on the IDE drive give roughly 20MB/sec, and I > >>>>have > >>>>no trouble keeping the tape drive streaming using dd from the IDE drive > >>>>to > >>>>the tape drive. Amanda tapetest speed on the tapedrive came in right at > >>>>10MB/sec. > >>> > >>>I have an SDLT 220 also, mine being set for high density but without > >>>HW compression, I specifically perform SW compression (client side though > >>>in this case the client==server). I am also peaking about 10MB/sec per > >>>the amdump report. > >>> > >>>I have to look at the tape specs... Sun online docs show a sustained > >>>transfer rate of 11 MB/Sec, so you and I are both doing pretty well > >>>in that dept. > >>> > >>>Tell me again why you feel your drive is shoe-shining ? > >>>Are the specs for your particular drive substantially different ? > >>> > >>> > >>>>I'm currently testing on a standalone SDLT drive first. But the final > >>>>config will be the same type of drive in an ADIC changer. > >>>> > >>>>Evan > >>>> > >>>>On Wed, 6 Dec 2006, Brian Cuttler wrote: > >>>> > >>>>>Evan, > >>>>> > >>>>>What OS, type of tape, type of drive, HW platform ? > >>>>> > >>>>>I'm unfamiliar with a parameter to do what your asking > >>>>>for for good measure, what version of Amanda ? > >>>>> > >>>>>Actually, how did you determine that a drive within a changer > >>>>>was shoe-shining ? What else shares the bus with the tape and > >>>>>with the amanda work area drive(s) ? > >>>>> > >>>>>On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote: > >>>>>> > >>>>>>I'm having a problem with my tape drive shoe-shining because the > >>>>>>holding > >>>>>>disk can't keep up with the tape drive if it is also being written to > >>>>>>by a > >>>>>>dumper. Without the extra disk seek overhead of dumpers writing to > >>>>>>the > >>>>>>holding disk at the same time, the holding disk should be plenty fast > >>>>>>enough to keep the tape drive streaming. > >>>>>> > >>>>>>Is there any way I can force amanda to serialize the dumper/taper so > >>>>>>that > >>>>>>they are never run concurrently? I've already set inparallel to 1, > >>>>>>but > >>>>>>that only affects how many dumpers can run, not the taper. > >>>>>> > >>>>>>I've also tried increasing the tapebufs parameter to 8000 (256MiB) to > >>>>>>see > >>>>>>if that would at least let the drive stay streaming for longer > >>>>>>periods, > >>>>>>but > >>>>>>if it made any difference, it wasn't significant. What thresholds > >>>>>>does > >>>>>>the > >>>>>>taper use to decide when the tapebufs are filled enough to start tape > >>>>>>motion? There doesn't seem to be any docs on that, or settings to > >>>>>>customize. > >>>>>> > >>>>>>I did get a suggestion that I should just leave the tape out of the > >>>>>>drive, > >>>>>>let the dumpers fill the holding disk and then load the tape and run > >>>>>>amflush, but that doesn't really work when using a changer, plus the > >>>>>>holding disk isn't large enough for the total size of all the backups, > >>>>>>though it can fit them one-by-one. > >>>>>> > >>>>>>Seems like there should be a "speed" config option for holding disks > >>>>>>like > >>>>>>there is for network interfaces and tape drives, so that amanda could > >>>>>>test > >>>>>>to see if the holding disk can't handle dumpers using the holding disk > >>>>>>at > >>>>>>the same time a taper is running. That seems like it'd solve the > >>>>>>problem > >>>>>>nicely, and even seems to fit with the scheme amanda uses for network > >>>>>>interfaces. > >>>>>> > >>>>>>Thanks. > >>>>>> > >>>>>>Evan > >>>>>--- > >>>>>Brian R Cuttler [EMAIL PROTECTED] > >>>>>Computer Systems Support (v) 518 486-1697 > >>>>>Wadsworth Center (f) 518 473-6384 > >>>>>NYS Department of Health Help Desk 518 473-0773 > >>>>> > >>>--- > >>> Brian R Cuttler [EMAIL PROTECTED] > >>> Computer Systems Support (v) 518 486-1697 > >>> Wadsworth Center (f) 518 473-6384 > >>> NYS Department of Health Help Desk 518 473-0773 > >>> > >--- > > Brian R Cuttler [EMAIL PROTECTED] > > Computer Systems Support (v) 518 486-1697 > > Wadsworth Center (f) 518 473-6384 > > NYS Department of Health Help Desk 518 473-0773 > > --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support (v) 518 486-1697 Wadsworth Center (f) 518 473-6384 NYS Department of Health Help Desk 518 473-0773
