Problem with amdump and PORT-WRITE mode
I've just run several failed tests on a tape-spanning amdump backup where I did not have enough holding disk space to stage the write to tape, so amanda picked PORT-WRITE mode, and the backup of that filesystem failed. I've run this test 3 times, and all three times it failed with a "Syncpipe failure during tape advance" error. Unfortunately, a google search for that error message turned up no useful results. Here's the pertinent sections from one of the report emails: --- cut here --- FAILURE AND STRANGE DUMP SUMMARY: localhost bigtest_20061203 lev 0 FAILED [data write: Broken pipe] localhost bigtest_20061203 lev 0 FAILED [dump to tape failed] taper: FATAL writer: Syncpipe failure during tape advance FAILED AND STRANGE DUMP DETAILS: /-- localhost bigtest_20061203 lev 0 FAILED [data write: Broken pipe] sendbackup: start [localhost:bigtest_20061203 level 0] sendbackup: info BACKUP=/bin/tar sendbackup: info RECOVER_CMD=/bin/tar -f - ... sendbackup: info end \ NOTES: ... taper: tape DailySet1-030 kb 158843296 fm 33 writing file: No space left on device taper: continuing localhost:dbtest_20061203.0 on new tape from 125829120kb mark: [writing file: No space left on device] taper: tape DailySet1-043 kb 176419104 fm 34 writing file: No space left on device taper: continuing localhost:dbtest_20061203.0 on new tape from 298844160kb mark: [writing file: No space left on device] taper: tape DailySet1-046 kb 56125984 fm 12 writing file: short write driver: taper pid 12151 exited with signal 11 --- cut here --- From the NOTES section, it looks like there was a write error on tape DailySet1-046, but it appears that amanda didn't handle it correctly and continue on, but instead died and failed to dump the rest of the data with the "Syncpipe failure during tape advance". Is this something that should be handled, or is failure to continue expected if there is a tape write error, even if the dump is setup to span tapes and there are more tapes available? It's also a little strange that after the dump died, it appears that the next tape (DailySet1-047) was loaded by the changer successfully, and it even looks like the first file on the tape (the amanda marker file) was updated, as "mt status" showed the tape was positioned at file #2 (file 1 from mt's zero-based counting) and "mt tell" showed it was at block 2 on the tape. Thanks. Evan
Bug? Excessively slow performance of amrecover/amindexd
I have been testing the new 2.5.1p1 (I realize 2.5.1p2 has been released, but amindexd does not appear to have changed between these versions). When attempting to use amrecover to browse an indexed backup of a large filesystem with 360,000 files, each browse operation (setdisk, cd, ls, etc) blocks for an extremely long time (significant fractions of an hour, 20 or more minutes at least). During this time, the amindexd process pegs the cpu. My next smaller backup partition (in the same saveset and tape) contains about 160,000 files, so more than half of the number, and it takes amrecover less than a second to respond to each command. This seems very strange. 160k files takes less than a seccond, yet not even twice as many files takes dozens of minutes? If any of the developers are monitoring the list, can you shed some light on why this may be happening? Is this a bug? Thanks. Evan
Re: How to force amanda to serialize dumper/taper
Unfortunately, I am out of IDE ports on the motherboard, and have no remaining open expansion slots either. Might try a USB drive, but I'd much rather find a software solution rather than a hardware one. Evan On Fri, 8 Dec 2006, Alexander Jolk wrote: Hi, Evan On 12/7/06, Evan Harris <[EMAIL PROTECTED]> wrote: The only way to reliably fix the shoe-shine problem (absent replacement of hardware) [...] Is adding another disk completely out of the question? I have just commissioned a new amanda server with an LTO-3 drive (60MB/s writing speed), and I'm using a software-RAID0 on Linux in order to guarantee bandwidth. That seems to work very fine so far. Since your holding disk is a PATA drive, a second one should not be that expensive. Alex -- Alexander Jolk * +33-1 42 62 31 95 * +33-6 89 65 36 58 [EMAIL PROTECTED]
Re: How to force amanda to serialize dumper/taper
On Thu, 7 Dec 2006, Brian Cuttler wrote: I've assumed that the spindle the amanda work area is on was dedicated to the work area and not split, I know nothing about the architecture That is correct. That disk is only used by amanda. of the specific system bus. If you are able to get through put with amdump than I'd guess its not a bus issue to either the tape nor the drive. That leaves perhaps CPU or sharing the bus access to the disk drive ? It is almost certainly an issue (as I think I stated previously) with the dumper writes and drive seeks between reading from and writing to the same spindle degrading the holding disk throughput enough to cause the disk to be unable to have sufficient bandwidth to keep the tape drive streaming. But I'm going to have to step back at this point, since it sounds like a problem specific to a system that I'm not readily able to visualize. Here's an attempt to explain the issue better: Let's say my holding disk has a throughput of about 20MB/sec. I think everyone can agree that with 20MB/sec of throuhput from the drive and 11MB/sec to the tape, there should be no problem keeping the tape drive streaming, assuming there isn't a CPU or bus bottleneck, and that there is no additional disk contention for the same spindle. We've got 9MB/sec of left-over bandwidth, which is plenty. Now, the tech specs on drive used for my holding disk say that the average seek time is 9ms. That means for every seek done (on average), I lose 184KB of bandwidth. Ok, so my disk has a total of 20MB/sec bandwidth. For the purposes of this discussion, lets assume that the read and write bandwidths are equal. Lets also say that the dumper (for the purposes of this discussion, we have limited amanda to only allow one dumper at a time, not several running concurrently, in order to avoid additional complications) is consuming about 6MB/sec on average (reasonable assumption for dumps coming from a remote machine on a 100mbit network backbone). That leaves 14MB/sec left. To keep the tape drive streaming (from your own specs for uncompressed writes) requres 11MB/sec. That leaves us with 3MB/sec. Now say that interleaving the reads and writes to the disk is causing 25 seeks per second. 25 seeks * 184KB/sec is 4.6MB/sec of throughput lost to seek overhead, and you can see that we are now have a deficit of 1.6MB/sec. From this example, you should be able to see that it is impossible to keep the tape drive streaming unless either the number of seeks or the amount of data being written by the dumper (or both) is reduced. Does that make the issue clear? The only way to reliably fix the shoe-shine problem (absent replacement of hardware) is to keep the dumper from using the disk while taper is using it, or being able to throttle the dumper's use of the disk drive to leave enough bandwidth to keep the taper from starving. And it is appears that neither method of limiting is currently supported by Amanda, which seems very surprising to me. Evan Evan On Thu, 7 Dec 2006, Brian Cuttler wrote: On Wed, Dec 06, 2006 at 04:33:16PM -0600, Evan Harris wrote: Debian testing, SDLT 110/220, Intel P4 2.8Ghz 3gig RAM. Amanda 2.5.1p1-2. The holding disk is a dedicated PATA IDE drive (master) on its own cable/bus (no slave). The tape drive is on its own SCSI bus (no other devices). Bonnie tests on the IDE drive give roughly 20MB/sec, and I have no trouble keeping the tape drive streaming using dd from the IDE drive to the tape drive. Amanda tapetest speed on the tapedrive came in right at 10MB/sec. I have an SDLT 220 also, mine being set for high density but without HW compression, I specifically perform SW compression (client side though in this case the client==server). I am also peaking about 10MB/sec per the amdump report. I have to look at the tape specs... Sun online docs show a sustained transfer rate of 11 MB/Sec, so you and I are both doing pretty well in that dept. Tell me again why you feel your drive is shoe-shining ? Are the specs for your particular drive substantially different ? I'm currently testing on a standalone SDLT drive first. But the final config will be the same type of drive in an ADIC changer. Evan On Wed, 6 Dec 2006, Brian Cuttler wrote: Evan, What OS, type of tape, type of drive, HW platform ? I'm unfamiliar with a parameter to do what your asking for for good measure, what version of Amanda ? Actually, how did you determine that a drive within a changer was shoe-shining ? What else shares the bus with the tape and with the amanda work area drive(s) ? On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote: I'm having a problem with my tape drive shoe-shining because the holding disk can't keep up with the tape drive if it is also being written to by a dumper. Without the extra disk seek overhead of dumpers writing to the holding disk at the
Re: How to force amanda to serialize dumper/taper
I am NOT peaking at 10MB/sec from the amdump report, the 10MB/sec figure is from several amtapetest runs, as well as my own manual testing with dd. I KNOW the tape is shoe-shining, both because I can HEAR it consistently having to reposition the tape every few seconds, and because I'm only getting 2-3MB/sec average throughput when writing to tape while a dumper is running. But I get close to the 10MB/sec when using amflush after all the dumpers have completed (under a stripped down testing config where all the data fits on and is dumped to the holding disk before the tape is brought online). Evan On Thu, 7 Dec 2006, Brian Cuttler wrote: On Wed, Dec 06, 2006 at 04:33:16PM -0600, Evan Harris wrote: Debian testing, SDLT 110/220, Intel P4 2.8Ghz 3gig RAM. Amanda 2.5.1p1-2. The holding disk is a dedicated PATA IDE drive (master) on its own cable/bus (no slave). The tape drive is on its own SCSI bus (no other devices). Bonnie tests on the IDE drive give roughly 20MB/sec, and I have no trouble keeping the tape drive streaming using dd from the IDE drive to the tape drive. Amanda tapetest speed on the tapedrive came in right at 10MB/sec. I have an SDLT 220 also, mine being set for high density but without HW compression, I specifically perform SW compression (client side though in this case the client==server). I am also peaking about 10MB/sec per the amdump report. I have to look at the tape specs... Sun online docs show a sustained transfer rate of 11 MB/Sec, so you and I are both doing pretty well in that dept. Tell me again why you feel your drive is shoe-shining ? Are the specs for your particular drive substantially different ? I'm currently testing on a standalone SDLT drive first. But the final config will be the same type of drive in an ADIC changer. Evan On Wed, 6 Dec 2006, Brian Cuttler wrote: Evan, What OS, type of tape, type of drive, HW platform ? I'm unfamiliar with a parameter to do what your asking for for good measure, what version of Amanda ? Actually, how did you determine that a drive within a changer was shoe-shining ? What else shares the bus with the tape and with the amanda work area drive(s) ? On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote: I'm having a problem with my tape drive shoe-shining because the holding disk can't keep up with the tape drive if it is also being written to by a dumper. Without the extra disk seek overhead of dumpers writing to the holding disk at the same time, the holding disk should be plenty fast enough to keep the tape drive streaming. Is there any way I can force amanda to serialize the dumper/taper so that they are never run concurrently? I've already set inparallel to 1, but that only affects how many dumpers can run, not the taper. I've also tried increasing the tapebufs parameter to 8000 (256MiB) to see if that would at least let the drive stay streaming for longer periods, but if it made any difference, it wasn't significant. What thresholds does the taper use to decide when the tapebufs are filled enough to start tape motion? There doesn't seem to be any docs on that, or settings to customize. I did get a suggestion that I should just leave the tape out of the drive, let the dumpers fill the holding disk and then load the tape and run amflush, but that doesn't really work when using a changer, plus the holding disk isn't large enough for the total size of all the backups, though it can fit them one-by-one. Seems like there should be a "speed" config option for holding disks like there is for network interfaces and tape drives, so that amanda could test to see if the holding disk can't handle dumpers using the holding disk at the same time a taper is running. That seems like it'd solve the problem nicely, and even seems to fit with the scheme amanda uses for network interfaces. Thanks. Evan --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773
Re: How to force amanda to serialize dumper/taper
Unfortunately, eliminating (or drastically reducing) the holding disk workaround won't solve the problem in my case, since several of my larger dumps are from nonlocal disks, and the machines they are on cannot sustain a write across the network that is fast enough to keep the tape drive streaming. It would just move the problem from not being fast enough to stream from a local disk to not being fast enough to stream from remote, resulting in the same shoe-shine problem. In a few cases, where a dump contains large directories filled with many small files, even local drives cannot keep the tape drive streaming because of the directory/stat lookup times. That is, without using a holding disk to stage them. Evan On Wed, 6 Dec 2006, Jaz Singh wrote: My experience is the same. For large dumps, it was significantly slower to allow dumper and taper to work at the same time. I fixed this by limiting the holding disk to 5GB. Our holding disk is a dedicated 300GB SCSi drive. This allows the small incrementals to build up but forces the 200GB+ dumps to go straight to tape. Our tape loader (Quantum M1500) pushes at about 25MB/sec when tar-ing straight to tape. When dumping to holding disk while taping at the same time, this drops down to 10MB/sec. For a backup that normally takes 8 hours on a weekday, this is too slow. This is all local disk backup of about 800GB per weeknight and 2.5TB on Friday. Network backups will probably benefit from larger holding disk, since the network limits the transfer to holding. On Wed, 6 Dec 2006, Brian Cuttler wrote: Evan, What OS, type of tape, type of drive, HW platform ? I'm unfamiliar with a parameter to do what your asking for for good measure, what version of Amanda ? Actually, how did you determine that a drive within a changer was shoe-shining ? What else shares the bus with the tape and with the amanda work area drive(s) ? On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote: I'm having a problem with my tape drive shoe-shining because the holding disk can't keep up with the tape drive if it is also being written to by a dumper. Without the extra disk seek overhead of dumpers writing to the holding disk at the same time, the holding disk should be plenty fast enough to keep the tape drive streaming. Is there any way I can force amanda to serialize the dumper/taper so that they are never run concurrently? I've already set inparallel to 1, but that only affects how many dumpers can run, not the taper. I've also tried increasing the tapebufs parameter to 8000 (256MiB) to see if that would at least let the drive stay streaming for longer periods, but if it made any difference, it wasn't significant. What thresholds does the taper use to decide when the tapebufs are filled enough to start tape motion? There doesn't seem to be any docs on that, or settings to customize. I did get a suggestion that I should just leave the tape out of the drive, let the dumpers fill the holding disk and then load the tape and run amflush, but that doesn't really work when using a changer, plus the holding disk isn't large enough for the total size of all the backups, though it can fit them one-by-one. Seems like there should be a "speed" config option for holding disks like there is for network interfaces and tape drives, so that amanda could test to see if the holding disk can't handle dumpers using the holding disk at the same time a taper is running. That seems like it'd solve the problem nicely, and even seems to fit with the scheme amanda uses for network interfaces. Thanks. Evan --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773
Re: How to force amanda to serialize dumper/taper
Debian testing, SDLT 110/220, Intel P4 2.8Ghz 3gig RAM. Amanda 2.5.1p1-2. The holding disk is a dedicated PATA IDE drive (master) on its own cable/bus (no slave). The tape drive is on its own SCSI bus (no other devices). Bonnie tests on the IDE drive give roughly 20MB/sec, and I have no trouble keeping the tape drive streaming using dd from the IDE drive to the tape drive. Amanda tapetest speed on the tapedrive came in right at 10MB/sec. I'm currently testing on a standalone SDLT drive first. But the final config will be the same type of drive in an ADIC changer. Evan On Wed, 6 Dec 2006, Brian Cuttler wrote: Evan, What OS, type of tape, type of drive, HW platform ? I'm unfamiliar with a parameter to do what your asking for for good measure, what version of Amanda ? Actually, how did you determine that a drive within a changer was shoe-shining ? What else shares the bus with the tape and with the amanda work area drive(s) ? On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote: I'm having a problem with my tape drive shoe-shining because the holding disk can't keep up with the tape drive if it is also being written to by a dumper. Without the extra disk seek overhead of dumpers writing to the holding disk at the same time, the holding disk should be plenty fast enough to keep the tape drive streaming. Is there any way I can force amanda to serialize the dumper/taper so that they are never run concurrently? I've already set inparallel to 1, but that only affects how many dumpers can run, not the taper. I've also tried increasing the tapebufs parameter to 8000 (256MiB) to see if that would at least let the drive stay streaming for longer periods, but if it made any difference, it wasn't significant. What thresholds does the taper use to decide when the tapebufs are filled enough to start tape motion? There doesn't seem to be any docs on that, or settings to customize. I did get a suggestion that I should just leave the tape out of the drive, let the dumpers fill the holding disk and then load the tape and run amflush, but that doesn't really work when using a changer, plus the holding disk isn't large enough for the total size of all the backups, though it can fit them one-by-one. Seems like there should be a "speed" config option for holding disks like there is for network interfaces and tape drives, so that amanda could test to see if the holding disk can't handle dumpers using the holding disk at the same time a taper is running. That seems like it'd solve the problem nicely, and even seems to fit with the scheme amanda uses for network interfaces. Thanks. Evan --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773
How to force amanda to serialize dumper/taper
I'm having a problem with my tape drive shoe-shining because the holding disk can't keep up with the tape drive if it is also being written to by a dumper. Without the extra disk seek overhead of dumpers writing to the holding disk at the same time, the holding disk should be plenty fast enough to keep the tape drive streaming. Is there any way I can force amanda to serialize the dumper/taper so that they are never run concurrently? I've already set inparallel to 1, but that only affects how many dumpers can run, not the taper. I've also tried increasing the tapebufs parameter to 8000 (256MiB) to see if that would at least let the drive stay streaming for longer periods, but if it made any difference, it wasn't significant. What thresholds does the taper use to decide when the tapebufs are filled enough to start tape motion? There doesn't seem to be any docs on that, or settings to customize. I did get a suggestion that I should just leave the tape out of the drive, let the dumpers fill the holding disk and then load the tape and run amflush, but that doesn't really work when using a changer, plus the holding disk isn't large enough for the total size of all the backups, though it can fit them one-by-one. Seems like there should be a "speed" config option for holding disks like there is for network interfaces and tape drives, so that amanda could test to see if the holding disk can't handle dumpers using the holding disk at the same time a taper is running. That seems like it'd solve the problem nicely, and even seems to fit with the scheme amanda uses for network interfaces. Thanks. Evan
