Re: tape filling algorithm? (flush, or dump)
>>>On Wed, Nov 20, 2002 at 01:47:50PM -0600, Deb Baddorf wrote:
What kind of optimiaztion does amanda use while flushing
dumps to tape?
>>>There is no optimazation, first in, first out.
>>>That's a need feature.
>>>
>>>Jean-Louis
Here's a basic algorithm (in pseudo-code).
Is this useful to somebody who is already working on this code?
Or should I attempt to fit it into the actual taper code myself?
Deb Baddorf
=
This *should* handle both the FLUSH case (all data is available from
the start) and the on-going AMDUMP case (data is added as we go).
I may notice more errors but I think I got them all.
To "grok" the basic algorithm (ridiculously simple) ignore
or white out the "SPIN" paragraphs which wait for more data to
arrive.
===
stuff = collection of items to be put into knapsack.
Dumpfiles in this case (includes tarfiles too)
Stuff is probably a doubly(?) linked list. Operations required:
status = insert(item)
status = remove(ptr) or maybe remove(item)
ptr = get_largest()
#boolean = is_empty()
boolean = has_data() #opposite sense to is_empty()
boolean = more_coming() #somebody elsewhere needs to tell us when no more
dumps are coming down the pipe
item
number = size(item)
ptr = get_next_smaller()# return item next smaller than self
(maybe should be a function of the COLLECTION instead of the item,
for project OO code)
next_ptr #if you use a doubly linked list
prev_ptr #if you use a doubly linked list
string = get_name(item) ?? for the index, or some such. Maybe for
error message when dump won't fit.
knapsack#(or, tape!)
number = size_remaining()
status = add(item) # responsible to DELETE from holding disk after
successful & from COLLECTION above
make_full()# decree it "full" when no available items can fit
in the remaining space
boolean = has_space()# is not yet full
bunch-o-knapsacks #(or, how many tapes are available?)
ptr = gimme_one()
boolean = is_more()
==
PACK_ALL_STUFF #i.e. start taping
{
mystuff = new stuff() # filled somewhere else, or here, but before
algorithm below
allsacks = new bunch-o-knapsacks() #number comes from RUNTAPES parameter
yada yada yada
maybe twiddle thumbs till significant data arrives, if holding disk allows
while ( allsacks.is_more() # stop when we run out of sacks (tapes)
or (not mystuff.has_data() # or when we run out of data
and not mystuff.more_coming() ) )# and know that no more
is coming
{
this_sack = allsacks.gimme_one()
while ( not mystuff.has_data() ) #wait for some data to arrive
{spin
check "more coming" in case a client dies
}
while ( mystuff.has_data()and #there is still data waiting to
be taped
this_sack.has_space() ) #the sack (tape) has room left
{
#remember, in "timed" version, a bigger one may be added during
any pass
#so, we start with biggest each time and add the first that
will fit
fill_sack (this_sack, mystuff.get_largest())
while ( not mystuff.has_data() ) #wait for some more data to arrive
{spin
check "more coming" in case a client dies
}
}
}
}
==
fill_sack(sackptr, item_ptr)
{
if (item_ptr is NULL) #there are no more smaller items
{
sackptr.make_full() #decree sack (tape) to be full, so we can
move on to another
return
}
isize=item_ptr.size(); ssize=sackptr.size_remaining(); #store for
possible error msg
#or assign to variable IN the if statement, but too messy to
do in psuedo-code here!
if ( item_ptr.size() <= sackptr.size_remaining() )#if item fits,
according to sack specs,
{ # put it in the sack
status = sackptr.add(item_ptr)# add MUST delete from holding
disk, if it finishes successfully
# add MUST also adjust
sack's size_remaining value
# it writes the INDEX, and does
other bookkeeping too
# Also removes ITEM from "stuff"
list of available dumps to flush
if ( not status) # failed -- tape size must've been
slightly wrong. That's life.
{
sackptr.make_full() # we've already written to EOT so
can't try another size item
write message "Sack(tape) wasn't as big as " isize + ssize.
write message "May want to correct sack specs (tapesize)"
}
}
else #find next smaller item and try that one
{
fill_sack (sackptr, item_ptr.get_next_smaller())
}
}
Re: chg-scsi and Quantum SuperLoader SDLT 3520L
On Thursday 21 November 2002 14:04, Damon Conway wrote: >On Wed, Nov 20, 2002 at 10:22:07PM -0500, Gene Heskett wrote: >> On Wednesday 20 November 2002 17:10, Damon Conway wrote: >> >Hello all, >> >I've been wrestling with chg-scsi and my Quantum changer all >> > day. Here are the specs: >> > >> >Red Hat 8.0 w/ kernel 2.4.18 >> >Quantum SuperLoader SDLT 3520L >> >Amanda 2.4.3 >> > >> >If I set changerdev to /dev/sg0 and dev to /dev/nst0, I get >> > this: >> > >> ># sudo /usr/local/libexec/chg-scsi -info >> >Can't get ElementStatus >> ># >> > >> >I can't find anything on that error in the list archives, >> > searching Google, or looking in the docs (TAPE.CHANGERS or >> > FAQ-O-MATIC). According to /proc/scsi/sg/device*, /dev/sg0 is >> > "QUANTUM SuperDLT1 2A2A". I'm assuming that the error means >> > that it can't talk to the device. >> >> One thing that can trip up the unwary is that, depending on the >> drive, the changer robot may be at the same bus address as the >> drive, but is lun=1 where the drive is lun=0. Thats the case >> here at my place. >> >> The tripup is that the stock kernel dos not AFAIK, do a "scan >> all luns" at startup, thereby missing the changer robot. This >> is an option you may have to turn on and rebuild the kernel to >> make work if thats the case with your changer. > >Ah ha! This solved the problem. See, I do manage to hit one once in a while. :-) > Thanks. I'll see about adding >something about this to docs/TAPE.CHANGERS and creating a diff. > Where does stuff like that get sent? There is the option to edit the FAQ-O-MATIC itself, exersizable by almost anyone. I tried it once, but the editor was worse than the old ms 'ed' for being about 3000% non-intuitive, so I wound up screwing it up and had to yell for help. Someone who is familiar with that editor could probably just type it in a couple of minutes. But obviously I wasn't that someone, never having done battle with it before. But, for the brave, its there. -- Cheers Damon, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.19% setiathome rank, not too shabby for a WV hillbilly
Re: DAT hardware or software compression
[EMAIL PROTECTED] wrote: > Thanks for your quick answer. Ok it sounds clear to me that all points to > the advantages of having software compression. I will then disable hardware > compression. If there is one flipped bit (read error), everything compressed with gzip after that position is lost. With HW compression you can continue restoring after that position plus some overhead. I use HW compression as long as the backup utility can not gzip every file independantly or use an algorithm that can recover after a read error. Sven
Re: Want more aggressive tape use
On Thu, Nov 21, 2002 at 11:48:56AM -0700, Orion Poplawski wrote: > > That's not really what I'm after. I'd like to have amanda to backup as > little as possible (i.e. high level backups, few level 0's -> long > dumpcycle), but I want it to stuff tapes until they are full. Take this as a given fact. Amanda does NOT add to tapes it has already used. It ONLY will overwrite from the beginning of the tape. That given, the closest thing to what you want to do is to let the dumps collect on the disk until you have more than a tape's worth. Then do your flush. > However, > currently when you run out of tape you are left with dumps in the > holding area that need to get flushed. I don't want to flush them, but > if you don't amanda believes that those backups have been completed and > figures it in to planning the next run. Those backups have been completed. Just not moved to tape. You can leave them there for a future flush. How else would you have amanda consider those completed dumps not yet flushed? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: One (of six) partitions returning "disk offline"
On Thu, Nov 21, 2002 at 09:46:36AM -0500, Joshua Baker-LePain wrote: > Nope. / has an entry in fstab just like everybody else. Well, not *just* like *everybody* else: :-) Solaris: #devicedevice mount FS fsckmount mount #to mount to fsck point typepassat boot options /dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no logging /dev/dsk/c1t3d0s7 /dev/rdsk/c1t3d0s7 /local ufs 2 yes logging Linux: LABEL=/ / ext3defaults1 1 LABEL=/local/local ext3defaults1 2 But you're right, I was wrong, sorry for the misinformation. -- Jay Lessert [EMAIL PROTECTED] Accelerant Networks Inc. (voice)1.503.439.3461 Beaverton OR, USA(fax)1.503.466.9472
Re: COMPAQ TSL-9000 DAT autoloader device
On Thu, Nov 21, 2002 at 01:55:27PM +0100, [EMAIL PROTECTED] wrote: > Hello, > > Does anyone have experience in configuration a COMPAQ/SONY TSL-9000 DAT > DDS-3 device with AMANDA ? I have only the device and no single > documentation about it. What I can tell is that it's an 8 DDS-3 tape > changer device which gets recognized on my Solaris 8 system under > /dev/rmt/0. Somehow it doesn't have two devices in it (as usual where you > have one device for the changer and one device for the tape) so that's > already something weird. I've tried to use mtx but mtx doesn't seem to like > the device at all. Once you know exactly what the target/lun setup for the changer is (you *really* need documentation), you can proceed. In any case, you should do: setenv auto-boot? false reset-all probe-scsi-all setenv auto-boot? true It is either two targets, or one target with 2 luns. On Solaris 8, you'll need to configure the sgen driver for the robot. In the mtx distribution, see contrib/config_sgen_solaris.sh. Do not just run the script please (there's a cvs command that will fail, and you may need lun!=0), read and understand first. You'll end up with a device like: /dev/scsi/changer/cNtMd0, that's what you point mtx (or chg-scsi, I suppose) at. -- Jay Lessert [EMAIL PROTECTED] Accelerant Networks Inc. (voice)1.503.439.3461 Beaverton OR, USA(fax)1.503.466.9472
Re: chg-scsi and Quantum SuperLoader SDLT 3520L
On Wed, Nov 20, 2002 at 10:22:07PM -0500, Gene Heskett wrote: > On Wednesday 20 November 2002 17:10, Damon Conway wrote: > >Hello all, > >I've been wrestling with chg-scsi and my Quantum changer all day. > > Here are the specs: > > > >Red Hat 8.0 w/ kernel 2.4.18 > >Quantum SuperLoader SDLT 3520L > >Amanda 2.4.3 > > > >If I set changerdev to /dev/sg0 and dev to /dev/nst0, I get this: > > > ># sudo /usr/local/libexec/chg-scsi -info > >Can't get ElementStatus > ># > > > >I can't find anything on that error in the list archives, > > searching Google, or looking in the docs (TAPE.CHANGERS or > > FAQ-O-MATIC). According to /proc/scsi/sg/device*, /dev/sg0 is > > "QUANTUM SuperDLT1 2A2A". I'm assuming that the error means > > that it can't talk to the device. > > > One thing that can trip up the unwary is that, depending on the > drive, the changer robot may be at the same bus address as the > drive, but is lun=1 where the drive is lun=0. Thats the case here > at my place. > > The tripup is that the stock kernel dos not AFAIK, do a "scan all > luns" at startup, thereby missing the changer robot. This is an > option you may have to turn on and rebuild the kernel to make work > if thats the case with your changer. Ah ha! This solved the problem. Thanks. I'll see about adding something about this to docs/TAPE.CHANGERS and creating a diff. Where does stuff like that get sent? Thanks, Damon -- GPG Key Info: http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA7F06E8E Key fingerprint = E899 017C 83C9 110B 10E9 44F6 6AD9 6D61 A7F0 6E8E "Sometimes your temperature, it really gets higher, and the music sets your soul on fire." - Peter Tosh 'Reggae Mylitis' msg16203/pgp0.pgp Description: PGP signature
Re: Want more aggressive tape use
Chris Marble wrote: Orion Poplawski wrote: A lot of file systems were not backed up that could have been because the planner rejected them. I'd like to be able to operate in a mode where it kept dumping until the tape was full, but then threw away whatever was left (i.e. - no amflush -> inefficient use of another tape). If you crank down dumpcycle in amanda.conf the program will do its best to fit more on each tape. If you crank it down too far you'll get daily complaints about it having to delay backups but it'll still do what it can. That's what I've done to maximize tape usage in my setup: single 50Gb AIT tape/day, 300Gb to back up. Tell it to manage full backups of everything over 4 days of backups. That's not really what I'm after. I'd like to have amanda to backup as little as possible (i.e. high level backups, few level 0's -> long dumpcycle), but I want it to stuff tapes until they are full. However, currently when you run out of tape you are left with dumps in the holding area that need to get flushed. I don't want to flush them, but if you don't amanda believes that those backups have been completed and figures it in to planning the next run. - Orion
RE: Jukebox Support
Title: RE: Jukebox Support I have the 15 and 30 tape treefrog working with amanda. They both have worked with out a problem for the last 2 years. -Original Message- From: Karl Davidson [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 21, 2002 12:45 PM To: [EMAIL PROTECTED] Subject: Jukebox Support Hello, We have a spectralogic 15 tape AIT2 TreeFrog tape library and we wondering if it was compatible with Amanda. My search has resulted in one person saying they were having problems getting it to work, but no document saying it is supported. Any help would be appreciated. Thank You Information contained in this email message is intended only for use of the individual or entity named above. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the [EMAIL PROTECTED] and destroy the original message.
Jukebox Support
Hello, We have a spectralogic 15 tape AIT2 TreeFrog tape library and we wondering if it was compatible with Amanda. My search has resulted in one person saying they were having problems getting it to work, but no document saying it is supported. Any help would be appreciated. Thank You
Re: Advice IDE Tape Drive
Robert wrote: > > Please advice me which type IDE-Tape drive will be better to buy for > 15 Gb storage. Doing a quick look at Pricewatch says that a Travan TR5 10/20Gb drive would be a good match: About $200 Seagate Hornet 10/20Gb Travan 5 Int IDE Tape backup drive p/n stt22a With the cost of tapes these days it's tempting to go with removable hard drives. But your needs are so minimal that tape's probably cheaper. -- [EMAIL PROTECTED] - HMC UNIX Systems Manager
Re: Newbie question; Write Protected? just Permission denied?
> On Mon, 18 Nov 2002 at 5:42pm, Lewis Getschel wrote > > > I tried the erase as the amanda user, and I got permission denied. I > > suspect that amlabel is simply saying write-protected when in fact it is > > just permission denied. > > Yep. > > > As a newbie that has looked in the FOM, and read the mailing list for > > almost a week, I'm confused as to WHAT group I should compile under. > > Some people say operator, others disk, others xxx (take your pick > > ). Is the group you use more dependant on what distro you use > > (hence the permission issue?) Is the group my problem? > > You can assign the user/group of you run amanda as in any way which will > work for you. On Red Hat Linux, I do amanda:disk (as you did). Make sure > that /dev/nst0 has the proper ownership and permissions -- for me > /dev/nst0 is owned by root:disk with a mode of 660. > Thanks so much. Is was a permission issue. /dev/nst0 was set to 640 (as are hd* and st*). After changing to 660, the amanda user can now write to the tape drive. Now I just need to figure out why my scsi pcmcia card and the ethernet pcmcia card 'step on each other' so the system only works with ONE of them plugged in, not both. It must be some kind of conflict, but nothing seems obvious yet. -- __ Sign-up for your own FREE Personalized E-mail at Mail.com http://www.mail.com/?sr=signup
Re: Want more aggressive tape use
Orion Poplawski wrote: > > Just started testing with using software compression on the server > (trying not to impact the clients too much), and got the following result: Advantageto doing some compression on the clients is that it saves you network bandwidth. > A lot of file systems were not backed up that could have been because > the planner rejected them. I'd like to be able to operate in a mode > where it kept dumping until the tape was full, but then threw away > whatever was left (i.e. - no amflush -> inefficient use of another tape). If you crank down dumpcycle in amanda.conf the program will do its best to fit more on each tape. If you crank it down too far you'll get daily complaints about it having to delay backups but it'll still do what it can. That's what I've done to maximize tape usage in my setup: single 50Gb AIT tape/day, 300Gb to back up. Tell it to manage full backups of everything over 4 days of backups. -- [EMAIL PROTECTED] - HMC UNIX Systems Manager
Re: One (of six) partitions returning "disk offline"
On Wed, 20 Nov 2002 at 4:10pm, Jay Lessert wrote > > selfcheck: device sda6 > > selfcheck: could not access sda6 (sda6): No such file or directory > > I'm not a Linux guy, but this is the one where Linux does not have > a conventional entry for "/" in /etc/fstab, right? Nope. / has an entry in fstab just like everybody else. > So if you're using tar, amanda cannot do the reverse-lookup from > device name to mount-point (directory path). > > Solution is to use directory names in disklist, not device names. > Much easier to read, anyway. Actually, it's a bug in amanda-2.4.2p2, fixed by the advfs.patch and/or 2.4.3. Even with directory names in the disklist, you'll hit this. > It's probably unfortunate that example/disklist uses device name > syntax, n'est pas? This I'll agree with. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: COMPAQ TSL-9000 DAT autoloader device
Marc, I am using just a Sony TSL-9000 with amanda. I use the chg-zd-mtx script. My os is linux though. I do have the sony docs and I will peruse through them to see if there is anything helpful. Also, I am pretty sure that I navigated through the Sony web and found pdf files for the 9000. If you need a tapetype respond back to the list and I will send one. Seems like the last one I ran was w/o HWCOMP so it is pretty accurate. Might save you about 5-7 hours of time. Jim On Thu, 2002-11-21 at 06:55, [EMAIL PROTECTED] wrote: > Hello, > > Does anyone have experience in configuration a COMPAQ/SONY TSL-9000 DAT > DDS-3 device with AMANDA ? I have only the device and no single > documentation about it. What I can tell is that it's an 8 DDS-3 tape > changer device which gets recognized on my Solaris 8 system under > /dev/rmt/0. Somehow it doesn't have two devices in it (as usual where you > have one device for the changer and one device for the tape) so that's > already something weird. I've tried to use mtx but mtx doesn't seem to like > the device at all. > > Any help would be appreciated. > > Regards > Marc >
COMPAQ TSL-9000 DAT autoloader device
Hello, Does anyone have experience in configuration a COMPAQ/SONY TSL-9000 DAT DDS-3 device with AMANDA ? I have only the device and no single documentation about it. What I can tell is that it's an 8 DDS-3 tape changer device which gets recognized on my Solaris 8 system under /dev/rmt/0. Somehow it doesn't have two devices in it (as usual where you have one device for the changer and one device for the tape) so that's already something weird. I've tried to use mtx but mtx doesn't seem to like the device at all. Any help would be appreciated. Regards Marc
Re: tape filling algorithm? (flush, or dump)
On Thursday 21 November 2002 00:47, Gene Heskett wrote:
>On Wednesday 20 November 2002 22:41, Gene Heskett wrote:
>
>Yeah, I know, I'm talking to myself :-) In some circles, thats
>considered to be 'poor form'. But, see below.
Continued, with another "ps:"
>>On Wednesday 20 November 2002 17:36, Jean-Louis Martineau wrote:
>>>On Wed, Nov 20, 2002 at 01:47:50PM -0600, Deb Baddorf wrote:
What kind of optimiaztion does amanda use while flushing
dumps to tape?
While doing the dump initially, she seems to
optimize to get the small jobs done first, and thus they
go onto tape in a similar order. But I'm currently without
a stacker. So only the first part of the dumps make it
to tape "live" (so to speak) and the rest get flushed as I
manually mount tapes. I'm hoping that some kind of
knapsack-packing algorithm is used . to fit the largest
files on the tape first, but then to add smaller ones to
fill the top of the "sack".
My archival "do all the level 0's" is taking up 5 tapes,
hence the optimization question!
>>>
>>>There is no optimazation, first in, first out.
>>>That's a need feature.
>>>
>>>Jean-Louis
>>
>>Humm, from a recent example/amanda.conf:
>>-
>>dumporder "sssS"# specify the priority order of each
>> dumper # s -> smallest size
>># S -> biggest size
>># t -> smallest time
>># T -> biggest time
>># b -> smallest bandwitdh
>># B -> biggest bandwitdh
>># try "BTBTBTBTBTBT" if you are not
>> holding # disk constrained
>>
>>I'm assuming that when Deb asked about it, using the word
>>'flushing', that Deb meant a normal amdump run as opposed to a
>> run of amflush, or an autoflush. I've no idea whether its
>> active for a flush, or just an amdump run. I have no idea if it
>> even works, but I'll find out tonight if I can stay awake till
>> then. I just modified mine to do the biggest ones first, and
>> amstatus should be able to see if its working that way pretty
>> shortly after the estimates are done.
>
>Yes, its working according to an 'SSSsss' setting from the looks
> of it, it started with /usr/src, nearly the last entry in the
> disklist which is nearly 2.7 gigs ATM. It also tells me its
> going to skip 2 entries tonight as they are too big, 1.6gb of
> music and 90 megs of /usr/X11R6 will get delayed till tomorrow
> night. Obviously I'm just now getting amanda restarted after a
> disk crash. I suppose it will take a week to get back to
> 'normal', whatever that is...
Ok, it appears that with the compression being done in software, it
smunched that 2.6 gigs worth of /usr/src down to 1.6 gigs. By the
time it was done, it had written only 2.6 gigs in real data down
the cable to the tape. There was I believe, room enough to fit the
original, uncompressed 1.6 gigs worth of music, and certainly room
enough for a compressed rendition of the 90 megs worth of
/usr/X11R6.
Since at the estmate phase, it hasn't compressed anything, is it
permissable to expand the value tapetype returns in order to
prevent this erronious FAILED message? And if so, do we believe
the vendors propaganda and arbitrarily double it? Or does amanda
do something similar internally? Here is why I ask, from the
email:
--
STATISTICS:
Total Full Daily
Estimate Time (hrs:min)0:23
Run Time (hrs:min) 3:03
Dump Time (hrs:min)1:09 0:54 0:15
Output Size (meg)2688.6 2499.4 189.2
Original Size (meg) 7214.6 6792.5 422.2
Avg Compressed Size (%)33.2 33.3 32.1
(level:#disks ...)
Filesystems Dumped 29 6 23 (1:23)
Avg Dump Rate (k/s) 660.9 788.9 210.2
Tape Time (hrs:min)1:58 1:49 0:08
Tape Size (meg) 2690.1 2499.7 190.3
Tape Used (%) 70.8 65.85.0
(level:#disks ...)
Filesystems Taped29 6 23 (1:23)
Avg Tp Write Rate (k/s) 390.5 390.3 391.8
Note that the reported Original Size is well above the roughly 3.8
gigs set in the DDS2 tapetype, and yet amdump knew it would fit
once compressed. I'm beginning to feel a bit like pooh, the bear
of very little brain here...
--
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz 512M
99.19% setiathome rank, not too shabby for a WV hillbilly
