Re: long estimate and run times
Gene Heskett wrote: On Saturday 21 June 2003 13:32, Jason Edgecombe wrote: Gene Heskett wrote: Can you post those 2 dumptypes from your amanda.conf? here they are, along with the global dumptype. define dumptype root-tar { global program "GNUTAR" comment "root partitions dumped with tar" compress none index #exclude list "/etc/amanda/DailySet1/exclude.gtar" priority low } define dumptype user-tar { root-tar comment "user partitions dumped with tar" priority medium } define dumptype global { comment "Global definitions" # This is quite useful for setting global parameters, so you don't have # to type them everywhere. All dumptype definitions in this sample file # do include these definitions, either directly or indirectly. # There's nothing special about the name `global'; if you create any # dumptype that does not contain the word `global' or the name of any # other dumptype that contains it, these definitions won't apply. # Note that these definitions may be overridden in other # dumptypes, if the redefinitions appear *after* the `global' # dumptype name. # You may want to use this for globally enabling or disabling # indexing, recording, etc. Some examples: # index yes # record no maxdumps 1 } And the only error I see in this is that index isn't defined, just used. I believe that should be as shown in the global comment. I wonder if that lack is preventing the lowered priority setting from taking place? I need more caffiene, I just woke up from an overnighter. Sincerely, Jason Edgecombe I'm sorry that I'm only writing this now. I wanted to thank everyone on the list who helped me. I did get my problem resolved. It was a firewall issue. By allowing all udp ports below 1024 I fixed the problem. thanks, Jason Edgecombe
Re: Estimate timeouts
Toomas Aas wrote: Date: Wed, 23 Jul 2003 11:04:10 -0400 From: Jason Edgecombe <[EMAIL PROTECTED]> I noticed that the amanda server uses udp port 993 as the src. so the first part of the amanda network conversation goes like this: server port 993 -> client port 10080 hi server port 993 <- client port 10080 hi back server port 993 -> client port 10080 sendsize server port 993 -> client port 10080 ack sendsize Is this correct? does amanda always use port 993 as the source port? It uses a random privileged (<1024) port. I hypothesize that my dumps are only finishing sometimes because port 993 is blocked by default. port 993 works work only a few minutes until the "allow related" rule timesout. I understand it you have iptables/netfilter on the server? Then this might well be the reason. However, then it should also fail during daytime. When amanda uses tcp instead of udp for the backup, are all of the tcp connections made from the server to the client? How I I force amanda to use tcp instead of udp? I have had the backup fail during the day when I run it, but it usuaully fails at night. I think that the estimate times vary and that when the estimate time exceeds some firewall timeout value, then it fails because it no longer fits the iptables "related" rule. Can I use the tcp connections to get around this? Sincerely, Jason Edgecombe
Re: Estimate tieouts
Jon LaBadie wrote: On Tue, Jul 22, 2003 at 09:15:39AM -0400, Jason Edgecombe wrote: Hi everyone, I am having estimate timeouts from an amanda client. The client's logs for amandad have the following error messages at the end: amandad: time 81.782: dgram_recv: timeout after 10 seconds amandad: time 81.782: waiting for ack: timeout, retrying amandad: time 91.782: dgram_recv: timeout after 10 seconds amandad: time 91.782: waiting for ack: timeout, retrying amandad: time 101.782: dgram_recv: timeout after 10 seconds amandad: time 101.782: waiting for ack: timeout, retrying amandad: time 111.782: dgram_recv: timeout after 10 seconds amandad: time 111.782: waiting for ack: timeout, retrying amandad: time 121.782: dgram_recv: timeout after 10 seconds amandad: time 121.782: waiting for ack: timeout, giving up! amandad: time 121.782: pid 17514 finish time Tue Jul 22 00:17:03 2003 My etimeout value on the server is -1800. Doesn't seem like estimate timeouts, starting after only 81 seconds. Sounds like some network communication attempt failing. I'm running amanda 2.4.3 on the client and 2.4.4 on the server. My firewall is properly letting packets through. Both client and server share a connection via the campus 100mbs network and a private gigabit network. I'm backing up over the gigabit network. I can run the backup manually and the estimate phase runs fine, but it doesn't want to run at night. Does anyone have any insight into this? Two networks. Sounds like the amanda connection works over one but not the other. And for some reason, during the day, with manual attempts the working network is used, in evening, from cron, the other, non-working network is used. Maybe name resolution differences on the two networks? I checked this with ethereal, everthing is in fact running over the gigabit ethernet. I noticed that the amanda server uses udp port 993 as the src. so the first part of the amanda network conversation goes like this: server port 993 -> client port 10080 hi server port 993 <- client port 10080 hi back server port 993 -> client port 10080 sendsize server port 993 -> client port 10080 ack sendsize Is this correct? does amanda always use port 993 as the source port? I hypothesize that my dumps are only finishing sometimes because port 993 is blocked by default. port 993 works work only a few minutes until the "allow related" rule timesout. Jason Edgecombe
Estimate tieouts
Hi everyone, I am having estimate timeouts from an amanda client. The client's logs for amandad have the following error messages at the end: amandad: time 81.782: dgram_recv: timeout after 10 seconds amandad: time 81.782: waiting for ack: timeout, retrying amandad: time 91.782: dgram_recv: timeout after 10 seconds amandad: time 91.782: waiting for ack: timeout, retrying amandad: time 101.782: dgram_recv: timeout after 10 seconds amandad: time 101.782: waiting for ack: timeout, retrying amandad: time 111.782: dgram_recv: timeout after 10 seconds amandad: time 111.782: waiting for ack: timeout, retrying amandad: time 121.782: dgram_recv: timeout after 10 seconds amandad: time 121.782: waiting for ack: timeout, giving up! amandad: time 121.782: pid 17514 finish time Tue Jul 22 00:17:03 2003 My etimeout value on the server is -1800. I'm running amanda 2.4.3 on the client and 2.4.4 on the server. My firewall is properly letting packets through. Both client and server share a connection via the campus 100mbs network and a private gigabit network. I'm backing up over the gigabit network. I can run the backup manually and the estimate phase runs fine, but it doesn't want to run at night. Does anyone have any insight into this? Thanks, Jason Edgecombe
Re: long estimate and run times
Hello again everyone, I have had some luck with getting amanda to work with my machines. I first had a firewall problem, which I have solved. I still timeout for the estimate phase on a certain client, but there is a twist. The the client and the server are dual-homed hosts. Both can talk to each other over either network interface. eth0 is our public network. eth1 is our private network. The estimate phase doesn't timeout when using the public network (100Mb/s), but it does timeout when using the private network (1Gb/s). This seems to be some type of name resolution problem, but I can't figure it out. Each machine has a different hostname for each interface and I have added both names in the /etc/hosts files on each machine. amcheck says it's ok whether I put the public or private name in the disklist file. BTW, both machines are Redhat 7.3 machines. how does the amanda client machine determine which machine to talk to when it sends it's estimates? Does anyone have any insight into this problem? Sincerely, Jason Edgecombe
Re: long estimate and run times
Gene Heskett wrote: Can you post those 2 dumptypes from your amanda.conf? here they are, along with the global dumptype. define dumptype root-tar { global program "GNUTAR" comment "root partitions dumped with tar" compress none index #exclude list "/etc/amanda/DailySet1/exclude.gtar" priority low } define dumptype user-tar { root-tar comment "user partitions dumped with tar" priority medium } define dumptype global { comment "Global definitions" # This is quite useful for setting global parameters, so you don't have # to type them everywhere. All dumptype definitions in this sample file # do include these definitions, either directly or indirectly. # There's nothing special about the name `global'; if you create any # dumptype that does not contain the word `global' or the name of any # other dumptype that contains it, these definitions won't apply. # Note that these definitions may be overridden in other # dumptypes, if the redefinitions appear *after* the `global' # dumptype name. # You may want to use this for globally enabling or disabling # indexing, recording, etc. Some examples: # index yes # record no maxdumps 1 } Sincerely, Jason Edgecombe
Re: long estimate and run times
I am also have a defunct amandad process, would that affect my estimate and dumper times? Jason Edgecombe
Re: long estimate and run times
Gene Heskett wrote: On Friday 20 June 2003 08:49, Jason Edgecombe wrote: Look in /tmp/amanda/sendsize*debug on that system. It'll have timestamped info on each estimate (probably level 0s and level 1s for each fs). Figure out which one is taking so long. Any errors in the system logs? What type of hardware RAID? What OS, and what types of FSs? Here are some snippets from the kernel messages: -- SCSI subsystem driver Revision: 1.00 scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10 aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10 aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs ...snip... scsi2 : 3ware Storage Controller Vendor: 3ware Model: Logical Disk 0Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 00 blk: queue c66caa18, I/O limit 4095Mb (mask 0x) Vendor: 3ware Model: Logical Disk 1Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 00 blk: queue c66ca618, I/O limit 4095Mb (mask 0x) ...snip... scsi3 : QLogic QLA2300 PCI to Fibre Channel Host Adapter: bus 3 device 1 irq 24 Firmware version: 3.01.13, Driver version 6.01.00-fo Vendor: Tornado- Model: F2 V2.0 Rev: Type: Direct-Access ANSI SCSI revision: 02 scsi(3:0:0:0): Enabled tagged queuing, queue depth 16. Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0 SCSI device sdc: 2109849600 512-byte hdwr sectors (1080243 MB) sdc: sdc1 - Software is Redhat linux 7.3 with custom redhat kernel 2.4.20-18.7bigmem with the adaptec 79xx drivers from adaptec's web site. The motherboard is a Supermicro X5DPi-G2 motherboard with dual Intel 2.4GHz Xeons and 4GB of RAM. /dev/sda and /dev/sdb are 180GB ide drives attached to a 3ware ide to scsi raid controller. The Overland Neo2000 with LTO2 tape drive is on the Adaptec controller and the hardware raid is on the Qlogic controller. drives are as follows: /dev/hda - 200GB ide disk - amanda holding disk /dev/sda and /dev/sdab - 180GB each three partitions each; /boot and / in a linux software RAID1 config. swap partitions on each disk /dev/sdc - lvm volume - rebranded Axus Brownie BR-8000FC 2GB fibre channel RAID5 array with IDE drives. /home is a logical volume on the /dev/sdc volume group all filesystems are ext3. --- Volume group --- VG Name vg01 VG Access read/write VG Status available/resizable VG # 0 MAX LV256 Cur LV1 Open LV 1 MAX LV Size 2 TB Max PV256 Cur PV1 Act PV1 VG Size 1005.94 GB PE Size 64 MB Total PE 16095 Alloc PE / Size 8800 / 550 GB Free PE / Size 7295 / 455.94 GB DLE's that are all on the same disk should all have the same 'spindle' number. Its possible that amanda is running 2 or more sessions against the same disk, in which case there will be some lost time due to thrashing of the disk with uncoordinated seeks to different partitions. here is the pertinent part of the disklist: localhost /boot root-tar 1 localhost / root-tar 1 localhost /home user-tar 2 Have you run an hdparm -Tt on the slow disks? On the raid? # hdparm -tT /dev/hda /dev/hda: Timing buffer-cache reads: 128 MB in 0.21 seconds =609.52 MB/sec Timing buffered disk reads: 64 MB in 1.65 seconds = 38.79 MB/sec # hdparm -tT /dev/sda /dev/sda: Timing buffer-cache reads: 128 MB in 0.21 seconds =609.52 MB/sec Timing buffered disk reads: 64 MB in 2.07 seconds = 30.92 MB/sec # hdparm -tT /dev/sdb /dev/sdb: Timing buffer-cache reads: 128 MB in 0.21 seconds =609.52 MB/sec Timing buffered disk reads: 64 MB in 1.39 seconds = 46.04 MB/sec # hdparm -tT /dev/sdc /dev/sdc: Timing buffer-cache reads: 128 MB in 0.23 seconds =556.52 MB/sec Timing buffered disk reads: 64 MB in 1.27 seconds = 50.39 MB/sec Not that I know of. amanda needs to figure out how much data there is to be stored. This estimate phase is one of amanda's real strengths as it makes for very efficient use of the available resources. Thanks for the info. #grep estimate sendsize.20030620014500.debug sendsize[21320]: time 0.004: waiting for any estimate child sendsize[21322]: estimate time for /boot level 0: 0.045 sendsize[21322]: estimate size for /boot level 0: 13010 KB sendsize[21322]: estimate time for /boot level 1: 0.007 sendsize[21322]: estimate size for /boot level 1: 10 KB sendsize[21320]: time 0.093: waiting for any estimate child sendsize[21325]: estimate time for / level 0: 394.156 sendsize[21325]: estimate size for / level 0: 11711950 KB sendsize[21325]: estimate time for / level 4: 103
Re: long estimate and run times
Joshua Baker-LePain wrote: On Tue, 17 Jun 2003 at 3:23pm, Jason Edgecombe wrote I'm trying to figure out why may backups and estimages take so long. I have a dual-processor machine with 4GB RAM. / and /boot are two software RAID1 drives and /home is a RAID5 hardware drive. My tape drive is an Overland Neo2000 LTO2 with 30 tape library. I am using a 200GB holding disk on an isolated drive dedicated to amanda. The estimate phase on localhost (my backup server) took two hours. I watched the tar process and it took anywhere from 5-30% of the cpu Look in /tmp/amanda/sendsize*debug on that system. It'll have timestamped info on each estimate (probably level 0s and level 1s for each fs). Figure out which one is taking so long. Any errors in the system logs? What type of hardware RAID? What OS, and what types of FSs? Can the estimate phase of amanda be bypassed or disabled? If so, what are the implications? Jason Edgecombe
long estimate and run times
Hi all, I'm trying to figure out why may backups and estimages take so long. I have a dual-processor machine with 4GB RAM. / and /boot are two software RAID1 drives and /home is a RAID5 hardware drive. My tape drive is an Overland Neo2000 LTO2 with 30 tape library. I am using a 200GB holding disk on an isolated drive dedicated to amanda. The estimate phase on localhost (my backup server) took two hours. I watched the tar process and it took anywhere from 5-30% of the cpu here is a sample of my amanda report: STATISTICS: Total Full Daily Estimate Time (hrs:min)2:01 Run Time (hrs:min) 2:47 Dump Time (hrs:min)1:10 0:17 0:53 Output Size (meg)6532.4 4628.1 1904.2 Original Size (meg) 6532.4 4628.1 1904.2 Avg Compressed Size (%) -- -- --(level:#disks ...) Filesystems Dumped 10 5 5 (1:4 3:1) Avg Dump Rate (k/s) 1597.4 4656.7 615.1 Tape Time (hrs:min)0:04 0:03 0:01 Tape Size (meg) 6532.6 4628.2 1904.3 Tape Used (%) 3.22.20.9 (level:#disks ...) Filesystems Taped10 5 5 (1:4 3:1) Avg Tp Write Rate (k/s) 25126.825105.025179.7 USAGE BY TAPE: Label Time Size %Nb gauss-21 0:046532.63.210 FAILED AND STRANGE DUMP DETAILS: /-- localhost / lev 3 STRANGE sendbackup: start [localhost:/ level 3] sendbackup: info BACKUP=/bin/tar sendbackup: info RECOVER_CMD=/bin/tar -f... - sendbackup: info end ? gtar: ./tmp/screens/S-root/16229.pts-6.master: Warning: Cannot stat: No such file or directory | gtar: ./tmp/ssh-XXHe66oR/agent.26555: socket ignored | gtar: ./tmp/ssh-XXXjwuZQ/agent.16145: socket ignored | gtar: ./tmp/ssh-XXtUIuYn/agent.12543: socket ignored ? gtar: ./var/lib/amanda/DailySet1/index/mathweb/_home/20030617_0.gz.tmp: Warning: Cannot stat: No such file or directory ? gtar: ./var/lib/ganglia/rrds/Math Gauss Cluster/node5.cluster/load_one.rrd: file changed as we read it | Total bytes written: 51947520 (49MB, 82kB/s) sendbackup: size 50730 sendbackup: end \ NOTES: planner: Incremental of localhost:/ bumped to level 3. planner: Full dump of mathweb:/home promoted from 3 days ahead. planner: Full dump of mathweb:/usr/local promoted from 3 days ahead. planner: Full dump of mathweb:/www promoted from 3 days ahead. planner: Full dump of localhost:/boot promoted from 3 days ahead. planner: Full dump of mathweb:/var/log promoted from 3 days ahead. taper: tape gauss-21 kb 6689664 fm 10 [OK] DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK LORIG-KBOUT-KB COMP% MMM:SS KB/s MMM:SS KB/s -- - - localhost / 3 50730 50752-- 10:18 82.1 0:02 21829.3 localhost /boot 0 13010 13024-- 0:01 10048.7 0:03 3919.4 localhost /home 1 205640205664-- 35:26 96.7 0:10 20045.5 mathweb / 1 68530 68544-- 1:25 810.3 0:05 12602.2 mathweb /boot 1 1032-- 0:00 79.9 0:02 14.8 mathweb /home 03138930 3138944-- 6:29 8071.3 1:50 28619.6 mathweb /usr/local 01135110 1135136-- 9:28 1997.6 0:49 23216.5 mathweb /var/lib/pgsql 11625000 1625024-- 5:41 4770.4 0:57 28380.8 mathweb /var/log 0 212930212960-- 0:23 9281.4 0:10 20687.6 mathweb /www 0 239240239264-- 0:36 6579.7 0:17 14419.5 (brought to you by Amanda version 2.4.4) Thanks, Jason Edgecombe
Re: recover boot diskette
Gregor Ibic wrote: does anyone have a boot diskette with amanda clients on it, to boot from the diskette and restore the server? regards, gregor Try knoppix at http://www.knoppix.net. It's a complete linux system on a bootable cd that includes amanda clients. Sincerely, Jason Edgecombe
Re: Disklist question
Joshua Baker-LePain wrote: On Mon, 2 Jun 2003 at 10:42am, Jason Edgecombe wrote I have a question about the disklist format. I am using tar to dump my filesystems. I would like to dump / (without /home) and have /home be on a separate DLE. How do I do that? Must I specify each directory in / as a separate DLE? It depends on your partition layout. Amanda runs tar with the --one-file-system option, so backing up / will, by default, grab only what's in the same partition with /. If home is in that partition, you'll need to exclude it, and then add a DLE for it. Thanks! On my system /home is a different partition, so I can just specify / and /home. Thanks, Jason Edgecombe
Disklist question
I have a question about the disklist format. I am using tar to dump my filesystems. I would like to dump / (without /home) and have /home be on a separate DLE. How do I do that? Must I specify each directory in / as a separate DLE? Sincerely, Jason Edgecombe
Re: Qualstar TLS-4480 barcode reader
Hi there, I just got a new Overland Data Neo2000 LTO-2 30-tape library. I found your post on creating barcode labels for tapes. Does is matter what data the barcode contains. For example, can I create a barcode that reads "DailySet1-01" through "DailySet1-30" Do you know if the Avery L7656 labels will work with LTO2 tapes? Sincerely, Jason Edgecombe Ron Snyder wrote: Thanks! That's one that I wasn't even aware of. -ron -Original Message- From: jens persson [mailto:[EMAIL PROTECTED] Sent: Thursday, February 27, 2003 11:37 PM To: [EMAIL PROTECTED] Subject: Re: Qualstar TLS-4480 barcode reader On Thu, Feb 27, 2003 at 01:42:53PM -0800, someone wrote: We use a TLS-4480, with the chg-zd-mtx that I found someware :-) and it working well (as long as our home printed labels are readable). What are you using to print your labels? About a year ago I looked for anything that would create labels for me, but never could find any software that would work with my TLS. Since this might be useful to others, I'm answering to the list (after removing the name of the questioner, that asked of list). I'm using gnu-barcode (http://www.gnu.org/software/barcode/barcode.h> tml) printing to Avery L7656 labels with the commandline: barcode -ucm -pa4 -iinfile.txt -ooutfile.ps -e code39 -t 4x21+0.6+1.5 \ -m.5x.1 where infile.txt contains the text of the labels I want and output.ps is a postscript file. Home this helps. /jp -- jens persson #One disk to rule them all, One disk to find <[EMAIL PROTECTED]>#them. One disk to bring them all and in the Mäster Olofsväg 24 #darkness grind them. In the Land of Redmond S-224 66 LUND;SWEDEN # where the shadows lie. # Henrique Holschuh "The Silicon Valley Tarot"
Re: tapeless changer
briner wrote: hi, I'm wondering if there is a way in the case where we use amanda with a tapeless system (HD disk), to emulate such a tape-changer that is a script which change the link "data" to the good HD in my case +amanda.conf has the follwing line: tapedev "file:/unige/amanda/bckp/" +ls -l /unige/amanda/bckp/ /unige/amanda/bckp/data -> sb03_1/ /unige/amanda/bckp/info /unige/amanda/bckp/sb01_1 -> /net/obssb1/export/diskB1/1/ /unige/amanda/bckp/sb01_2 -> /net/obssb1/export/diskB1/2/ /unige/amanda/bckp/sb02_1 -> /net/obssb2/export/diskB1/1/ /unige/amanda/bckp/sb02_2 -> /net/obssb2/export/diskB1/2/ /unige/amanda/bckp/sb03_1 -> /net/obssb3/export/diskB1/1/ /unige/amanda/bckp/sb03_2 -> /net/obssb3/export/diskB1/2/ /unige/amanda/bckp/sb05_1 -> /net/obssb5/export/diskB1/1/ /unige/amanda/bckp/sb05_2 -> /net/obssb5/export/diskB1/2/ so as you imagine, i'm changing automatically with a script the link data to one of the "tape"Disk. But is there an other smarty way to do this through a special changerdev which will make things much more easier! briner Hi there, There is a way to use a virutal tape changer with a diskless config in Amanda. It doesn't use links as you ask about though. Here is how my setup goes: I created directories /var/amanda/backup1 through backup10. and I created a "data" directory under each of those directories. Configure amanda to use the chg-multi tape changer. Use a contrived tapetype that is as big as you want your virtual tapes to be. Here are the relevant parts of my chg-multi.conf file: - statefile /etc/amanda/DailySet1/changer-status # Names a status file where the current ``changer'' state is stored. statefile /etc/amanda/DailySet1/changer-status # What are the slot numbers used in the tape rack? firstslot 1 lastslot 10 # Enumerate the device files that go with the particular tape changer # slots. Don't forget to specify the no-rewind version of the device. slot 1 file:/var/amanda/backup1 slot 2 file:/var/amanda/backup2 slot 3 file:/var/amanda/backup3 slot 4 file:/var/amanda/backup4 slot 5 file:/var/amanda/backup5 slot 6 file:/var/amanda/backup6 slot 7 file:/var/amanda/backup7 slot 8 file:/var/amanda/backup8 slot 9 file:/var/amanda/backup9 slot 10 file:/var/amanda/backup10 - The rest of the chg-multi.conf is the same as the chg-multi.conf example file that cones with amanda. use amtape to manipulate your virtual tape changer and label your virtual tapes. Amanda will do the rest. I have had this setup working on RedHat7.3 for a few months now. Sincerely, Jason Edgecombe
How do the holding disk and dumping to tape interact?
Hello all, How does the holding disk work in amanda with relation to dumping to tape? Specifically, does Amanda wait for all filesystems to be spooled before it starts dumping to tape? Does amanda start dumping to tape while it is still spooling the filesystems to the holding disk? Which is the default behavior and can I change it? Sincerely, Jason Edgecombe
Re: How do I configure Amanda 2.4.3 to backup to disk
Marc W. Mengel wrote: You set the "tapedev" entry in amanda.conf to a value like tapedev "file:/some/directory/somewhere" - Marc Mengel <[EMAIL PROTECTED]> On Tue, 3 Dec 2002, Jason Edgecombe wrote: How do I configure Amanda 2.4.3 to backup to a directory or file on the hard drive? I have looked though the documentation, and I have searched the amanda-users archive, but I can't find any documentation on this. Would someone clue me in please? Hi all, Thanks for all of the replies. My problem was that I had not created a data directory under the backup directory. Sincerely, Jason Edgecombe
How do I configure Amanda 2.4.3 to backup to disk
Hi all, How do I configure Amanda 2.4.3 to backup to a directory or file on the hard drive? I have looked though the documentation, and I have searched the amanda-users archive, but I can't find any documentation on this. Would someone clue me in please? Sincerely, Jason Edgecombe