Re: [Veritas-bu] 6.0 MP4 appearing on FTP site
On 11/20/06, Ed Wilts <[EMAIL PROTECTED]> wrote: -snip- > It's not all there yet (e.g. no CLT or JAV yet) but is popping up now. > I'm guessing it will be all there by Tuesday morning. > > Enjoy! > .../Ed > > -- > Ed Wilts, Mounds View, MN, USA > mailto:[EMAIL PROTECTED] Ed (and others), how's it all going with 6.0 MP4? I'm still with 5.1 (MP6) but I'd like to use the improved staging capabilities of 6.0. Using RHEL 4, Update 4. Someone mentioned that 3-way NDMP performance was worse but I'd like to hear from the list how MP4 is going in general. Regards. -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NetBackup 5.1 MP6
On 12/20/06, Backup reports <[EMAIL PROTECTED]> wrote: About a week ago I downloaded MP6 for NetBackup 5.1 in response to a security bulletin ( http://entsupport.symantec.com/docs/285082 ) issued by Symantec. It went on fine and things seem to be working ok, but I've noticed a couple bits of strangeness: 1. The status monitor no longer shows what path/file is currently being backed up; the field is blank. The Pathname field in the job field box that appears after clicking on an individual job is also blank. Yeah. I installed MP6 to address some annoyances I've seen with disk staging. The "path last written" in bpdbjobs output (and therefore also in the gui) is now showing part of the bpsched debug log. Now I'm wondering what else they messed up. 2. NetBackup sends out emails for each client that completes (or fails) backup. The subject line is now blank, or very minimal. The client name used to be part of the subject line. I don't use the email feature. Can't commment on this. Anyone else notice these things? Thanks! Eric C. Anderson 407-823-5474 Computer Services, Systems Support [EMAIL PROTECTED] http://pegasus.cc.ucf.edu/~anderson/ University of Central Florida UCF Stands for Opportunity ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Backups vs archives
On 12/7/06, Curtis Preston <[EMAIL PROTECTED]> wrote: Based on some previous posts, I'd like to throw out the following thought, and see what you folks think about it. Thanks for posting to the netbackup list, this is a great article. My thoughts on this are the following. Netbackup's archive feature I've never cared for netbackup's archive feature because it mechanizes removal of data. Removal of data should only be done manually and with great care. Bugs like this one ( http://entsupport.symantec.com/docs/284450 ) reinforce my dislike of systems that perform automated removal. But I agree that the process of doing backups should not be the same as performing archives. They are two different things. Critical metadata required for recovery For both backups and archives, the single most vital piece of metadata to me is automount history. Data inevitably gets moved from one storage server to another or from one volume to another. Users don't know or care where it lives, all they know is that they need /proj/roswell/autopsy recovered from 1/1/1999. Backup/archive systems probably only know about it's current true location, blackhole:/vol/vol0/autopsy. Everyone should keep a running history of all automount map changes. (I can share the scripts I've got, but they're site-specific). Poor man's archive server I work at a small company. Here's what we do. Clearly this is not a great system but it has some advantages. A request comes in to archive a home directory or engineering related project files. We copy the requested data to inexpensive big disk raid5 systems. These systems get backed up periodically such that there's never more than two copies on tape and they have little overlap. We periodically run updatedb so that a single file can be found with locate. When copying we keep the entire path intact. Other metadata retained: md5sum of each file, original server name, volume, archive date. This has none of the other great advantages that a CAS or HSM system might provide, and filename search doesn't scale well for multiple archive servers. A search layer can be added above locate to use each archive server's locate database. This is on my to-do list. ;-) In many cases a request comes in to recover monster-spreadsheet.xls but they don't always remember where it used to live. locate is extremely useful in cases like this. Final Thoughts Modern CAS systems look very appealing. A CAS system would be too expensive for all our general purpose data but I'm going to seriously consider one for at least financial data. For small data sets a combination of rdiff-backup and rsync for replication might just be enough to do the trick. In any case today's commercial CAS systems look great. I'd be interested in hearing about experiences with them from anyone on the list. Email archiving is another thing I really need to look into using at our site. Cheers. [snip] -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Adding a Media server
On 11/10/06, ZIMMER, RANDY K [AG/1000] <[EMAIL PROTECTED]> wrote: All, We currently have a HP Master and Media Server. We also have Sun V880 that 's running Veritas Storage Migrator and we are in the process of adding tape drives to it and a want to make it a 2 nd media server to our master server. What configuration changes will I have to make to make it a media server of our HP Master. I' m hoping I can do this without having to re-install the software.Can you clarify a bit for us Randy? It should just be a matter of: - buy a media server license- install netbackup- add tape drives- create storage unit Not sure what you mean by "re-install the software".Regards.-snip- -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] LTO3 and Disk Staging
On 9/28/06, Weber, Philip <[EMAIL PROTECTED]> wrote: Trouble acquiring decent hardware. Interested though, is that concurrent operation, i.e. writing at these speeds and reading at these speeds concurrently? I'm not familiar with bonnie++. cheers, PhilDue to the demanding nature of disk staging you never want to read and write concurrently. This will definately kill performance no matter what disk type is used. Hard drives have sets of heads that can only be in one place at a time. This is not a SATA vs SCSI issue. For high performance operations you don't ever want them thrashing around. It's certainly possible to throw money at the problem to buy faster disk, but I'd argue that you're better off using the money for more staging space to avoid concurrent reads and writes. -snip--- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] LTO3 and Disk Staging
On 9/25/06, Joe Royer <[EMAIL PROTECTED]> wrote: As one of the others to ask this question, here are my findings so far:- Most people aren't keeping their LTO3 happy, some even falling victimto the marketing crap that "LTO technology isn't harmed by shoeshining." I have personally seen a 20-25% (per month) drive failure rate in anenvironment of constant shoeshining on LTO1.- Software striping of multiple LUNs may be a good quick fix or stop gapsolution. Especially if you can aggregate your FC interfaces this way. - A few have mentioned that they started using FC disk because SATA wastoo slow.I have no trouble at all keeping LTO3 streaming with SATA R5 or R0 array staging areas.Here's a bonnie++ report from such a raid (R0). 197MB/sec writes, 240MB/sec reads. Fast enough for LTO3 or LTO4, for that matter. Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP fiik 4G 197803 67 95682 24 240898 23 589.0 1Oridnary 7200 RPM 300GB disks on a 3ware 9k series raid controller.Maybe some people are having touble acquiring decent hardware or configuring their raids. 7200 RPM SATA drives will never be as fast as 1 RPM SCSI drives, but IMHO they're perfect for staging because they're fast enough when striped and cheap. Using FC for staging area seems wrong somehow. They're crazy expensive for use as scratch space. -snip--- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] make_scsi_dev woes under Linux
On 8/15/06, Daniel Cox <[EMAIL PROTECTED]> wrote: We've got a few media servers running NetBackup 5.1 MP5 under Linux (RedHat AS4) and we're having no end of problems with FC attached tape drive device mappings. I see when NB starts it runs make_scsi_dev, which creates the following devices: If you have people installing new hba drivers or re-arranging hba's on your systems without your knowledge, then you clearly have procedural issues to deal with. Not a technical problem. In any event, how about just moving make_scsi_dev aside and replace it with with your own script that does what you want? All it does is make symlinks because netbackup is too stupid to use normal linux device names. It wants to use crazy solaris style names for some unknown reason. [snip]-- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] LTO3 Drive Speed Poll
On 7/19/06, Bahnmiller, Bryan <[EMAIL PROTECTED]> wrote: Tim, If it's ok with you, could you send out a bit more about your setup? Hosts, OS, NBU version, buffer configuration and (especially for me) what kind of disk array for staging? I've been trying to get our LTO3's to stream and so far have been stymied trying to pull data off of the disk – Clariion using either SATA or FC disk and have tried both RAID 5 and RAID 3. AIX 5.3, 2 Gb/s FC to disk and tape. Thanks, Bryan Bryan Bahnmiller ISD Business Continuity Pier 1 Imports, Inc 817-252-8570nbu 5.1 MP5 on redhat enterprise 4 using 3ware 9000S controlers with 3ware raid50 (striped raid 5's). This is sata with seagate 400GB ST3400832AS drives. The newer 9550SX cards are faster. Just use the bonnie++ benchark or something similar to see what you raid can do. Raid 5 will do, but I get better write performance with "50". I can get over 300MB/sec reads from these raids, well over anything i'll need from LTO3 and probably good enough for LTO4 when that comes out. The machines are dual xenon 3.4 GHz, 2GB ram with qlogic 2322 fibre channel cards. # cat NUMBER_DATA_BUFFERS 16 # cat SIZE_DATA_BUFFERS 524288 # cat SIZE_DATA_BUFFERS_DISK 1048576 # cat NUMBER_DATA_BUFFERS_DISK 16 # cat /usr/openv/netbackup/NET_BUFFER_SZ 65536 # cat /etc/rc.local echo 134217728 >/proc/sys/kernel/shmall echo 134217728 >/proc/sys/kernel/shmmax Hope this helps.-- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] LTO3 Drive Speed Poll
On 7/11/06, Justin Piszcz <[EMAIL PROTECTED]> wrote: What speeds have other members gotten when doing speed testing to LTO3drives?Justin.With real-world data, netbackup is reporting from 6MB/sec ( this does not seem right) to about 143.5MB /sec with 2GB fragments over a sample of 1572 fragments for the past 15 days. Average performance is 78.5MB/sec. These are directly connected IBM LTO3 drives to linux systems using Qlogic 2322's. Most of our backups fall in the 80 to 125MB/sec range. Creating /usr/openv/netbackup/NET_BUFFER_SZ with a reasonable value seems to have helped us a lot even though its just going from staging to tape.-- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Symantec and Linux support
On 5/16/06, Ed Wilts <[EMAIL PROTECTED]> wrote: On Tue, May 16, 2006 at 12:05:51PM -0400, Paul Keating wrote:> Honestly though, that smacks of management thinking Linux is free (as in> beer) rather than free (as in speech.)Not my management. > Any question like that coming from mgmt is usually related to percieved> dollar savings.The perceived dollar savings is typically in the hardware, not thesoftware. We pay as much, if not more, for Red Hat Enterprise Linux subscriptions than we do for our Microsoft Windows licenses - it's allthe add-ons to Windows that really kill you. When comparing hardwarethough, Intel-based systems are cheaper than Sparc-based systems. *nod* Not to mention that x86 and x86_64 runs circles around sparc. Although isn't windows 2003 > $2k whereas rhes 4 is $350 for basic support?Well, we're already committed down the RHEL4 / NBU 5.1 road, and it's been working out great so far. What a relief being off the 2000 era E450 and R420R's! Symantec would not provide much in they way of providing steps needed to migrate from Solaris to Linux insisting that PS absolutely had to be involved. I did it myself. It wasn't really that difficult. We're using CentOS, but due to Symantec only supporting the packages that Redhat builds, I have to install the genuine article if I want my support calls answered. Planning on doing that in a week or so. Any remaining infrastructure suns in our datacenter are all going out the door. We still have a few suns and windows servers for a *few* specific application needs, but we're a linux shop. All engineers here use Linux. Management and finance types use windows. .../Ed--Ed Wilts, Mounds View, MN, USAmailto: [EMAIL PROTECTED]___Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu-- -Tim, Transmeta
[Veritas-bu] Netbackup 5.1 MP5 released into the wild
FYI, It appears that 5.1 MP5 has been released.Among lots of fixes, seems to address some dssu relocation "issues".-- -Tim
Re: [Veritas-bu] Disk backup - Storage Hardware configurations
On 3/24/06, Sander <[EMAIL PROTECTED]> wrote: Tim Berger wrote (ao):>Agreed. 70MB/sec writes is disappointing but acceptable for my>requirements. On a fast cpu machine you'll see better performance>with software raid. That's normal. There's lots of other reasons to >go with 3ware. Here's what led me there:>- Auto-sector repair. With big disks, this is critical. I've lost>software raids in the past due to latent bad block loss that led to>more than one disk getting failed out. All big disks have this >problem.I agree with you. But Linux sw raid improves. Nowadays you can (andshould if you care) run a background read test over the raid array.This detects latent bad blocks. Detection and correction are two different things. With sw raid, a persistant bad block is not always recoverable resulting in failing out a disk or hanging your system. Hardware raids can remap them on the fly. >- Very reliable driver support.Dito for both Linux software raid and the disk controllers. > smartd support.That goes for anything in Linux.Are you using a 2.6 kernel? The smartmontools require ATA-passthrough ioctl() calls with 2.6 for sata support which isn't there yet (unless it came out like last week or something). In any event, everyone should select what works best for them.[snip] --Humilis IT Services and Solutionshttp://www.humilis.net-- -Tim
Re: [Veritas-bu] Disk backup - Storage Hardware configurations
Agreed. 70MB/sec writes is disappointing but acceptable for my requirements. On a fast cpu machine you'll see better performance with software raid. That's normal. There's lots of other reasons to go with 3ware. Here's what led me there: - Auto-sector repair. With big disks, this is critical. I've lost software raids in the past due to latent bad block loss that led to more than one disk getting failed out. All big disks have this problem. - Very reliable driver support. smartd support.- Hot swap support, good management interface.Software raid is good for some things, but I would not use it in a machine that requires high uptime. Each of my media servers has 24 400GB drives so ease of managment and reliability is critical. I used to strictly use software raid. No more. On 3/23/06, Sander <[EMAIL PROTECTED]> wrote: Tim Berger wrote (ao):>I have two Linux media servers setup with 3ware 9500S controllers>using raid "50" with seagate disks. With our configuration this is>two raid 5's of 6 disks each that are then striped together. I get up >to about 70MB/sec writes and 300MB/sec reads after the readahead is>tweaked up.I'm not impressed at all by 70MB/sec on a raid config. I currently haveeight Maxtor 300GB disks (slow) sw raid5 on a Marvell 8-port sata controller, and see at least twice as much for writing.Not to mention that the Marvell 8-port controller is much, much cheaperthan a 3ware 9500S controller.With 3 controllers and 24 WD Raptors raid5, 500MB/sec should not be a problem.Can you try this for me, just for fun? It creates four 10GB files.for i in `seq 4`do time dd if=/dev/zero of=bigfile.$i bs=1024k count=1time dd if=bigfile.$i of=/dev/null bs=1024k count=1 donetime rm bigfile.*With a modern 'dd' it will also output the MB/sec speed.Kind regards, Sander--Humilis IT Services and Solutionshttp://www.humilis.net -- -Tim
Re: [Veritas-bu] Disk backup - Storage Hardware configurations
I have two Linux media servers setup with 3ware 9500S controllers using raid "50" with seagate disks. With our configuration this is two raid 5's of 6 disks each that are then striped together. I get up to about 70MB/sec writes and 300MB/sec reads after the readahead is tweaked up. They are used for DSSU's going to IBM LTO3's and I'm seeing up to 160MB/sec writes to tape. I'm very happy so far.I get better read performance with raid5 or raid50 than with raid 10. But I use raid 10 for the catalog raid for best write performance. I see a bit over 100MB/sec reads & writes with raid 10. On 3/22/06, Ambrose, Monte <[EMAIL PROTECTED]> wrote: I wanted to get a feel for what type of disk storage is being used out there for disk based backups. I am specifically looking for SAN attached storage ATA, SATA and what type of RAID is used and how it is performing overall. The NBU 6.0 tuning guide shows RAID-5 but I am wondering if anyone is using RAID-10 or others. Thanks, Monte -- -Tim
Re: [Veritas-bu] backup speed
With LTO3, you *really* ought to be using disk staging if at all possible. As someone already stated, the minimum speeds are quite high. For IBM LTO3, it's 40MB/sec. If you can't meet the mimimum speeds you're going to prematurely wear out your tapes and drives and of course, you're not making good use of these awesome drives. I'm using Linux staging media servers and seeing up to 160MB/sec for 2GB fragments. With this throughput, one drive per media server does quite nicely. cat /usr/openv/netbackup/NET_BUFFER_SZ65536was the single most important tweak I made to get better performance with these drives.On 3/20/06, Atif Munir <[EMAIL PROTECTED]> wrote:I have a gigabit ethernet. Recently i have installed 5 LTO Gen 3 drives. But I am not getting a very good speed. As a total I amgetting 100Mb/sec, and it take a lot of time to finish my backup.Why I could not get a good speed.Regards,atif___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.eduhttp://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- -Tim
Re: [Veritas-bu] Change server name in NBU 6.0
Out of curiosity, what exactly qualifies as a host name change? Only the "short" name or the FQDN? If "foo.bar.com" goes to "foo.bat.com", is that a hostname change? Or how about this one: a host is only known as "foo" but gets changed to "foo.bar.com". I'm really not clear on this. Thanks! On 2/16/06, Ed Wilts <[EMAIL PROTECTED]> wrote: > On Thu, Feb 16, 2006 at 07:16:34AM -0800, Thomas Jones wrote: > > Does anyone know how to move the NetBackup master/media server to a > > new hostname in 6.0, Sybase DB and all (tape drives, robot, media, > > ...)? > > Moving the media server isn't that hard but moving the master is > definitely not supported and non-trivial. Keep the master host name. > > -- > Ed Wilts, Mounds View, MN, USA > mailto:[EMAIL PROTECTED] > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Netbackup 6 and nearstore storage units with millions of files per qtree
We're still at 5.1 right now, but I'm reading over the 6.0 docs, and it looks like they've added a really useful option called nearstore storage units. Has anyone here tried it out? We have 3 netapps that snapvault over to an R200 and we use NDMP to back them up. Some of our qtrees have several million files. The three problem qtrees have 12M, 17M, and 23M files in them. It can take 48 hours to snapvault them over. Because the target then also has millions of files, backup performance is horrible, and this is still with AIT2. Part of the problem is our R200 has 5400 RPM disks, but I don't think this is the primary issue. The problem is, we are migrating to LTO3. We can't continue on this path! It *appears* that nearstore storage units might help address this issue because the target uses netbackup tarballs. No more need to create millions of inodes on the target and no more need to traverse them to back them up. If anyone has tried this out with millions of files per qtree, please let us know if this new option has helped at all. Thanks! -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Examples of SSO vm.conf files
The drive name hagrid-1 should correspond to the same physical drive across all hosts.On 1/28/06, Kathryn Hemness < [EMAIL PROTECTED]> wrote:So the hagrid-* names are the logical names, right? And I should have used the same names with the media server device paths when I configuredthem to Robot 0 with tpconfig on the media server?On Sat, 28 Jan 2006, Tim Berger wrote:> Date: Sat, 28 Jan 2006 11:38:00 -0800 > From: Tim Berger <[EMAIL PROTECTED]>> To: Kathryn Hemness <[EMAIL PROTECTED]>> Cc: veritas-bu@mailman.eng.auburn.edu> Subject: Re: [Veritas-bu] Examples of SSO vm.conf files>> Kathryn, each physical drive needs to have a unique name, not each logical> one.>> For example, drive 1, hagrid-1, should be hagrid-1 across all media servers. >> The output of>> vmdareq -h masterserver>> should show you eash drive and corresponding media servers sharing it.>> You'll get there soon enough. ;-)> -- -Tim
Re: [Veritas-bu] yes sir, obviously insane (6.0 upgrade question)
The only problem I had with SSO on Linux with 5.1MP3 was adding tape drives that were in use by other hosts. Other than that, it's working fine on our spectra 64k with AIT-2 (eeep). I'm upgrading to LTO-3 real soon now. That's why I'm moving to Linux - for the speed and massive storage for dssu's for staging and restores. The master is still an ancient, slow e450 at the moment. I've replaced four of our 420R's with two linux servers and I'm real happy so far. On 11/30/05, Jeff Lightner <[EMAIL PROTECTED]> wrote: > I don't have answers to what you wrote but did want to caution you about > using RH for your master. I've not been able to get RH EL AS 3 to > properly recognize my SAN environment. From an earlier thread on this > list it appears I wasn't the only one. > > Not sure if you're running SAN or direct SCSI but if the former thought > it best to mention it. > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Dan Dobbs > Sent: Wednesday, November 30, 2005 1:21 PM > To: Veritas-bu@mailman.eng.auburn.edu > Subject: [Veritas-bu] yes sir, obviously insane (6.0 upgrade question) > > Greetings, list. > > Here's my situation. I have a HP-UX media/master server on 5.1 with a > newer Neo SDLT bot and an ancient DLT4000 Surestore bot. We're going to > be getting a shiny new Red Hat box soon to take over migrate backup > services. > > I've just read the upgrade docs for 6.0, and I have to say that I'm a > little overwhelmed. If I read it right, an upgrade to 6 requires all the > clients to get the security software installed and upgraded to 6 as well > before they can get backed up, something I'm not sure I can accomplish > in a day. > > In short, I'm considering leaving my HP-UX box at 5.1, and building the > new RH box on 6. My plan was to remove the Neo SDLT juke out of the 5.1 > environment, plug it into my RH box,and import tapes there (as we've > used less than 30 SDLT's, it shouldn't be a huge deal). This would allow > me to do the icky client-side upgrades piecemeal, as I moved them to > 6.0, they would then back up to the new box. Eventually, as all the > clients moved over, I could then move the old jukebox and stop > operations on the HP-UX box. > > The obvious problem is recovery; until I import every single tape, I > wouldn't know what tapes had backups from the 'old' system. > > I guess the thrust of the matter is: > > 1) Is the upgrade of an in-place 5.1 server to 6 easier than it reads > on paper? > 1a) Do I really have to upgrade my clients to 6 before the server will > talk to them? > 2) If I was running 6 on the 'old' system, could I follow the previous > advice, and move /usr/openv/netbackup/db and > /usr/openv/volmgr/database, run the proper vmglob and bpmedia commands > on the new box and be done? > > Thanks again for all your help! > > -dd > > > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > -- -Tim ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
