Re: Troubles, troubles, troubles.
Rick, We had the same problem that went away with 4.1.1.11. -Bern Ruelas Sr. Systems Engineer - Storage Cadence Design Systems -Original Message- From: Rushforth, Tim [mailto:[EMAIL PROTECTED]] Sent: Friday, June 07, 2002 8:44 PM To: [EMAIL PROTECTED] Subject: Re: Troubles, troubles, troubles. Try running manual backup, does client abend? Can you narrow down the directories backed up to determine if it is one bad file/location on disk Check for drwatson log files. I've seen this happen on a bad sector of disk which chkdsk fixed. -Original Message- From: Rick Harderwijk [mailto:[EMAIL PROTECTED]] Sent: June 6, 2002 7:11 AM To: [EMAIL PROTECTED] Subject: Troubles, troubles, troubles. Hi all, One of my servers is giving me a major headache at the moment. Every morning when I come in, I check, as anyone does, I presume, whether all scheduled backups completed succesfully. The past couple of days, one of my servers doesn't complete, nor does it fail. Or so it tells me, because the result of the scheduled backup is '(?)' . Nice, isn't it? Since I'm not going to lay awake all night because of one backup, I decided the first time it happened, to just let it go. 'It'll all be better tomorrow,', I told myself. Unfortunately, it didn't get better. Nope, the same thing happened the next night, and the one after. So, this morning, I started to check if there was a serious glitch. So I checked the dsmsched.log on the client. Strange thing... it sent a file, then it gets a new schedule. It feels as if something is missing here. Next, I checked the dsierror.log. Three entries here: 06-06-2002 03:52:27 sessOpen: Error 137 from signon authentication. 06-06-2002 03:53:30 sessOpen: Error 137 from signon authentication. 06-06-2002 03:54:33 sessOpen: Error 137 from signon authentication. Messages like that make me pop out a fresh browser window, call up www.adsm.org and go looking for that message... as it turns out 'this message just is there, can be ignored and will go away in the next couple of upgrades'. So, not very helpful *but* it also tells me this message occurs when the Scheduler service starts. Now there's something I can do something with. Opening up ye olde Eventlog on that same W2k-client tells me that indeed, at that time, the Scheduler Service was started - it had stopped just before, and since we've had some problems with earlier versions with services just quitting on us, the services are automatically restarted when they go down. As it turns out, the backup stopped when the service crashed and when the service came back, it just got a new schedule and the backup in progress stopped and never continued. It seems to me that all I have to fix is that darned Scheduler Service. But where to start? Why does it stop? Where to look for more clues? And ofcourse: why me? :-) Anyone care to join me and hunt this bugger down to fix it once and for all? You're most welcome - I feel like I'm on a dead end trail... and that headache is keeping me from thinking clearly... Kind regards, Rick Harderwijk Systems Administrator Factotum Media BV Oosterengweg 44 1212 CN Hilversum P.O. Box 335 1200 AH Hilversum The Netherlands Tel: +31-35-6881166 Fax: +31-35-6881199 E: [EMAIL PROTECTED]
Re: Tuning TSM
Hi Don, I looked for "incrbydate" in the installation guide and reference manual for this option but haven't found it. Can I use this for Solaris clients? -Bern Ruelas Cadence Design Systems - Storage Sr. Systems Engineer -Original Message- From: Don France (TSMnews) [mailto:[EMAIL PROTECTED]] Sent: Friday, May 17, 2002 2:27 PM To: [EMAIL PROTECTED] Subject: Re: Tuning TSM Reading thru this thread, no one has mentioned that backup will be slower than archive -- for TWO significant reasons: 1. The "standard" progressive-incremental requires alot of work in comparing the attributes of all files in the affected file systems, especially for a LARGE number of files/directories (whereas archive has minimal database overhead -- it just moves data). 2. Writes to disk are NOT as fast as tape IF the data can be delivered to the tape device at "streaming" speed; this is especially true if using no-RAID or RAID-5 for disk pool striping (with parity)... RAID-0 might compete if multiple paths & controllers are configured. The big advantage to disk pools is more concurrent backup/archive operations, then disk-migration can stream offload the data to tape. So, firstly, debug fundamentals using tar and archive commands (to eliminate db overhead comparing file system attributes to identify "changed" files/objects); once you are satisfied with the thruput for archive, allow 20-50% overhead for daily incremental. If your best "incremental" experience is not satisfactory, (but archive is okay) consider other options discussed in the performance-tuning papers -- such as, reducing the number of files per file system, use incrbydate during the week, increase horsepower on the client machine and/or TSM server (depending on where the incr. bottlenecks are). The SHARE archives do not yet have the Nashville proceedings posted; when they do show up, they are in the members-only area (I was just there, searching for other sessions). - Original Message - From: "Ignacio Vidal" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, May 17, 2002 6:30 AM Subject: Re: Tuning TSM Zlatko: OK: tcpwindowsize is in KBytes (actually my setting is 1280). I have the same question (why client is idling?). We have users working on that server from 9:00 until 22:00 (every day...), backup jobs start about 00:15/01:15 I've monitored the nodes in disk i/o operations and network transfers, in different moments of the day. About cpu load/memory usage/pagination: the values are all OK, for example: - cpu load (usr) has an average of 5~7 during all day - memory usage (have 7GB RAM) is not a problem (about 30~40% for computational the same for noncomp) - pagination: max use may be 10~12% (mostly of the day 0.5%, peaks during user's work time) Viewing your results (500GB in 3:10hs), and trying to compare: we're backing up 250GB starting 00:15 and ending 07:00... 250GB in 6:45hs (it's not good) Yesterday I set resource utilization = 10 too, and performance was the same (really a bit worst). I think there are multiple problems (and our provider -IBM- cannot help us): First of all: disk performance (IBM should tell us how to get the best from FastT500), we have from 15 to 25 MB/sec in the storage... Then: all TSM nodes in our installation have not the same file configuration. I explain a bit more this: we have nodes merging a lot of files (>25) with an average size of 40KB each and a few files (<1000) with an average size of 50MB (it's Oracle Financials: the database server keeps datafiles and files belonging to application and DB motor). We have 4 nodes such as just described, with a total of 35~40GB for each (average and growing...) Well, here was a brief description. I'm listening for new ideas. Thanks Ignacio > -Mensaje original- > De: Zlatko Krastev [mailto:[EMAIL PROTECTED]] > Enviado el: viernes, 17 de mayo de 2002 5:29 > Para: [EMAIL PROTECTED] > Asunto: Re: Tuning TSM > > > Just a point > TCPWindowsize parameter is measured in kilobytes not bytes. > And according > to Administrator's reference it must be between 0 and 2048. If not in > range on client complains with ANS1036S for invalid value. On server > values out of range mean 0, i.e. OS default. > However this are side remarks. The main question is why > client is idling. > Have you monitored the node during to disk and to tape operation? Is > migration starting during backup? Are you using DIRMC. > You wrote client compression - what is the processor usage > (user)? What is > the disk load - is the processor I/O wait high? Is the paging > space used - > check with svmon -P. > You should get much better results. For example recently > we've achieved > 500 GB in 3h10m - fairly good. It was similar to your config - AIX > node&server, cli
Is there a 5.1 client for Solaris?
On the ftp site: ftp://service.boulder.ibm.com/storage/tivoli-storage-management/patches/ the v5r1 server and storage agent for Solaris are there, however there is no client code. Is this planned? -Bern Sr. Systems Engineer Cadence Design Systems - Storage
Re: SUN FC HBA adapter
Hmmgood point Jozefwell, maybe I'm missing something too. Did you run "drvconfig" then "tapes" after you rebooted? I've acquired the devices several times doing these two commands. Note that in almost every instance, I needed to double check with the network guys to ensure that my fibre connections were zoned correctly. -Bern -Original Message- From: Jozef Zatko [mailto:[EMAIL PROTECTED]] Sent: Tuesday, May 14, 2002 12:02 AM To: [EMAIL PROTECTED] Subject: Re: SUN FC HBA adapter Bern, thank you for your advice. But I have non IBM DLT drive. The problem is, that software driver for my HBA does not create standard SCSI devices (with SCSI target and LUN) in Solaris, so it is not possible to use TSM device driver, which is AFAIK, driven to use only SCSI devices. This is part of my messages file: /pci@4,4000/SUNW,qlc@3/fp@0,0/st@w10e00221f7f5,0 (st85): May 7 18:32:53 tsm May 7 18:32:53 tsm unix: st85 at fp0: May 7 18:32:53 tsm unix: name w10e00221f7f5,0, bus address 1 May 7 18:32:53 tsm unix: st85 is /pci@4,4000/SUNW,qlc@3/fp@0,0/st@w10e00221f7f5,0 May 7 18:32:53 tsm unix: /pci@4,4000/SUNW,qlc@3/fp@0,0/st@w10e00221f7f5,0 (st85) online AFAIK TSM device driver for nonIBM devices support only SCSI devices, not FC devices. Or am I missing something? Now I am going to change Fcode in HBA so that it will be original Qlogic 2200 and I will use Qlogic device drivers. According to Tivoli it should be supported configuration. Ing. Jozef Zatko Login a.s. Dlha 2, Stupava tel.: (421) (2) 60252618 Bern Ruelas <[EMAIL PROTECTED]> Sent by: "ADSM: Dist Stor Manager" <[EMAIL PROTECTED]> 09.05.2002 20:11 Please respond to "ADSM: Dist Stor Manager" To: [EMAIL PROTECTED] cc: Subject:Re: SUN FC HBA adapter Hello Jozef, I just got thru doing something like this with a similar setup: we use Emulex LP8000 cards for the HBA's onto an IBM 3584 with LTO drives and a Mcdata switch. There are 4 things that need to be done or it won't work: 0. Ensure that you've moved the /kernel/drv/st.conf somewhere else. Solaris will try and grab your scsi tape devices otherwise. 1. Make sure your fabric is set up correctly and that at boot up (ck the /var/adm/messages files) you see your IBMtape device drivers and HBA acquired targets. Here's a clip of what I'm talking about: May 3 18:04:35 bst7-1 unix: IBMtape254 at lpfc1: May 3 18:04:35 bst7-1 unix: IBMtape178 at lpfc1: May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 1 May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 2 May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 3 3. Next check that your /usr/kernel/drv/IBMtape.conf file has the requisite objects not #'d out. We typically leave all 16 instances open. Bern Ruelas Sr. Systems Engineer - Storage Cadence Design Systems -Original Message- From: Jozef Zatko [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 09, 2002 6:32 AM To: [EMAIL PROTECTED] Subject: SUN FC HBA adapter Helo *SM-ers In my test environment I am trying to connect TSM server on Solaris to SAN-attached SCSI library Compaq TL891. My environment is as follows: TSM server 4.2.1.9 on Sun Solaris 2.8 with Sun PCI FC HBA (OEM of Qlogic 2200) - part No. X6799A with device driver Qlogic FCA Driver v0.40.5. Library contains one Quantum DLT7000 drive and is attached to Crossroads 4250. TSM server and Crossroads are attached to FC switch Brocade Silkworm. In Solaris I can see DLT drive and Library, but when I try to use TSM device driver (using command "add_drv mt") it runs without any error messages, but it does not create special device file in directory /dev/rmt. Does anyone of you, guys, have gone through this with success? If yes, can you please share your configuration and versions of device drivers, firmware and so on? I am also going to ask Tivoli support if Sun FC HBA has already been tested with TSM. Ing. Jozef Zatko Login a.s. Dlha 2, Stupava tel.: (421) (2) 60252618
Re: starting tsm server on solaris
HI Gerald, That is a very offending line. We've removed it. Not only that, but Solaris usually doesn't start daemons (except the console) in that way. Solaris uses run control scripts. We put ours in /etc/init.d. It is pointed to from /etc/rc3.d which starts up when the system goes into multiuser mode. The script we use is: # cat /etc/init.d/dsmcad_v1 #!/bin/sh DSM_CONFIG=/opt/tivoli/tsm/client/ba/bin/dsm.opt.v1 export DSM_CONFIG /bin/dsmcad-v1 -optfile=/opt/tivoli/tsm/client/ba/bin/dsm.opt.v1 > /var/log/nohup-v000 2>&1 & We call it from a link in rc3.d: lrwxrwxrwx 1 root other 21 Apr 19 16:24 S88dsmcad_01 -> /etc/init.d/dsmcad_v1 We also put in this line in the dsm.sys file: ManagedServices Schedule WebClient That way dsmcad runs the dsmc daemon by turning it on and off. This gets rid of the dsmc memory problem. Another reason we went to this solution is because we use console servers in order to do work from anywhere. Using that original inittab line actually hangs the console so you can't get into it from the console server, thus making the console server fairly useless. Hope this helps, -Bern -Original Message- From: Gerald Wichmann [mailto:[EMAIL PROTECTED]] Sent: Monday, May 13, 2002 4:34 PM To: [EMAIL PROTECTED] Subject: starting tsm server on solaris dsmserver:234:once:/opt/tivoli/tsm/server/bin/dsmserv >/dev/null 2>&1 Put that line into /etc/inittab but on reboot the TSM server doesn't come up.. I have to cd into the bin directory and manually start it.. I notice you have to set an environmental variable to start the tsm server from another directory other then it's installed directory but in order for it to work in inittab, I'm not sure where to put the environmental variable nor even if that's why it isn't starting (though I suspect it is why). Appreciate any help. Not used to solaris.. Regards, Gerald Wichmann Senior Systems Development Engineer Zantaz, Inc. 925.598.3099 (w)
How do you eject dbsnapshot tapes automagically
I'd like to set my system up to automagically eject the dbsnapshots when they are done via an admin command script. Does anyone know how? -Bern Sr. Systems Engineer Cadence Design Systems
Re: SUN FC HBA adapter
Hello Jozef, I just got thru doing something like this with a similar setup: we use Emulex LP8000 cards for the HBA's onto an IBM 3584 with LTO drives and a Mcdata switch. There are 4 things that need to be done or it won't work: 0. Ensure that you've moved the /kernel/drv/st.conf somewhere else. Solaris will try and grab your scsi tape devices otherwise. 1. Make sure your fabric is set up correctly and that at boot up (ck the /var/adm/messages files) you see your IBMtape device drivers and HBA acquired targets. Here's a clip of what I'm talking about: May 3 18:04:35 bst7-1 unix: IBMtape254 at lpfc1: May 3 18:04:35 bst7-1 unix: IBMtape178 at lpfc1: May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 1 May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 2 May 3 18:04:35 bst7-1 unix: NOTICE: lpfc1: Acquired FCP/SCSI Target 9 LUN 3 3. Next check that your /usr/kernel/drv/IBMtape.conf file has the requisite objects not #'d out. We typically leave all 16 instances open. Bern Ruelas Sr. Systems Engineer - Storage Cadence Design Systems -Original Message- From: Jozef Zatko [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 09, 2002 6:32 AM To: [EMAIL PROTECTED] Subject: SUN FC HBA adapter Helo *SM-ers In my test environment I am trying to connect TSM server on Solaris to SAN-attached SCSI library Compaq TL891. My environment is as follows: TSM server 4.2.1.9 on Sun Solaris 2.8 with Sun PCI FC HBA (OEM of Qlogic 2200) - part No. X6799A with device driver Qlogic FCA Driver v0.40.5. Library contains one Quantum DLT7000 drive and is attached to Crossroads 4250. TSM server and Crossroads are attached to FC switch Brocade Silkworm. In Solaris I can see DLT drive and Library, but when I try to use TSM device driver (using command "add_drv mt") it runs without any error messages, but it does not create special device file in directory /dev/rmt. Does anyone of you, guys, have gone through this with success? If yes, can you please share your configuration and versions of device drivers, firmware and so on? I am also going to ask Tivoli support if Sun FC HBA has already been tested with TSM. Ing. Jozef Zatko Login a.s. Dlha 2, Stupava tel.: (421) (2) 60252618
Re: Solaris TSM Server Performance
Hi Brenda, There are several commands we used to set this up: DEFINE DBVOLUME /dev/rdsk/c2t0d0s4 FORMATSIZE=24000 WAIT=NO DEFINE LOGVOLUME /dev/rdsk/c1t0d1s4 FORMANTSIZE=1 WAIT=NO They format in about a minute and we are getting an average of about 50-60 GB/hr transfer rate. On Exchange tests we ran an average of 80 GB/hr to back up and 39 GB/hr to restore the full database. Remember that using raw volumes is quick but it can EASILY get you into trouble. Make *very* sure you have this mapped out and know *exactly* which slices you are using and where. -Bern Ruelas Sr. Systems Engineer Cadence Design Systems - Storage -Original Message- From: Brenda Collins [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 09, 2002 8:34 AM To: [EMAIL PROTECTED] Subject: Solaris TSM Server Performance We are having trouble with a Solaris Server we just set up. The disk was set up as filesystems but the performance was so poor that it was recommended we stripe the disk and set it us as raw logical volumes. We are now having problems with defining the raw volumes to TSM.Does anyone have more detailed instructions than what is in the reference book? Also, could you tell me what types of performance you are seeing on your backups? Any help would be appreciated. Thanks, Brenda Collins, ING Americas Infrastructure Services
Re: Backup of Sun OS
Hi Andrea, There is a product just for that - The Kernel Group's "Bare Metal Restore". -Bern HI everybody, The first thing I have to say to this is that I am NOT a UNIX guru, so I greatly would appreciate all the help I can get. The problem is that our Unix team wants to backup a running Sun OS and then later on restore this backed up OS backup and boot the machine from there, which of course did not work. I know AIX has the make sysb solution for this, does anybody know of a solution for SUN. TSM version 4.2.1.9 on AIX 4.3.3 Thanks Andrea Mac Murray Sen. Systems Administrator ConAgra Foods, Inc. 7300 World Communication Drive Omaha,NE 68122 Tel: (402) 577-3603 [EMAIL PROTECTED] Bern Ruelas| <[EMAIL PROTECTED]>_ Sr. Systems Engineer | () |_> Cadence Design Systems | /\___|__ 2655 Seely Ave.|Are you excited?... \ \ / | \ San Jose, CA 951344| /_ \o) __\__/__ On Fri, 25 Jan 2002, MacMurray, Andrea (CC-ETS Ent Storage Svcs) wrote: [NON-Text Body part not included]
Re: Drive Mapping for W2K over SAN to LTO with Sun TSM server
Paul, Thanks for the tip. We figured it out and here's how: Note that I'm including explicit details because we never found these in any Redbook or manual...and much of this came from our experience with IBMtape's tapeutil program running on Sun. List the drives: C:\Program Files\Tivoli\TSM\Storageagent\tsmdlst Then map the drives: 0.) Load a tape into every drive in your jukebox. 1.) On your NT client go to and envoke: C:\Program Files\IBM Corporation\IBM Ultrium device drivers\5.0.2.4\ntutil and within this program: 2.) #1 set special device name 3.) Tape0 (Or the name of your device) 4.) #20 Open 5.) #1 RW 6.) #33 unload 7.) #21 close -Bern At 12:26 AM 12/15/2001 -0500, Seay, Paul wrote: >I can tell you right now that the QLOGIC card is going to have problems on >the W2K server. Every time you do a fabric update a RSCN is sent out and >QLogic will kill the drives in W2K and require a reboot to get them back. >The fix for this problem is an unreleased driver 8.1.3 that is supposed to >go GA in a couple of weeks. > >This is standard stuff. Follow the instructions in the book otherwise. I >can probably dig up all the commands that you will need to issue. I can >tell you this is really slick stuff based on our experiences. So far, we >have only used AIX and Solaris Storage Agents, but we will be doing W2K next >week on exactly what you are doing except it is an AIX TSM server. > >-Original Message- >From: Bern Ruelas [mailto:[EMAIL PROTECTED]] >Sent: Friday, December 14, 2001 9:40 PM >To: [EMAIL PROTECTED] >Subject: Drive Mapping for W2K over SAN to LTO with Sun TSM server > > >Hello, > >Has anyone ever mapped a Storage Agent running TDP for Exchange from a >Win2k client to an LTO tape device on a Sun Solaris (5.7) TSM server? > >Here's the setup: > >Server: SAN:Client: > >Sun E420SAN Data GatewayWin2K >TSM 4.2.1.72108-G07 Storage Agent >4.2.1.7 >Emulex LP8000 TDP for Exchange 2.2 >Fiber Channel cards ==> McData Switch BA Client 4.2.1 >IBM 3584 LTOUltrium Device >Driver 5.0.2.4 >IBM 2580 Ultrium Tape DrivesQLogic QLA2200 PCI >Fiber Adapter 8.0.9.6 > >Any clues would help. > >Thanks, > >-Bern Ruelas >Sr. Systems Engineer - Storage >Cadence Design Systems >2655 Seely Ave. >San Jose, CA 95134 >(408) 428-5246
Drive Mapping for W2K over SAN to LTO with Sun TSM server
Hello, Has anyone ever mapped a Storage Agent running TDP for Exchange from a Win2k client to an LTO tape device on a Sun Solaris (5.7) TSM server? Here's the setup: Server: SAN:Client: Sun E420SAN Data GatewayWin2K TSM 4.2.1.72108-G07 Storage Agent 4.2.1.7 Emulex LP8000 TDP for Exchange 2.2 Fiber Channel cards ==> McData Switch BA Client 4.2.1 IBM 3584 LTOUltrium Device Driver 5.0.2.4 IBM 2580 Ultrium Tape DrivesQLogic QLA2200 PCI Fiber Adapter 8.0.9.6 Any clues would help. Thanks, -Bern Ruelas Sr. Systems Engineer - Storage Cadence Design Systems 2655 Seely Ave. San Jose, CA 95134 (408) 428-5246
Re: LTO restore performance - seek problem?
John, This problem is very interesting to me as we are thinking of throwing away our 17 ATL libraries running Legato's NetWorker 6.0.1 in favor of 2 LTO 3584's running Tivoli Storage Manager (we will actually use a Sun E420). I thought colocation was suppose to speed up recoveries, it sounds as if it's not working as advertized -Bern At 02:47 PM 5/4/2001 +0200, Karsten Hüttmann wrote: >Content-Type: text/plain; charset=iso-8859-1 >X-MIME-Autoconverted: from 8bit to quoted-printable by mailserver.carus.de >id OAA14706 > >Hi! >Your problem sounds familiar to me. >We've similar problems running HSM (or now: Tivoli Space Manager) with >a LTO 3583. (Ok, it is not the best way to use a LTO Library with >HSM, but that was not my decision). For a recall of 2 GBs we need half a >day. >But: a normal restore of a client is as fast as we expected. > >John Schneider wrote: > > > Greetings, > > I have seen many posts on adsm.org about LTO, but none that > > addresses the problem I am seeing. Backup speed seems to be fine, at > > least as far as I can tell. But when I do a restore, I see a strange > > behavior. The restore seems to happen in bursts, where the restore > > pulls back files lickety-split for a awhile, then stops for 15-20 > > minutes, then continues. I think that the wait is the seek from one > > spot on the tape to another. We have two tape drives in the IBM 3583 > > library, and the problem seems to be happening on both of them. We > > upgraded to the 0CE1 version of the LTO microcode, but it has not > > improved things that I can tell. > > We restored 1.5 GB from our RS/6000 TSM server to a NT client, and > > it took 47762 seconds, or 13 hours. All the data was on one tape, so it > > is not a colocation problem. How long should it take an LTO drive to > > seek? I have done these sort of restores on DLT, and it can usually get > > from any place to any place on a tape within a few minutes. I expected > > LTO to be at least as good. A 13 hour restore is completely > > unacceptable. > > Any input or things to try would be appreciated. By the way, the > > TSM server is an RS/6000 F50 running AIX 4.3.3 ML4, with TSM 4.1.2. It > > is running atape 6.0.4, and atldd 4.1.5.0, which are both the latest, I > > believe. The NT client was TSM 4.1.2. Both are on the same switch on a > > 100MB Ethernet network, so I don't think the network is related in any > > way. > > > > TIA, > > John Schneider > > > > *** > > * John D. Schneider Email: [EMAIL PROTECTED] * Phone: 636-492-0247 > > * Lowery Systems, Inc. > > * 1329 Horan Disclaimer: Opinions expressed here are > > * Fenton, MO 63026 mine and mine alone. > > *** > >-- >Mit freundlichen Grüssen / with regards >Karsten Hüttmann > >Content-Type: text/x-vcard; charset=us-ascii; > name="Karsten.Huettmann.vcf" >Content-Description: Card for Karsten H|ttmann >Content-Disposition: attachment; > filename="Karsten.Huettmann.vcf" >X-MIME-Autoconverted: from 8bit to quoted-printable by mailserver.carus.de >id OAA14706
