Re: Slow restore for large NT client outcome.. appeal to Tivoli
Correction... the traceflag is instr_client_detail, then use tracemax=4000 (for wrap-around) and tracefile=xxx.out. Don France Technical Architect - Unix Engineering/P.A.C.E. San Jose, CA mailto:[EMAIL PROTECTED] PACE - http://www.pacepros.com Bus-Ph: (408) 257-3037 -Original Message- From: France, Don G (Pace) Sent: Friday, May 04, 2001 9:50 PM To: [EMAIL PROTECTED] Subject:RE: Slow restore for large NT client outcome.. appeal to Tivoli I have two fundamental suggestions, I have not seen item 2 mentioned yet: 1. Use Microsoft folder-copy (be sure to run WinDiff - it's notorious for *not* copying all the files/directories) - and compare performance for copying large number of customer files; 2. Run the instrum_client_detail trace - this is a client-level trace option, very cool, documented in a number of old ATSC and Performance info papers from the old ADSM days... it still works, and provides a very clear view of where time is being spent during backup/restore. We are exactly in that position with a very large client; we just ordered 6-drive, 72-slot LTO and H80 to handle existing 1.6 TB of file served data occupancy across 5 NT servers, some 7 million files/dirs. We will also mitigate the failure-recovery scenario to a large extent with fibre-attached EMC storage, replacing old dual P-Pro with latest Compaq server gear. As part of our little fix-it project, we will be doing some benchmark measurements; I, for one, plan to use both of the above listed methods to confirm/reset our expectations... sure hope we can do better than 5-10 GB/Hr restore speed. (Maybe will try Win2K as well as NT - if there is a possibility of keeping the resulting member server.) BTW, we did get some numbers from Tivoli/IBM indicating NT is the slowest, but all 3 platforms (NT, Solaris, AIX) are slowest with large numbers of small files (AIX is, by far, the fastest in their benchmarks!). We considered switching to Samba-like solution (with AIX), just so we could use the IMAGE backup for fast restore; that is how ArcServe can beat TSM - also, that's (somewhat) how Replica beats TSM, and that's one advantage of TDP for Workgroups... too bad the owning children could not get along well-enough to move that solution forward (ie, sending TDPfW data across LAN to TSM server). At this point, we're betting on (a) EMC reliability recoverability (RAID-S), (b) combination of EMC and newer Compaq gear for better performance, and (c) TSM-5.1 (1q02) delivery of image backup for Win2K! Regards, Don -Original Message- From: Kelly J. Lipp [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 2:38 PM To: [EMAIL PROTECTED] Subject:Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the incremental forever logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys
Re: Slow restore for large NT client outcome.. appeal to Tivoli
I have two fundamental suggestions, I have not seen item 2 mentioned yet: 1. Use Microsoft folder-copy (be sure to run WinDiff - it's notorious for *not* copying all the files/directories) - and compare performance for copying large number of customer files; 2. Run the instrum_client_detail trace - this is a client-level trace option, very cool, documented in a number of old ATSC and Performance info papers from the old ADSM days... it still works, and provides a very clear view of where time is being spent during backup/restore. We are exactly in that position with a very large client; we just ordered 6-drive, 72-slot LTO and H80 to handle existing 1.6 TB of file served data occupancy across 5 NT servers, some 7 million files/dirs. We will also mitigate the failure-recovery scenario to a large extent with fibre-attached EMC storage, replacing old dual P-Pro with latest Compaq server gear. As part of our little fix-it project, we will be doing some benchmark measurements; I, for one, plan to use both of the above listed methods to confirm/reset our expectations... sure hope we can do better than 5-10 GB/Hr restore speed. (Maybe will try Win2K as well as NT - if there is a possibility of keeping the resulting member server.) BTW, we did get some numbers from Tivoli/IBM indicating NT is the slowest, but all 3 platforms (NT, Solaris, AIX) are slowest with large numbers of small files (AIX is, by far, the fastest in their benchmarks!). We considered switching to Samba-like solution (with AIX), just so we could use the IMAGE backup for fast restore; that is how ArcServe can beat TSM - also, that's (somewhat) how Replica beats TSM, and that's one advantage of TDP for Workgroups... too bad the owning children could not get along well-enough to move that solution forward (ie, sending TDPfW data across LAN to TSM server). At this point, we're betting on (a) EMC reliability recoverability (RAID-S), (b) combination of EMC and newer Compaq gear for better performance, and (c) TSM-5.1 (1q02) delivery of image backup for Win2K! Regards, Don -Original Message- From: Kelly J. Lipp [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 2:38 PM To: [EMAIL PROTECTED] Subject:Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the incremental forever logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has
Re: Slow restore for large NT client outcome.. appeal to Tivoli
So I'd heard, but I don't have the liberty of choice, so I'm stuck with it on the AS system. Mike -Original Message- From: Joshua S. Bassi [SMTP:[EMAIL PROTECTED]] Sent: ä ñôèîáø 21 2000 17:33 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli AS/400 you say? It's probably the slowest *SM server out of the 9 supported. NT TSM server are typically much faster than their 400 counterparts. -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Mike Glassman - Admin Sent: Wednesday, September 20, 2000 11:58 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has some tricks or tweaks that can help in this area. Anyone from any universities out there? Reply Separator Subject:Slow restore for large NT client outcome.. appeal to Tivoli Author: Jeff Connor [EMAIL PROTECTED] Date: 09/20/2000 12:21 PM Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't impro
Re: Slow restore for large NT client outcome.. appeal to Tivoli
To answer your q's James, 1. ADSM sits on an AS400. BE sits on either an NT seerver or another Novell server. Client on the server is (ADSMwise) version 3, Release 1, Level 0.8. 2. Network is Ethernet. All servers are connected 100MBHD. Backbone is 650MB FO. 3. Damned if I know I'm afraid. On the Netware and NT is a DLT 35/70MB. I don't know the model on the AS, but it's a changer (IBM) auto with 350 Tapes installed (One huge monster of a box). 4. Over 4000 files in the directory. 5. The AS server is dedicated for ADSM, altho it can be a backup system to one of our other AS systems (holds the files which are stored there once a day via VISION link and not the Network link). I too do a lot more single file restores then full directory or system, and thank the gods for that. I cannot even try to imagine a full system restore at these slow rates. Single file restores also take their time, but I can wait 5 minutes for a file as apposed to having to wait 6 hours for 4000 files, or longer if I need to restore Giga's of data. All restores on ADSM are done via the web interface, which may possibly have something to do with speed, altho I doubt to that extent. I live with it. Sadly, but I do. Mike -Original Message- From: Purdon, James [SMTP:[EMAIL PROTECTED]] Sent: ä ñôèîáø 21 2000 15:55 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli When reading this I have to ask 600Mb how? 1. What kind of systems (ADSM/BE server, client)? 2. What kind of network? 3. What kind of tape devices? 4. How many files in the directory? 5. Is the ADSM server shared or dedicated? Its possible to create patholgical cases that can trash any backup system's backup and restore times. In our case, thanks to raid, we do a lot more single file restores than restores of entire filesystems. Jim -- From: Mike Glassman - Admin[SMTP:[EMAIL PROTECTED]] Reply To: ADSM: Dist Stor Manager Sent: Thursday, September 21, 2000 2:57 AM To: [EMAIL PROTECTED] Subject:Re: Slow restore for large NT client outcome.. appeal to Tivoli Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that
Re: Slow restore for large NT client outcome.. appeal to Tivoli
I couldn't tell you. Each persons definition of a small file and large amount of small files can be completely different. In one area a 100meg file can be considered small while another group may consider it huge. I have a volume with over 3 million files which average about 24k bytes in size. Even Arcserve would roll-over on that one. In my experience Arcserve will out perform on a full volume restore by at least 3 to 1 on tests I have done on a 40Gig backup then within hours a full restore. No chance of anything other than a single initial full backup. As time goes on transfer rates drop to maybe 100Megs/hour on as little as a directory which contains 20 some files. The 17Gig/hour is actual transfer of data in that hour. Peak transfer rates on 100+meg files exceeds 30Gigs/hour ( across the LAN) while smaller files of course slow it down. The 30Gig/hour seems to be the best it can transfer even on Gigabyte files ( Arcserve database ) on local filesystems. Sorry, I don't have a direct answer to your question. On Netware, TSM on restores seem to peek out at 10Gig/hour under the best of conditions. I'm still trying to understand exactly what is going on to better define the problem. The hardware I'm using for Arcserve .. Compaq Proliant 1850R PIII 500Mhz, 396Meg RAM , 10K RPM drive holds the Arcserve database. Tape Drives are DLT35/70 mounted in TL892 tape libraries. connected via differential SCSI. The server is dedicated to Arcserve. The server is connected to the LAN via Gigabit NIC, George Yang [EMAIL PROTECTED] on 09/21/2000 10:37:32 AM Please respond to "ADSM: Dist Stor Manager" [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc:(bcc: Blair Wickstrand/Poco) Fax to: Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Blair, We can restore 13GB in one hour for a fairly medium amount of medium size file by using TSM. But not for a large amount of small files. Can you do 17GB/hr or what ever for a small file systems with ARCserver? What is your wire--LAN or SCSI? George
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
I haven't seen in this thread (or I missed it) any comments about using BACKUPSETS. Has anyone tried that? Does anyone know if the backupset is constructed in a way that overcomes any of the problems with recreating zillions of small files? Or do we need an image backup capability like the AIX client has to get past that issue? Signed, Yes I know there's a problem but I'm as confused as everyone else -Original Message- From: Joshua S. Bassi [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 6:56 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Not today. But I have heard it is coming. Also MS is adding a native NTFS image backup to NT 6.0 (aka Whistler). That will definitely be nice... -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Greazel, Alex Sent: Wednesday, September 20, 2000 9:44 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Can TSM do an image backup on NT? (If it can)That will dramatically decrease the amount of time it takes to do backups/restores for file systems that have tons of small files.
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
Thanks Wanda. Actually I did try to generate a server backupset for a different smaller NT client with similar data characteristics. The generate ran for a couple days. I was thinking of giving it a try for this NT client. Jeff "Prather, Wanda" [EMAIL PROTECTED]@VM.MARIST.EDU on 09/22/2000 01:07:46 PM Please respond to "ADSM: Dist Stor Manager" [EMAIL PROTECTED] Sent by: "ADSM: Dist Stor Manager" [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc: Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support I haven't seen in this thread (or I missed it) any comments about using BACKUPSETS. Has anyone tried that? Does anyone know if the backupset is constructed in a way that overcomes any of the problems with recreating zillions of small files? Or do we need an image backup capability like the AIX client has to get past that issue? Signed, Yes I know there's a problem but I'm as confused as everyone else -Original Message- From: Joshua S. Bassi [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 6:56 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Not today. But I have heard it is coming. Also MS is adding a native NTFS image backup to NT 6.0 (aka Whistler). That will definitely be nice... -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Greazel, Alex Sent: Wednesday, September 20, 2000 9:44 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Can TSM do an image backup on NT? (If it can)That will dramatically decrease the amount of time it takes to do backups/restores for file systems that have tons of small files.
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
OK, another question and I'm going to answer this one myself in a couple of weeks when I participate in a trial of it... Let's talk about a big time TSM restore that worked well and was reasonably fast. Anyone with such an experience? I know in some testing I restored 18 GB in an hour using three streams. I also had a reasonable dataset: a good mix of large and small files. Test environment. Downhill, wind at the back. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Jeff Connor Sent: Friday, September 22, 2000 11:51 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support Thanks Wanda. Actually I did try to generate a server backupset for a different smaller NT client with similar data characteristics. The generate ran for a couple days. I was thinking of giving it a try for this NT client. Jeff "Prather, Wanda" [EMAIL PROTECTED]@VM.MARIST.EDU on 09/22/2000 01:07:46 PM Please respond to "ADSM: Dist Stor Manager" [EMAIL PROTECTED] Sent by: "ADSM: Dist Stor Manager" [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc: Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support I haven't seen in this thread (or I missed it) any comments about using BACKUPSETS. Has anyone tried that? Does anyone know if the backupset is constructed in a way that overcomes any of the problems with recreating zillions of small files? Or do we need an image backup capability like the AIX client has to get past that issue? Signed, Yes I know there's a problem but I'm as confused as everyone else -Original Message- From: Joshua S. Bassi [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 6:56 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Not today. But I have heard it is coming. Also MS is adding a native NTFS image backup to NT 6.0 (aka Whistler). That will definitely be nice... -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Greazel, Alex Sent: Wednesday, September 20, 2000 9:44 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Can TSM do an image backup on NT? (If it can)That will dramatically decrease the amount of time it takes to do backups/restores for file systems that have tons of small files.
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
Hee Hee. Out of the frying pan... So much for that idea! -Original Message- From: Jeff Connor [SMTP:[EMAIL PROTECTED]] Sent: Friday, September 22, 2000 1:51 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support Thanks Wanda. Actually I did try to generate a server backupset for a different smaller NT client with similar data characteristics. The generate ran for a couple days. I was thinking of giving it a try for this NT client. Jeff "Prather, Wanda" [EMAIL PROTECTED]@VM.MARIST.EDU on 09/22/2000 01:07:46 PM Please respond to "ADSM: Dist Stor Manager" [EMAIL PROTECTED] Sent by: "ADSM: Dist Stor Manager" [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc: Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support I haven't seen in this thread (or I missed it) any comments about using BACKUPSETS. Has anyone tried that? Does anyone know if the backupset is constructed in a way that overcomes any of the problems with recreating zillions of small files? Or do we need an image backup capability like the AIX client has to get past that issue? Signed, Yes I know there's a problem but I'm as confused as everyone else -Original Message- From: Joshua S. Bassi [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 6:56 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Not today. But I have heard it is coming. Also MS is adding a native NTFS image backup to NT 6.0 (aka Whistler). That will definitely be nice... -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Greazel, Alex Sent: Wednesday, September 20, 2000 9:44 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Can TSM do an image backup on NT? (If it can)That will dramatically decrease the amount of time it takes to do backups/restores for file systems that have tons of small files.
Re: Slow restore for large NT client outcome.. appeal to Tivoli
Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has some tricks or tweaks that can help in this area. Anyone from any universities out there? Reply Separator Subject:Slow restore for large NT client outcome.. appeal to Tivoli Author: Jeff Connor [EMAIL PROTECTED] Date: 09/20/2000 12:21 PM Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with small files. I've heard different reasons for slow performance with small files over the years like the amount of TSM database lookups, NT file system processing/inefficiencies, etc.I have suggested to our NT admins that we break that big D: partition into multiple smaller partitions so I can collocate by filespace and restore multiple drives concurrently. Frankly, they are not in
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
That may be because from what I have seen in TSM 3.7, directories are actually kept in the TSM database and not on tape. Careful, there. It depends upon the file system type and added baggage. Ordinary Unix file system directory information is like an empty file, and so its limited info can be kept in the database. But add an Access Control List (ACL) to an AIX directory, for example, and it has to be stored in a storage pool. From what I've seen, file systems such as NTFS, laden with additional info, always have to be stored in a storage pool. When in doubt, a Query Content should reveal what's really happening. Richard Sims, BU
Re: Slow restore for large NT client outcome.. appeal to Tivoli
When reading this I have to ask 600Mb how? 1. What kind of systems (ADSM/BE server, client)? 2. What kind of network? 3. What kind of tape devices? 4. How many files in the directory? 5. Is the ADSM server shared or dedicated? Its possible to create patholgical cases that can trash any backup system's backup and restore times. In our case, thanks to raid, we do a lot more single file restores than restores of entire filesystems. Jim -- From: Mike Glassman - Admin[SMTP:[EMAIL PROTECTED]] Reply To: ADSM: Dist Stor Manager Sent: Thursday, September 21, 2000 2:57 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has some tricks or tweaks that can help in this area. Anyone from any universities out there? Reply Separator Subject:Slow restore for large NT client outcome.. appeal to Tivoli Author: Jeff Connor [EMAIL PROTECTED] Date: 09/20/2000 12:21 PM Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product
Re: Slow restore for large NT client outcome.. appeal to Tivoli
AS/400 you say? It's probably the slowest *SM server out of the 9 supported. NT TSM server are typically much faster than their 400 counterparts. -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Mike Glassman - Admin Sent: Wednesday, September 20, 2000 11:58 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has some tricks or tweaks that can help in this area. Anyone from any universities out there? Reply Separator Subject:Slow restore for large NT client outcome.. appeal to Tivoli Author: Jeff Connor [EMAIL PROTECTED] Date: 09/20/2000 12:21 PM Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I h
Re: Slow restore for large NT client outcome.. appeal to Tivoli
Ok, I tried to resist the thread. Arcserve on Netware5 SP5 can backup and restore across the wire 17Gigs an hour. Nothing spectacular but 10 times faster than what I can get out of TSM. I'm still looking for why TSM seems so slow. Whats more, TSM has a problem in getting long filename support while Arcserve is OK. On the other hand the stability of Arcserve on Netware is poor, but then so is dsmc on Netware. We use the Arcserve push-agents to get transfer rates from remote server to the Host server up, I wish TSM had something similiar. The killer for the TSM is that I refuse to put the TSM agents on the production servers due to the frequent abends, instead I put them on a high-horsepower box that does nothing else but run the TSM incrementals. Normal day to day TSM is great, but in a disaster recovery,. It scares the hell out of me. Then I would be glad I maintained Arcserve. Blair Mike Glassman - Admin [EMAIL PROTECTED] on 09/21/2000 12:57:47 AM Please respond to "ADSM: Dist Stor Manager" [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc:(bcc: Blair Wickstrand/Poco) Fax to: Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Kelly, I don't know regarding Arcserve as you couldn't pay me to go near it, but BE I do know. Restore of a 600MB directory (talking small here) on ADSM to a Netware server takes up to (no exageration here) 6 hours. And this is after we made all sorts of changes (not me, our AS400 guy as that's where it sits, I just complain) to the system. Under BE, the same 600MB takes under 45 minutes. In both cases we are talking about a backup system sitting on another system and not the backed up one. Mike -Original Message- From: Kelly J. Lipp [SMTP:[EMAIL PROTECTED]] Sent: ã ñôèîáø 20 2000 23:38 To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Could someone with experience doing large restores with ArcServe or BackupExec provide some performance numbers? I've been in shops where the backups were taking a very long time. Longer than my TSM backup took. I never witnessed a restore but how can it be better. I want the facts. I'm tired of hearing about how much faster ArcServe and BackupExec are (in theory) compared to TSM in reality. I'm sick and tired of it and I won't take it anymore! This is what happens when you TSM 24 hours per day. Your brain. Your brain on TSM. Not a pretty picture. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Keith E. Pruitt Sent: Wednesday, September 20, 2000 12:03 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivo Jeff, we too have a problem with small files. At first I thought it was a Netware thing because the servers we have the greatest amount of files on reside on the Netware servers. But reading emails from several users I see that I may have a future problem on the NT side. We store Word and WordPerfect docs on two Netware 5 machines and each server holds about 1.8 Million files apiece. Needless to say these files are not that big. It took over 11 hours to back each of the servers up and they total around 30GB per server. We were forced to perform a Full backup because our director and other new admins don't understand and feel comfortable with the "incremental forever" logic. I would hate to see what a restore would look like. In contrast, we just backed up a directory on an NT server we are using for our Backoffice conversion and that dir totals 35GB. That took 2h20m. We also performed a large restore from one AIX machine to another one of about 25GB. Less than 2 hours to restore. We have tweaked our Netware and AIX ADSM server according to performance guides and other suggestions and still have issues with small files. We will be moving our documents from Netware to NT soon and our NT guys like to refer to ADSM as crap. They are used to Arcserve but our now raving about BackupExec. It is going to be extremely difficult to explain if our huge machine can't keep up with their backup server. I know that overall ADSM is a better and more stable product but what do you do when you have a mixture of servers with large databases(ADSM's favorite) and (the more common) servers with small files that Arcserve and others like? I'm hoping another ADSM/TSM user has some tricks or tweaks that can help in this area. Anyone from any universities out there? Reply Separator____ Subject: Slow restore for large NT client outcome.. appeal to Tivoli Author: Jeff Connor [EMAIL PROTECTED] Date: 09/20/2000 12:21 PM Our NT g
Re: Slow restore for large NT client outcome.. appeal to Tivoli
Blair, We can restore 13GB in one hour for a fairly medium amount of medium size file by using TSM. But not for a large amount of small files. Can you do 17GB/hr or what ever for a small file systems with ARCserver? What is your wire--LAN or SCSI? George
Re: Slow restore for large NT client outcome.. appeal to Tivoli
My experience here has been that Backups fly, Restores snore.
My best rate for a restore of files of mixed length, mainly
small, has been 1.5 GB /hr from tape. (We are using IBM
3590B drives, soon to become 3590E.)
In a test, a restore from the disk pool brought
me way up to 1.66 GB/ hour. Yee ha!
Another test I did was to FTP the same data
from my client to the server where ADSM lives and
bring it back again. That got me about 6 GB/hour,
both directions.
We are using ADSM on AIX, with NT clients.
Since the backup rate is acceptable, I feel I can
rule out network and hardware issues to a
large degree. (I suppose I could try tweaking
the read/write ratio on the RAID controller.)
We have found that ADSM is a decent solution
for 'ad-hoc' restores, (the everyday confusion users
suffer over the delete key) but large disaster
recovery type restores scare the heck outta us.
At one point when we thought we may have lost
a logical drive (don't ask!) we started a restore and
at the rates we were getting from a highly fragmented,
non-co-located library, we estimated that it was going
to take 3.4 weeks to restore 45 GB. Not the kind of
news mgmt likes to hear. We are now working on
turning co-location on. We hope that will bring it down
to a mere 30 hours. Yippee!
John Wiersma
Network Analyst
Rochester Gas Electric Corporation
[EMAIL PROTECTED]
(716) 724-8053
RRR !! !!
R !! !!
RRR RRR !!! !!!
RRR RRR !!! A N D !!!
R !!! !!!
!!! !!!
RRRRRR!!!!!! !!!
RRR RRR !! !
RRRRRR ! !
Re: Slow restore for large NT client outcome.. appeal to Tivoli
Jeff, I'd begin by looking at your Raid array definition. I've found that when you use more then 4 drives in a single array the performance goes in toilet. if you can test it try runninthe same restore to a raid 0 configuration. - Do You Yahoo!? Send instant messages get email alerts with Yahoo! Messenger.
Re: Slow restore for large NT client outcome.. appeal to Tivoli
With the new copper chips the AS/400s can hit and sustain 4000MIPS. Edward(Ed) J. Finnell, III Enterprise Systems/Proj. Mgr. url:www.ua.edu
Slow restore for large NT client outcome.. appeal to Tivoli Development/Support
I posted the memo below to this listserv last week when we were having trouble with the performance restoring a large NT drive. This memo is for the people who wanted to know how we made out in the end. I am also writing to bring to the attention of TSM development what I feel is a pretty big problem in the area of performance for clients with lots of small files. I am pursuing this issue with Tivoli through other channels but thought others on this listserv might have the same concern. For a summary of our TSM config see my first memo below. First lets get a couple things out of the way. I have been working with TSM/ADSM for approximately five years since version 1. I am a HUGE fan of the product and have fought very hard to get our company to standardize on TSM and leave Arcserve, Backup exec, Legato, and the like. I am pleased with improvements in TSM functionality over the years. The second thing is we run TSM on OS/390. Over time I've seen many posts on the listserv about users that have achieved better performance with UNIX based TSM servers. We are currently piloting TSM on AIX to test the performance. Now that we've established my loyalties, back to my concern about backup, and more importantly restore, performance for TSM clients with lots of small files. Most of our UNIX servers are database servers so my concerns about small files really pertain mostly to Windows NT server clients. Others may have issues with other platforms. The NT clients I have restore issues with are big file and print servers. The data partition is typically the D: drive and can be anywhere from 20GB to 160GB in size. The best restore time we can achieve for the file and print servers is somewhere between 1.5GB and 3.5GB per hour generally on the lower side. Now we could go through a lot of the common, is your network performing, is your database cache hit high enough, tcp window sizes, txn sizes, and the usual things but assume for a moment that we are optimally configured and done all "the right stuff". To make a performance comparison, we have a couple NT clients that contain a small number of file and they are large files. We restored 20GB of data on one of those servers recently in 1hr 45mins. The restore of the one directory on the D: partition for the client mentioned in my first memo below with an average file size of 64K ran for 6hrs 5mins and transferring 4.8GB. The whole drive took 45hrs. Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with small files. I've heard different reasons for slow performance with small files over the years like the amount of TSM database lookups, NT file system processing/inefficiencies, etc. When looking at future directions for SAN backups I can understand the argument that the SAN pipes will be faster and TCPIP overhead will be eliminated leading to faster restores/backups. But if the poor performance for small files has a lot to do with TSM database lookups/overhead then how will performance be different when the data travels over the SAN versus the LAN/WAN? The database processing about file information will be pretty much the same won't it? I have suggested to our NT admins that we break that big D: partition into multiple smaller partitions so I can collocate by filespace and restore multiple drives concurrently. Frankly, they are not interested in changing the way they configure their servers to accommodate the backup software. They feel they would not have to do this with Arcserve or other more common NT backup products. I've tried tests using share names for folders and performing backups/restores using the UNC name, collocating the data by filespace and running concurrent restores. My tests showed improved elapsed time but this scheme would be tough to maintain. In a full server restore scenario I'd need to create the folders and shares for the target restore which means we'd need to keep track of that info some place. I'd constantly have to monitor growth in all the folders to make sure I've carved up the drive in fairly equal parts to optimize for restore, etc. Not a good solution either. Does anyone else see the poor performance for restoring clients with lots of small files and feel that this is a problem Tivoli needs to address?
Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support
Jeff, One thing to show your NT Admins is just how much overhead NTFS has. The way I've done this before is to copy a drive, either locally or over the network. If you take one of the drives with a lot of small files, can copy it, the performance will drop as the copy goes on. The more files you stick on an NTFS partition, the higher the overhead becomes. The way the NTFS allocation tables work, the more files you put in, the more complex the tables become. Here's the test I did, in basic form: On a system with a large drive, net use (NT speak - "Map Network Drive") to an empty drive on an adjacent machine. Copy a 20GB directory (from the command line - no NT speak) to this drive. Measure performance. Delete data on target system. Copy entire drive to target. Measure performance. Laugh as NT Admins realize how bad the performance gets with the larger drive. (If you can't tell from the above, I'm not much of an NT fan) Nick Cassimatis [EMAIL PROTECTED] "I'm one cookie away from happy." - Snoopy (Charles Schulz)
Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support
Can TSM do an image backup on NT? (If it can)That will dramatically decrease the amount of time it takes to do backups/restores for file systems that have tons of small files.
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
Just a thought - Take a look at the new TSM Implementation Redbook. They set up separate storage pools for NT, one for directory info and one for files. According to the Redbook restoring the directory structure first allows for faster restore times. I happened to be in a TSM session last week, the presenter said there was no need to separate storage pools(confused me). Please let us know if you get it resolved Good Luck. -Original Message- From: Jeff Connor [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 11:22 AM To: [EMAIL PROTECTED] Subject: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support I posted the memo below to this listserv last week when we were having trouble with the performance restoring a large NT drive. This memo is for the people who wanted to know how we made out in the end. I am also writing to bring to the attention of TSM development what I feel is a pretty big problem in the area of performance for clients with lots of small files. I am pursuing this issue with Tivoli through other channels but thought others on this listserv might have the same concern. For a summary of our TSM config see my first memo below. First lets get a couple things out of the way. I have been working with TSM/ADSM for approximately five years since version 1. I am a HUGE fan of the product and have fought very hard to get our company to standardize on TSM and leave Arcserve, Backup exec, Legato, and the like. I am pleased with improvements in TSM functionality over the years. The second thing is we run TSM on OS/390. Over time I've seen many posts on the listserv about users that have achieved better performance with UNIX based TSM servers. We are currently piloting TSM on AIX to test the performance. Now that we've established my loyalties, back to my concern about backup, and more importantly restore, performance for TSM clients with lots of small files. Most of our UNIX servers are database servers so my concerns about small files really pertain mostly to Windows NT server clients. Others may have issues with other platforms. The NT clients I have restore issues with are big file and print servers. The data partition is typically the D: drive and can be anywhere from 20GB to 160GB in size. The best restore time we can achieve for the file and print servers is somewhere between 1.5GB and 3.5GB per hour generally on the lower side. Now we could go through a lot of the common, is your network performing, is your database cache hit high enough, tcp window sizes, txn sizes, and the usual things but assume for a moment that we are optimally configured and done all "the right stuff". To make a performance comparison, we have a couple NT clients that contain a small number of file and they are large files. We restored 20GB of data on one of those servers recently in 1hr 45mins. The restore of the one directory on the D: partition for the client mentioned in my first memo below with an average file size of 64K ran for 6hrs 5mins and transferring 4.8GB. The whole drive took 45hrs. Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with small files. I've heard different reasons for slow performance with small files over the years like the amount of TSM database lookups, NT file system processing/inefficiencies, etc. When looking at future directions for SAN backups I can understand the argument that the SAN pipes will be faster and TCPIP overhead will be eliminated leading to faster restores/backups. But if the poor performance for small files has a lot to do with TSM database lookups/overhead then how will performance be different when the data travels over the SAN versus the LAN/WAN? The database processing about file information will be pretty much the same won't it? I have suggested to our NT admins that we break that big D: partition into multiple smaller partitions so I can collocate by filespace and restore multiple drives concurrently. Frankly, they are not interested in changing the way they configure their servers to accommodate the backup software. They feel they would not have to do this with Arcserve or other m
Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support
Would other backup tools suffer the same problem then? If this is a filesystem problem I would expect so. To the original poster: how long does it take Barfserve to restore a 60 GB filesystem with 10M files? As you can tell, I'm no fan of Arcserve. Kelly J. Lipp Storage Solutions Specialists, Inc. PO Box 51313 Colorado Springs CO 80949-1313 (719) 531-5926 Fax: (719) 260-5991 Email: [EMAIL PROTECTED] www.storsol.com www.storserver.com -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Nicholas Cassimatis/Raleigh/IBM Sent: Wednesday, September 20, 2000 10:53 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support Jeff, One thing to show your NT Admins is just how much overhead NTFS has. The way I've done this before is to copy a drive, either locally or over the network. If you take one of the drives with a lot of small files, can copy it, the performance will drop as the copy goes on. The more files you stick on an NTFS partition, the higher the overhead becomes. The way the NTFS allocation tables work, the more files you put in, the more complex the tables become. Here's the test I did, in basic form: On a system with a large drive, net use (NT speak - "Map Network Drive") to an empty drive on an adjacent machine. Copy a 20GB directory (from the command line - no NT speak) to this drive. Measure performance. Delete data on target system. Copy entire drive to target. Measure performance. Laugh as NT Admins realize how bad the performance gets with the larger drive. (If you can't tell from the above, I'm not much of an NT fan) Nick Cassimatis [EMAIL PROTECTED] "I'm one cookie away from happy." - Snoopy (Charles Schulz)
Re: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support
How so?? George
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
Jeff, I have the same concerns. My company is in the process of server consolidation. I have concerns that when the time comes to consolidate these servers into cluster servers I (TSM) will not be able to restore 1.2 TB of data to a cluster server in a timely manner. I am quickly loosing the battle in defending TSM. My NT admin's are moving toward hardware mirroring. If improvements to small file restores does not come soon I may loose the battle. I typically I achieve a 1-3gb per hour restore rate on file servers, Arturo Lopez -Original Message- From: Jeff Connor [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 11:22 AM To: [EMAIL PROTECTED] Subject:Slow restore for large NT client outcome.. appeal to Tivoli Development/Support I posted the memo below to this listserv last week when we were having trouble with the performance restoring a large NT drive. This memo is for the people who wanted to know how we made out in the end. I am also writing to bring to the attention of TSM development what I feel is a pretty big problem in the area of performance for clients with lots of small files. I am pursuing this issue with Tivoli through other channels but thought others on this listserv might have the same concern. For a summary of our TSM config see my first memo below. First lets get a couple things out of the way. I have been working with TSM/ADSM for approximately five years since version 1. I am a HUGE fan of the product and have fought very hard to get our company to standardize on TSM and leave Arcserve, Backup exec, Legato, and the like. I am pleased with improvements in TSM functionality over the years. The second thing is we run TSM on OS/390. Over time I've seen many posts on the listserv about users that have achieved better performance with UNIX based TSM servers. We are currently piloting TSM on AIX to test the performance. Now that we've established my loyalties, back to my concern about backup, and more importantly restore, performance for TSM clients with lots of small files. Most of our UNIX servers are database servers so my concerns about small files really pertain mostly to Windows NT server clients. Others may have issues with other platforms. The NT clients I have restore issues with are big file and print servers. The data partition is typically the D: drive and can be anywhere from 20GB to 160GB in size. The best restore time we can achieve for the file and print servers is somewhere between 1.5GB and 3.5GB per hour generally on the lower side. Now we could go through a lot of the common, is your network performing, is your database cache hit high enough, tcp window sizes, txn sizes, and the usual things but assume for a moment that we are optimally configured and done all "the right stuff". To make a performance comparison, we have a couple NT clients that contain a small number of file and they are large files. We restored 20GB of data on one of those servers recently in 1hr 45mins. The restore of the one directory on the D: partition for the client mentioned in my first memo below with an average file size of 64K ran for 6hrs 5mins and transferring 4.8GB. The whole drive took 45hrs. Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with small files. I've heard different reasons for slow performance with small files over the years like the amount of TSM database lookups, NT file system processing/inefficiencies, etc. When looking at future directions for SAN backups I can understand the argument that the SAN pipes will be faster and TCPIP overhead will be eliminated leading to faster restores/backups. But if the poor performance for small files has a lot to do with TSM datab
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
That may be because from what I have seen in TSM 3.7, directories are actually kept in the TSM database and not on tape. (That is just what I have seen in the field, but haven't seen documented anywhere). -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Farris, Raeana Sent: Wednesday, September 20, 2000 11:19 AM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support Just a thought - Take a look at the new TSM Implementation Redbook. They set up separate storage pools for NT, one for directory info and one for files. According to the Redbook restoring the directory structure first allows for faster restore times. I happened to be in a TSM session last week, the presenter said there was no need to separate storage pools(confused me). Please let us know if you get it resolved Good Luck. -Original Message- From: Jeff Connor [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 11:22 AM To: [EMAIL PROTECTED] Subject: Slow restore for large NT client outcome.. appeal to Tivoli Development/Support I posted the memo below to this listserv last week when we were having trouble with the performance restoring a large NT drive. This memo is for the people who wanted to know how we made out in the end. I am also writing to bring to the attention of TSM development what I feel is a pretty big problem in the area of performance for clients with lots of small files. I am pursuing this issue with Tivoli through other channels but thought others on this listserv might have the same concern. For a summary of our TSM config see my first memo below. First lets get a couple things out of the way. I have been working with TSM/ADSM for approximately five years since version 1. I am a HUGE fan of the product and have fought very hard to get our company to standardize on TSM and leave Arcserve, Backup exec, Legato, and the like. I am pleased with improvements in TSM functionality over the years. The second thing is we run TSM on OS/390. Over time I've seen many posts on the listserv about users that have achieved better performance with UNIX based TSM servers. We are currently piloting TSM on AIX to test the performance. Now that we've established my loyalties, back to my concern about backup, and more importantly restore, performance for TSM clients with lots of small files. Most of our UNIX servers are database servers so my concerns about small files really pertain mostly to Windows NT server clients. Others may have issues with other platforms. The NT clients I have restore issues with are big file and print servers. The data partition is typically the D: drive and can be anywhere from 20GB to 160GB in size. The best restore time we can achieve for the file and print servers is somewhere between 1.5GB and 3.5GB per hour generally on the lower side. Now we could go through a lot of the common, is your network performing, is your database cache hit high enough, tcp window sizes, txn sizes, and the usual things but assume for a moment that we are optimally configured and done all "the right stuff". To make a performance comparison, we have a couple NT clients that contain a small number of file and they are large files. We restored 20GB of data on one of those servers recently in 1hr 45mins. The restore of the one directory on the D: partition for the client mentioned in my first memo below with an average file size of 64K ran for 6hrs 5mins and transferring 4.8GB. The whole drive took 45hrs. Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with small files. I've heard different reasons for slow performance with small files over the years like the amount of TSM database lookups, NT file system processing/inefficiencies, etc. When looking at future directions for SAN backups I can understand the argument that the SAN pipes will be faster and TCPIP overhead will be eliminated leading to faster restores/backups. But if the poor performance for small files has a lot to do with TSM datab
Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support
My NT admin's are moving toward hardware mirroring. That should be the first feature added for any highly available configuration. -- Joshua S. Bassi Senior Technical Consultant Symatrix Technology, Inc. [EMAIL PROTECTED] -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Arturo Lopez Sent: Wednesday, September 20, 2000 3:07 PM To: [EMAIL PROTECTED] Subject: Re: Slow restore for large NT client outcome.. appeal to Tivoli D evelopment/Support Jeff, I have the same concerns. My company is in the process of server consolidation. I have concerns that when the time comes to consolidate these servers into cluster servers I (TSM) will not be able to restore 1.2 TB of data to a cluster server in a timely manner. I am quickly loosing the battle in defending TSM. My NT admin's are moving toward hardware mirroring. If improvements to small file restores does not come soon I may loose the battle. I typically I achieve a 1-3gb per hour restore rate on file servers, Arturo Lopez -Original Message- From: Jeff Connor [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, September 20, 2000 11:22 AM To: [EMAIL PROTECTED] Subject:Slow restore for large NT client outcome.. appeal to Tivoli Development/Support I posted the memo below to this listserv last week when we were having trouble with the performance restoring a large NT drive. This memo is for the people who wanted to know how we made out in the end. I am also writing to bring to the attention of TSM development what I feel is a pretty big problem in the area of performance for clients with lots of small files. I am pursuing this issue with Tivoli through other channels but thought others on this listserv might have the same concern. For a summary of our TSM config see my first memo below. First lets get a couple things out of the way. I have been working with TSM/ADSM for approximately five years since version 1. I am a HUGE fan of the product and have fought very hard to get our company to standardize on TSM and leave Arcserve, Backup exec, Legato, and the like. I am pleased with improvements in TSM functionality over the years. The second thing is we run TSM on OS/390. Over time I've seen many posts on the listserv about users that have achieved better performance with UNIX based TSM servers. We are currently piloting TSM on AIX to test the performance. Now that we've established my loyalties, back to my concern about backup, and more importantly restore, performance for TSM clients with lots of small files. Most of our UNIX servers are database servers so my concerns about small files really pertain mostly to Windows NT server clients. Others may have issues with other platforms. The NT clients I have restore issues with are big file and print servers. The data partition is typically the D: drive and can be anywhere from 20GB to 160GB in size. The best restore time we can achieve for the file and print servers is somewhere between 1.5GB and 3.5GB per hour generally on the lower side. Now we could go through a lot of the common, is your network performing, is your database cache hit high enough, tcp window sizes, txn sizes, and the usual things but assume for a moment that we are optimally configured and done all "the right stuff". To make a performance comparison, we have a couple NT clients that contain a small number of file and they are large files. We restored 20GB of data on one of those servers recently in 1hr 45mins. The restore of the one directory on the D: partition for the client mentioned in my first memo below with an average file size of 64K ran for 6hrs 5mins and transferring 4.8GB. The whole drive took 45hrs. Our NT group was a hard sell for replacing Arcserve with TSM. Since the switch, I have taken quite a beating about TSM restore performance. Our NT admins take the position, "we'll try TSM but if the performance doesn't improve we are going with a tried and true solution like Compaq Enterprise Backup. TSM seems to us like a UNIX product trying to make it in the NT space. It is not typically selected by companies for NT backup and recovery". Not a word for word quote but generally sums up their position. The Compaq solution would use Arcserve from what I've been told. I know Tivoli/IBM have tried to address the small files issue with things like small file aggregation but I haven't noticed much improvement from version to version for big restores of servers with smal
