Re: Large disk concerns/alternatives ?
Good day everyone, Thank you for your input. After careful review of my config I found a typo in my opt file and my data was being sent to an undersized STG and in turn was writing directly to tape. After correcting the include statement the throughput is where it should be for my primary site to primary site data transfers. ( 1gb between the two sites). However, that does not address my remote sites. My remote sites are multiple TBs of data. Does anyone have any suggestions of how to most efficiently backup/recover a large remote site with limited bandwidth ? One alternate method we have used in the past, for the continental US, is restoring the data locally to the site the TSM server resides then copying the data to disks and shipping them overnight to the location. A little cumbersome and shipping fully populated disks is a concern but it is fairly efficient. Any suggestions would be appreciated. Thanks, Duane From: Ochs, Duane Sent: Monday, August 24, 2009 11:54 AM To: 'ADSM: Dist Stor Manager' Subject: Large disk concerns/alternatives ? Good day everyone, I'm looking for some workable suggestions for larger lun backups. I'm talking 1.5tb and larger lun sizes. I have a number of remote sites that have large quantities of hi-res image files and they need an offsite backup solution. Usually we are notified before the disks go into production and the backups grow as the disk is populated. Normally not a big deal. Now we are in the middle of a large conversion for many remote sites which will require a full backup for all of the sites. And each has in excess of 6 tb of images. In addition we are also migrating a number of rogue sites to TSM which also have in excess of 4tb locally. I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM diskpool on Sata disks. It has been running for 3 days. Not only is it not acceptable in a real DR scenario (Currently we restore to the same site as the TSM server then move data back in order of importance),but, it will also take weeks before I can safely say we are back to being fully backed up. Other than local backups for each site... has anybody come across a similar scenario and care to offer some advice. Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 library. No data is backed up locally. All data is sent over the network to a remote TSM server. Thanks, Duane
Re: Large disk concerns/alternatives ?
On Aug 25, 2009, at 10:02 AM, Gerald Michalak wrote: My money is on the SATA disk pool. I ran into something similar with many spindles of SATA disk and was expecting it to fly. nope. Indeed. Vendors often cite SATA and SAS technology throughput rates, leading the customer to believe that is what they will get. But when you delve into the box to find out what vendor disk model is actually inside, and check the disk specs for the data rates for that model, you'll often find that the achievable rate is substantially less, factoring in RAID configuration. Richard Sims
Re: Large disk concerns/alternatives ?
My money is on the SATA disk pool. I ran into something similar with many spindles of SATA disk and was expecting it to fly. nope. We found out that not only was the disk I/O not as expected but that device the disks were in only had 2 I/O ports that were supposed to be in failover mode. But we had both running at over 80% utilization !!! Replaced that external box with some internal drives and controllers and then the system starting moving data. Gerald Michalak TSM - Certified V5 Administrator "ADSM: Dist Stor Manager" wrote on 08/24/2009 09:21:52 PM: > Sumthin' wrong with those numbers. > > If you have a GigE pipe, you should be able to send 70-80 MB/sec. thats > 400+ MB/min, 240+ GB per hour, 4+ TB in 24 hours. > > So, either > - your clients are having performance issues with their own disks > - maybe you have compression or something else slowing down the client:? > - maybe your client NIC isn't really set to GigE > - You have a switch somewhere that is not pumping GigE... > - Your NIC on the TSM SERVER is maxed out > > Or something like that. I'd try some testing with FTP, to see where the > bottleneck is, before deciding this isn't feasible. > > W > > > On Mon, Aug 24, 2009 at 12:53 PM, Ochs, Duane wrote: > > > Good day everyone, > > I'm looking for some workable suggestions for larger lun backups. > > I'm talking 1.5tb and larger lun sizes. I have a number of remote sites > > that have large quantities of hi-res image files and they need an offsite > > backup solution. Usually we are notified before the disks go into production > > and the backups grow as the disk is populated. > > Normally not a big deal. Now we are in the middle of a large conversion for > > many remote sites which will require a full backup for all of the sites. And > > each has in excess of 6 tb of images. In addition we are also migrating a > > number of rogue sites to TSM which also have in excess of 4tb locally. > > > > I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM > > diskpool on Sata disks. It has been running for 3 days. Not only is it not > > acceptable in a real DR scenario (Currently we restore to the same site as > > the TSM server then move data back in order of importance),but, it will > > also take weeks before I can safely say we are back to being fully backed > > up. > > > > Other than local backups for each site... has anybody come across a similar > > scenario and care to offer some advice. > > > > Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 > > library. No data is backed up locally. All data is sent over the network to > > a remote TSM server. > > > > Thanks, > > Duane > >
Re: Large disk concerns/alternatives ?
Duane, You might consider re-architecting to take advantage of deduplication and replication opportunities. Put your remote site storage on a NetApp or Data Domain NAS device - both of which perform deduplication. Replicate the deduplicated data to your home site where you can also make a TSM backup to tape for a third copy if necessary. I have a Data Domain device which has 5.7TB of data which deduplicates to 300GB - the 300GB is the amount of data that is replicated. My NetApp devices don't quite achieve the same deduplication ratio because the data is significantly different than what I store on the Data Domain (NetApp=live data, DD=VM backup data) but, I keep about 40TB synchronized between two sites on my NetApps over a 1G WAN. You will need to discuss with the various vendors how you data might deduplicate and what impact to application performance you might incur. These are my experiences - there are other disk vendors which may provide your environment with a better solution. This solution may require a significant departure in how you are currently doing business but it should be something you consider. Cheers Neil Strand Storage Engineer - Legg Mason Baltimore, MD. (410) 580-7491 Whatever you can do or believe you can, begin it. Boldness has genius, power and magic. -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Ochs, Duane Sent: Monday, August 24, 2009 12:54 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Large disk concerns/alternatives ? Good day everyone, I'm looking for some workable suggestions for larger lun backups. I'm talking 1.5tb and larger lun sizes. I have a number of remote sites that have large quantities of hi-res image files and they need an offsite backup solution. Usually we are notified before the disks go into production and the backups grow as the disk is populated. Normally not a big deal. Now we are in the middle of a large conversion for many remote sites which will require a full backup for all of the sites. And each has in excess of 6 tb of images. In addition we are also migrating a number of rogue sites to TSM which also have in excess of 4tb locally. I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM diskpool on Sata disks. It has been running for 3 days. Not only is it not acceptable in a real DR scenario (Currently we restore to the same site as the TSM server then move data back in order of importance),but, it will also take weeks before I can safely say we are back to being fully backed up. Other than local backups for each site... has anybody come across a similar scenario and care to offer some advice. Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 library. No data is backed up locally. All data is sent over the network to a remote TSM server. Thanks, Duane IMPORTANT: E-mail sent through the Internet is not secure. Legg Mason therefore recommends that you do not send any confidential or sensitive information to us via electronic mail, including social security numbers, account numbers, or personal identification numbers. Delivery, and or timely delivery of Internet mail is not guaranteed. Legg Mason therefore recommends that you do not send time sensitive or action-oriented messages to us via electronic mail. This message is intended for the addressee only and may contain privileged or confidential information. Unless you are the intended recipient, you may not use, copy or disclose to anyone any information contained in this message. If you have received this message in error, please notify the author by replying to this message and then kindly delete the message. Thank you.
Re: Large disk concerns/alternatives ?
Always, always perform benchmark analysis of any new hardware, software, or network units before committing their use to an application such as TSM. This is an essential element in the acceptance phase of new acquisitions, to assure that they meet advertised performance levels. If the new stuff fails to measure up, it should not be accepted for use, and if its problems cannot be resolved, it should be returned to the vendor (per terms & conditioned negotiated into the contract). Too many sites put in new equipment, start using it in a production manner, and then have no idea why it's not working well, and cannot then perform isolated tests because it is then integrated into the environment. Richard Sims at Boston University On Aug 24, 2009, at 10:21 PM, Wanda Prather wrote: Sumthin' wrong with those numbers. If you have a GigE pipe, you should be able to send 70-80 MB/sec. thats 400+ MB/min, 240+ GB per hour, 4+ TB in 24 hours. So, either - your clients are having performance issues with their own disks - maybe you have compression or something else slowing down the client:? - maybe your client NIC isn't really set to GigE - You have a switch somewhere that is not pumping GigE... - Your NIC on the TSM SERVER is maxed out Or something like that. I'd try some testing with FTP, to see where the bottleneck is, before deciding this isn't feasible. W On Mon, Aug 24, 2009 at 12:53 PM, Ochs, Duane wrote: Good day everyone, I'm looking for some workable suggestions for larger lun backups. I'm talking 1.5tb and larger lun sizes. I have a number of remote sites that have large quantities of hi-res image files and they need an offsite backup solution. Usually we are notified before the disks go into production and the backups grow as the disk is populated. Normally not a big deal. Now we are in the middle of a large conversion for many remote sites which will require a full backup for all of the sites. And each has in excess of 6 tb of images. In addition we are also migrating a number of rogue sites to TSM which also have in excess of 4tb locally. I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM diskpool on Sata disks. It has been running for 3 days. Not only is it not acceptable in a real DR scenario (Currently we restore to the same site as the TSM server then move data back in order of importance),but, it will also take weeks before I can safely say we are back to being fully backed up. Other than local backups for each site... has anybody come across a similar scenario and care to offer some advice. Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 library. No data is backed up locally. All data is sent over the network to a remote TSM server. Thanks, Duane
Re: Large disk concerns/alternatives ?
Sumthin' wrong with those numbers. If you have a GigE pipe, you should be able to send 70-80 MB/sec. thats 400+ MB/min, 240+ GB per hour, 4+ TB in 24 hours. So, either - your clients are having performance issues with their own disks - maybe you have compression or something else slowing down the client:? - maybe your client NIC isn't really set to GigE - You have a switch somewhere that is not pumping GigE... - Your NIC on the TSM SERVER is maxed out Or something like that. I'd try some testing with FTP, to see where the bottleneck is, before deciding this isn't feasible. W On Mon, Aug 24, 2009 at 12:53 PM, Ochs, Duane wrote: > Good day everyone, > I'm looking for some workable suggestions for larger lun backups. > I'm talking 1.5tb and larger lun sizes. I have a number of remote sites > that have large quantities of hi-res image files and they need an offsite > backup solution. Usually we are notified before the disks go into production > and the backups grow as the disk is populated. > Normally not a big deal. Now we are in the middle of a large conversion for > many remote sites which will require a full backup for all of the sites. And > each has in excess of 6 tb of images. In addition we are also migrating a > number of rogue sites to TSM which also have in excess of 4tb locally. > > I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM > diskpool on Sata disks. It has been running for 3 days. Not only is it not > acceptable in a real DR scenario (Currently we restore to the same site as > the TSM server then move data back in order of importance),but, it will > also take weeks before I can safely say we are back to being fully backed > up. > > Other than local backups for each site... has anybody come across a similar > scenario and care to offer some advice. > > Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 > library. No data is backed up locally. All data is sent over the network to > a remote TSM server. > > Thanks, > Duane >
Re: Large disk concerns/alternatives ?
>> On Mon, 24 Aug 2009 11:53:52 -0500, "Ochs, Duane" said: > I'm looking for some workable suggestions for larger lun backups. > I'm talking 1.5tb and larger lun sizes. I have a number of remote > sites that have large quantities of hi-res image files and they need > an offsite backup solution. Usually we are notified before the disks > go into production and the backups grow as the disk is populated. > Normally not a big deal. Now we are in the middle of a large > conversion for many remote sites which will require a full backup > for all of the sites. And each has in excess of 6 tb of images. In > addition we are also migrating a number of rogue sites to TSM which > also have in excess of 4tb locally. The critical point here is not 'Large Lun' but 'initial incremental'. Instead of scanning 4TB of stuff, you want to _send_ 4TB; nearly serially. Putting disk on the write side of that is just silly, unless you have an XIV. (drool) So you should be thinking in terms of direct write to tape. A single 3592 will happily eat 80 MB/s of uncompressible data for as long as you can keep the buffers filled. That adds up to 3.mumble hours per TB. If you send two streams, you'll be able to eke out the rest of the ~112MB/s that a gig link will get you, at the cost of slowing down each individual stream (i.e. they each go from approaching 80-mumble to approaching 56-mumble MB) > Some specifics: I have 3 TSM servers (AIX) with either a L700 or a > T950 library. No data is backed up locally. All data is sent over > the network to a remote TSM server. Drive technology? - Allen S. Rout
Re: Large disk concerns/alternatives ?
Duane, You may find worthy to take a look at LAN-Free backups configurations. This would not only alleviate the load on your bandwidth but also decrease the backup duration. That may or may not work for you but wouldn't hurt to look into it. Good luck. BERTAUT TCHUISE TSM/NetApp Storage Administrator Legg Mason Technology Services *410-580-7032 btchu...@leggmason.com -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Ochs, Duane Sent: Monday, August 24, 2009 12:54 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Large disk concerns/alternatives ? Good day everyone, I'm looking for some workable suggestions for larger lun backups. I'm talking 1.5tb and larger lun sizes. I have a number of remote sites that have large quantities of hi-res image files and they need an offsite backup solution. Usually we are notified before the disks go into production and the backups grow as the disk is populated. Normally not a big deal. Now we are in the middle of a large conversion for many remote sites which will require a full backup for all of the sites. And each has in excess of 6 tb of images. In addition we are also migrating a number of rogue sites to TSM which also have in excess of 4tb locally. I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM diskpool on Sata disks. It has been running for 3 days. Not only is it not acceptable in a real DR scenario (Currently we restore to the same site as the TSM server then move data back in order of importance),but, it will also take weeks before I can safely say we are back to being fully backed up. Other than local backups for each site... has anybody come across a similar scenario and care to offer some advice. Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 library. No data is backed up locally. All data is sent over the network to a remote TSM server. Thanks, Duane IMPORTANT: E-mail sent through the Internet is not secure. Legg Mason therefore recommends that you do not send any confidential or sensitive information to us via electronic mail, including social security numbers, account numbers, or personal identification numbers. Delivery, and or timely delivery of Internet mail is not guaranteed. Legg Mason therefore recommends that you do not send time sensitive or action-oriented messages to us via electronic mail. This message is intended for the addressee only and may contain privileged or confidential information. Unless you are the intended recipient, you may not use, copy or disclose to anyone any information contained in this message. If you have received this message in error, please notify the author by replying to this message and then kindly delete the message. Thank you.
Large disk concerns/alternatives ?
Good day everyone, I'm looking for some workable suggestions for larger lun backups. I'm talking 1.5tb and larger lun sizes. I have a number of remote sites that have large quantities of hi-res image files and they need an offsite backup solution. Usually we are notified before the disks go into production and the backups grow as the disk is populated. Normally not a big deal. Now we are in the middle of a large conversion for many remote sites which will require a full backup for all of the sites. And each has in excess of 6 tb of images. In addition we are also migrating a number of rogue sites to TSM which also have in excess of 4tb locally. I tried a test and ran one disk as a full over a 1gb pipe to 2tb of TSM diskpool on Sata disks. It has been running for 3 days. Not only is it not acceptable in a real DR scenario (Currently we restore to the same site as the TSM server then move data back in order of importance),but, it will also take weeks before I can safely say we are back to being fully backed up. Other than local backups for each site... has anybody come across a similar scenario and care to offer some advice. Some specifics: I have 3 TSM servers (AIX) with either a L700 or a T950 library. No data is backed up locally. All data is sent over the network to a remote TSM server. Thanks, Duane
