Have you examined the network topology altogether. Our Exchange environment has network hops all over the place so our networking guys recommended adding a VLAN tag to both the TSM server and the Exchange DB servers to do backups. In that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the way of the backup stream. Haven't had to tweak performance on Exchange backups at all yet. Of course, now that I say that...
SF On 2/10/14 11:16 AM, "Prather, Wanda" <wanda.prat...@icfi.com> wrote: >Thank you - forgot to mention this is a Windows TSM server. >I am curious that the drive is the bottleneck - a big file of zeros >should compress, and give you > 200MB/sec on LTO5, yes? > > >-----Original Message----- >From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of >Hans Christian Riksheim >Sent: Monday, February 10, 2014 9:04 AM >To: ADSM-L@VM.MARIST.EDU >Subject: Re: [ADSM-L] Exchange 2010 backup performance > >In my experience there is nothing wrong with the TCP stack in Windows. >Especially Windows2008R2 performs very well. For a single stream from a >2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km >away over 10Gig directly to LTO5 has a speed of around 200MB/ at our >setup. >Bottleneck being the drive. > >After too much experimenting I have found the critical factor to be to >set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the >tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS >can be left alone as its default is quite OK. YMMV of course. > >Regards, > >Hans Chr. > > >On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve ><steve_sch...@bcbst.com>wrote: > >> Wanda, >> >> I have fought with this problem myself, and here is what I concluded >> (at least in our environment, YMMV): >> >> 1. Running single-stream backups (one db at a time) you will never see >> the performance you expect, due to the Windows O/S tcpip stack. I >> haven't had a chance to stress-test Win2012-R2 yet, but at least >> through 2008-R2, there seems to be a single-thread constraint that >> prevents any backup from getting much more than about 20% of the >>bandwidth. >> >> 2. The only way to get around this is to do as Del suggests and >> parallelize your backups. If you can get 4-6 concurrent jobs running, >> you can push the network card pretty close to 100%. The catch, as >> Dell also pointed out, is that you can't run concurrent backups on >> databases that live on the same disk (since the vss snap is at the disk >>level). >> >> Bottom line is that you would need to divide up your Exchange >> databases so they are on different disks (or at least, create as many >> disks as you want to have concurrent backups, then create separate jobs >>to backup each group). >> >> Good luck, >> >> Steve Schaub >> System Engineer II, Backup/Recovery >> Blue Cross Blue Shield of Tennessee >> >> >> -----Original Message----- >> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >> Of Prather, Wanda >> Sent: Sunday, February 09, 2014 1:08 PM >> To: ADSM-L@VM.MARIST.EDU >> Subject: Re: [ADSM-L] Exchange 2010 backup performance >> >> Del, you are a national treasure! >> You are very kind to take time to respond. >> >> My backups are already very well balanced, I have 2 servers, the DBA's >> have the DBs split between them so well that they backup almost the >> same amount of data, and finish within 30 minutes of each other. >> (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, >> with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 >> disk so should be spiffy speed there.). >> >> I tried setting resourceutilization 10 once before, was an impressive >> failure. The backup appeared to be looping doing VSS snaps (or rather >> failing to); I think it was doing as you mentioned in 2 below, trying >> to snap the same LUN multiple times. >> >> Will go through the references you included, then open a performance >> PMR if no improvement. >> >> Thank you so much! >> >> W >> >> -----Original Message----- >> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >> Of Del Hoobler >> Sent: Friday, February 07, 2014 6:48 PM >> To: ADSM-L@VM.MARIST.EDU >> Subject: Re: [ADSM-L] Exchange 2010 backup performance >> >> Hi Wanda, >> >> I have a few ideas for you... >> >> -------------------------- >> >> Are you running in a DAG environment? If so, you could do some load >> balancing between DAG Servers: >> >> Most of this in the Exchange book under "Managing Exchange Database >> Availability Group members by using a single policy": >> >> >> >> http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. >> ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html >> >> The key to "load balance" when setting up the scheduled backup script >> is to have a separate invocation of each database. For example: >> >> TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE >> TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE >> TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE >> TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE >> TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE >> >> Then, run this command from each of the Exchange servers at or about >> the same time. >> >> -------------------------- >> >> Here are a few more things to look at: >> >> To help with some performance issues, some customers have split their >> backups into multiple "threads" or "processes" in two ways: >> >> 1. Increase the value of the RESOURCEUTILIZATION parameter in the >> DSM.OPT file for the DSMAGENT. Trying setting this to "10". >> Important: This needs to the DSM.OPT file for the DSMAGENT >> not the DP/Exchange options file. >> >> 2. Split the backups into multiple parallel instances of the >> TDPEXCC backup execution. >> i.e. the create separate invocations of DP/Exchange that back >> up a different set of databases. For example: >> TDPEXCC BACKUP db1,db2,db3,db4 FULL >> TDPEXCC BACKUP db5,db6,db7,db8 FULL >> TDPEXCC BACKUP db9,db10,db11,db12 FULL >> Put these in separate command files and stagger the >> launching of them by 10 minutes or so. >> The key here is that you need to make sure that you don't >> have any LUNs that appears in more than one invocation. >> In other words, you don't want to snapshot the >> same LUN in separate invocations. >> >> Note: The integrity check is a Microsoft tool. IBM has no control over >> the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL >> program to perform the integrity check. It's a very I/O intensive >> program that must examine every page of the database file (.EDB) and >>all log files. >> >> -------------------------- >> >> If none of these help, you should open a PMR to get the performance >> team to look at your environment. >> >> >> >> Thank you, >> >> Del >> >> ---------------------------------------------------- >> >> "ADSM: Dist Stor Manager" <ADSM-L@vm.marist.edu> wrote on 02/07/2014 >> 06:04:01 PM: >> >> > From: "Prather, Wanda" <wanda.prat...@icfi.com> >> > To: ADSM-L@vm.marist.edu, >> > Date: 02/07/2014 06:06 PM >> > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor >> > Manager" <ADSM-L@vm.marist.edu> >> > >> > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in >> > the baclient dsm.opt? >> > Or is there anything else I can tweak to improve TSM throughput of a >> > 2010 full backup? >> > Got a 10G network, but Exchange full backup performance not >>impressive. >> > >> > Thanks for any ideas - links to relevant doc also appreciated! >> > >> > Wanda >> > >> > >> > **Please note new office phone: >> > Wanda Prather | Senior Technical Specialist | >> > wanda.prat...@icfi.com >> | >> > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF >> > International >> > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md >> > |443-718-4900 (o) >> > >> ----------------------------------------------------- >> Please see the following link for the BlueCross BlueShield of >> Tennessee E-mail disclaimer: >> http://www.bcbst.com/email_disclaimer.shtm >>
