Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you > 200MB/sec on LTO5, yes?
-----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <steve_sch...@bcbst.com>wrote: > Wanda, > > I have fought with this problem myself, and here is what I concluded > (at least in our environment, YMMV): > > 1. Running single-stream backups (one db at a time) you will never see > the performance you expect, due to the Windows O/S tcpip stack. I > haven't had a chance to stress-test Win2012-R2 yet, but at least > through 2008-R2, there seems to be a single-thread constraint that > prevents any backup from getting much more than about 20% of the bandwidth. > > 2. The only way to get around this is to do as Del suggests and > parallelize your backups. If you can get 4-6 concurrent jobs running, > you can push the network card pretty close to 100%. The catch, as > Dell also pointed out, is that you can't run concurrent backups on > databases that live on the same disk (since the vss snap is at the disk > level). > > Bottom line is that you would need to divide up your Exchange > databases so they are on different disks (or at least, create as many > disks as you want to have concurrent backups, then create separate jobs to > backup each group). > > Good luck, > > Steve Schaub > System Engineer II, Backup/Recovery > Blue Cross Blue Shield of Tennessee > > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > Of Prather, Wanda > Sent: Sunday, February 09, 2014 1:08 PM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] Exchange 2010 backup performance > > Del, you are a national treasure! > You are very kind to take time to respond. > > My backups are already very well balanced, I have 2 servers, the DBA's > have the DBs split between them so well that they backup almost the > same amount of data, and finish within 30 minutes of each other. > (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, > with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 > disk so should be spiffy speed there.). > > I tried setting resourceutilization 10 once before, was an impressive > failure. The backup appeared to be looping doing VSS snaps (or rather > failing to); I think it was doing as you mentioned in 2 below, trying > to snap the same LUN multiple times. > > Will go through the references you included, then open a performance > PMR if no improvement. > > Thank you so much! > > W > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > Of Del Hoobler > Sent: Friday, February 07, 2014 6:48 PM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] Exchange 2010 backup performance > > Hi Wanda, > > I have a few ideas for you... > > -------------------------- > > Are you running in a DAG environment? If so, you could do some load > balancing between DAG Servers: > > Most of this in the Exchange book under "Managing Exchange Database > Availability Group members by using a single policy": > > > > http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. > ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html > > The key to "load balance" when setting up the scheduled backup script > is to have a separate invocation of each database. For example: > > TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE > > Then, run this command from each of the Exchange servers at or about > the same time. > > -------------------------- > > Here are a few more things to look at: > > To help with some performance issues, some customers have split their > backups into multiple "threads" or "processes" in two ways: > > 1. Increase the value of the RESOURCEUTILIZATION parameter in the > DSM.OPT file for the DSMAGENT. Trying setting this to "10". > Important: This needs to the DSM.OPT file for the DSMAGENT > not the DP/Exchange options file. > > 2. Split the backups into multiple parallel instances of the > TDPEXCC backup execution. > i.e. the create separate invocations of DP/Exchange that back > up a different set of databases. For example: > TDPEXCC BACKUP db1,db2,db3,db4 FULL > TDPEXCC BACKUP db5,db6,db7,db8 FULL > TDPEXCC BACKUP db9,db10,db11,db12 FULL > Put these in separate command files and stagger the > launching of them by 10 minutes or so. > The key here is that you need to make sure that you don't > have any LUNs that appears in more than one invocation. > In other words, you don't want to snapshot the > same LUN in separate invocations. > > Note: The integrity check is a Microsoft tool. IBM has no control over > the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL > program to perform the integrity check. It's a very I/O intensive > program that must examine every page of the database file (.EDB) and all log > files. > > -------------------------- > > If none of these help, you should open a PMR to get the performance > team to look at your environment. > > > > Thank you, > > Del > > ---------------------------------------------------- > > "ADSM: Dist Stor Manager" <ADSM-L@vm.marist.edu> wrote on 02/07/2014 > 06:04:01 PM: > > > From: "Prather, Wanda" <wanda.prat...@icfi.com> > > To: ADSM-L@vm.marist.edu, > > Date: 02/07/2014 06:06 PM > > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor > > Manager" <ADSM-L@vm.marist.edu> > > > > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in > > the baclient dsm.opt? > > Or is there anything else I can tweak to improve TSM throughput of a > > 2010 full backup? > > Got a 10G network, but Exchange full backup performance not impressive. > > > > Thanks for any ideas - links to relevant doc also appreciated! > > > > Wanda > > > > > > **Please note new office phone: > > Wanda Prather | Senior Technical Specialist | > > wanda.prat...@icfi.com > | > > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF > > International > > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md > > |443-718-4900 (o) > > > ----------------------------------------------------- > Please see the following link for the BlueCross BlueShield of > Tennessee E-mail disclaimer: > http://www.bcbst.com/email_disclaimer.shtm >