Re: Exchange 2010 backup performance
El Domingo 9 de febrero de 2014 19:12, Prather, Wanda wanda.prat...@icfi.com escribió: Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your environment. Thank you, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 02/07/2014 06:04:01 PM: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 02/07/2014 06:06 PM Subject: Exchange 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in the baclient dsm.opt? Or is there anything else I can tweak to improve TSM throughput of a 2010 full backup? Got a 10G network, but Exchange full backup performance not impressive. Thanks for any ideas - links to relevant doc also appreciated! Wanda **Please note new office phone: Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.comhttp://www.icfi.com | 410-868-4872 (m) ICF International | 7125 Thomas Edison Dr., Suite 100, Columbia, Md |443-718-4900 (o)
Re: Exchange 2010 backup performance
Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your
Re: Exchange 2010 backup performance
In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL
Re: Exchange 2010 backup performance
Thank you for sharing that. Will save me banging my head against the wall for nothing! :) -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Schaub, Steve Sent: Monday, February 10, 2014 7:57 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of
Re: Exchange 2010 backup performance
Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important:
Re: Exchange 2010 backup performance
Have you examined the network topology altogether. Our Exchange environment has network hops all over the place so our networking guys recommended adding a VLAN tag to both the TSM server and the Exchange DB servers to do backups. In that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the way of the backup stream. Haven't had to tweak performance on Exchange backups at all yet. Of course, now that I say that... SF On 2/10/14 11:16 AM, Prather, Wanda wanda.prat...@icfi.com wrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL
Re: Exchange 2010 backup performance
Hi Sergio! Yep, we had had some other issues with network disconnects, so we just 2 weeks ago added NICS to the Exchange servers and put them on the same segment as the TSM server. No routers, firewalls, hops. Think my next step is a performance trace. W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Sergio O. Fuentes Sent: Monday, February 10, 2014 12:41 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Have you examined the network topology altogether. Our Exchange environment has network hops all over the place so our networking guys recommended adding a VLAN tag to both the TSM server and the Exchange DB servers to do backups. In that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the way of the backup stream. Haven't had to tweak performance on Exchange backups at all yet. Of course, now that I say that... SF On 2/10/14 11:16 AM, Prather, Wanda wanda.prat...@icfi.com wrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy:
Re: Exchange 2010 backup performance
Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The
Re: Exchange 2010 backup performance
Woot, that's cool to know! Thanks Dave. I will run a test tonight and look for a spike. Is RSS something that is enabled in the NIC or in Windows? I know the network guys already had to do some tweaking of the 10G TOE card. Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Dave Canan Sent: Monday, February 10, 2014 1:07 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU
Re: Exchange 2010 backup performance
Both Wanda http://technet.microsoft.com/en-us/library/gg162712%28v=ws.10%29.aspx -Rick Adamson 904.783.5264 -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Monday, February 10, 2014 1:18 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Woot, that's cool to know! Thanks Dave. I will run a test tonight and look for a spike. Is RSS something that is enabled in the NIC or in Windows? I know the network guys already had to do some tweaking of the 10G TOE card. Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Dave Canan Sent: Monday, February 10, 2014 1:07 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap
Restoring Linux 6.1 server DB to new machine
I am testing a restore and conversion/upgrade of my last 6.1 server, before it becomes a pumpkin in April! I am having issues following the wonderful, detail, draft manual, since it doesn't give specifics as to who I should be when I perform various tasks. For instance, after installing and configuring 6.1, the restore process says to do a dsmserv removedb TSMDB1. But it fails to tell me (and the command failed) that I should be logged in as the DB2 instance user (tsminst1), not root. Yes, all filesystems have the proper ownership. So, I figured I would need to stay as the DB2 instance (vs root), for the restore process. While most of the restore process seemed to go OK, I had some errors that seem to indicate I should have been root. In the beginning of the dsmserv restore db, I got this error: *rm: cannot remove `/tsmarchlog': Permission denied* *mkdir: cannot create directory `/tsmarchlog': File exists* Of course this confused me since I needed to define these for the install process and the book didn't say I should delete it, so why would the restore try to recreate them? But since the empty directory was already there with the proper ownership, I ignored it. Then after the restore ran for 2-hours (200GB DB and low-power test machine), it ends with this: *ANR4917I Point-in-time database restore with snapshot complete, restore date 01/16/2014 09:18:53 AM.* *ANR0222E Error opening for write disk definition file dsmserv.dbid.* *Error 2104 updating database ID file.* *ANR2988W Attempt to add the last backup db volume used entry back in to the volume history was unsuccessful.* The ever helpful user manual for ANR0222E says Attempt to determine the cause of the write error and correct it. - Thanks, guys.. This sounds more serious? Before I waste another 2-hours, should I start all over again, doing the restore via root or is this something I can correct, and how? Thoughts? Suggestions? -- *Zoltan Forray* TSM Software Hardware Administrator Virginia Commonwealth University UCC/Office of Technology Services zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://infosecurity.vcu.edu/phishing.html
