Re: Spectrum Protect Plus Backup Performance
Thanks Steve, I've been digging into it a bit more with the help of IBM support and I forgot a very important setting. Currently I have a vsnap combined with vadp proxy, but I neglected to set the softcap for the vadp. Consequently, backups were running on all available cores and crowding out the vsnap server. Setting the softcap based on the blueprints greatly improved overall throughput. Single VM performance was up and overall throughput was in excess of 400 MBps. So that goes to show, RTFM. __ Matthew McGeary Service Delivery Manager / Solutions Architect Data Center & Network Management, Nutrien IT T: (306) 933-8921 www.nutrien.com -Original Message- From: ADSM: Dist Stor Manager On Behalf Of Schaub, Steve Sent: Wednesday, July 8, 2020 7:45 AM To: ADSM-L@VM.MARIST.EDU Subject: [EXT] Re: [ADSM-L] Spectrum Protect Plus Backup Performance WARNING: This email originated from outside of the organization. Exercise caution when viewing attachments, clicking links, or responding to requests. Matthew, Not sure if this is your issue or not, but verify that you are using nbd transport. Hotadd performs better on initial full backups, but the overhead involved in setup/teardown makes it a poor choice once you start doing incrementals consistently. Steve Schaub Senior Platform Engineer II, Backup & Recovery BlueCross BlueShield of Tennessee -Original Message- From: ADSM: Dist Stor Manager On Behalf Of Matthew McGeary Sent: Wednesday, July 8, 2020 12:32 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Spectrum Protect Plus Backup Performance Hey folks, Starting a deployment of spectrum protect plus using virtual vsnap/vadp systems backed by v5030 RDM storage. The vsnap system has been sized based on the recommendation of the blueprints for a 100TB repo and all of our VMWare infrastructure is hosted on Cisco UCS with multiple 10G interfaces per blade. Doing backup testing today and single backup performance is atrocious, averaging just 22 MB/s. I've got a large TDP for VMWare deployment and I am familiar with setting parallelism with that product and we see much better individual VM backup performance with that product. Is there anything similar I can tweak for the Plus deployment? I can't find anything in the docs and the current performance makes me quite nervous as we push this product forward into production in the next month. Any assistance or experience with a similar all-virtual deployment would be helpful. Thanks __ Matthew McGeary Service Delivery Manager / Solutions Architect Data Center & Network Management, Nutrien IT T: (306) 933-8921 http://www.nutrien.com * For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important -- Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: https://urldefense.com/v3/__https://www.bcbst.com/about/our-company/corporate-governance/privacy-security/email-policy.page__;!!DMYgIb-w!mA4Jnh1-SlCAcvIulHYlnHky8a8OIhx81yfKaxzK7_gkwMsm854QwvJFlt3qb7OEBHGXteE0$ <https://urldefense.com/v3/__https://www.bcbst.com/about/our-company/corporate-governance/privacy-security/email-policy.page__;!!DMYgIb-w!mA4Jnh1-SlCAcvIulHYlnHky8a8OIhx81yfKaxzK7_gkwMsm854QwvJFlt3qb7OEBHGXteE0$ > -- This email was sent by "Schaub, Steve" securely using Transport Layer Security (TLS). For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important
Re: Spectrum Protect Plus Backup Performance
Matthew, Not sure if this is your issue or not, but verify that you are using nbd transport. Hotadd performs better on initial full backups, but the overhead involved in setup/teardown makes it a poor choice once you start doing incrementals consistently. Steve Schaub Senior Platform Engineer II, Backup & Recovery BlueCross BlueShield of Tennessee -Original Message- From: ADSM: Dist Stor Manager On Behalf Of Matthew McGeary Sent: Wednesday, July 8, 2020 12:32 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Spectrum Protect Plus Backup Performance Hey folks, Starting a deployment of spectrum protect plus using virtual vsnap/vadp systems backed by v5030 RDM storage. The vsnap system has been sized based on the recommendation of the blueprints for a 100TB repo and all of our VMWare infrastructure is hosted on Cisco UCS with multiple 10G interfaces per blade. Doing backup testing today and single backup performance is atrocious, averaging just 22 MB/s. I've got a large TDP for VMWare deployment and I am familiar with setting parallelism with that product and we see much better individual VM backup performance with that product. Is there anything similar I can tweak for the Plus deployment? I can't find anything in the docs and the current performance makes me quite nervous as we push this product forward into production in the next month. Any assistance or experience with a similar all-virtual deployment would be helpful. Thanks __ Matthew McGeary Service Delivery Manager / Solutions Architect Data Center & Network Management, Nutrien IT T: (306) 933-8921 www.nutrien.com * For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important -- Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: https://www.bcbst.com/about/our-company/corporate-governance/privacy-security/email-policy.page <https://www.bcbst.com/about/our-company/corporate-governance/privacy-security/email-policy.page> -- This email was sent by "Schaub, Steve" securely using Transport Layer Security (TLS).
Spectrum Protect Plus Backup Performance
Hey folks, Starting a deployment of spectrum protect plus using virtual vsnap/vadp systems backed by v5030 RDM storage. The vsnap system has been sized based on the recommendation of the blueprints for a 100TB repo and all of our VMWare infrastructure is hosted on Cisco UCS with multiple 10G interfaces per blade. Doing backup testing today and single backup performance is atrocious, averaging just 22 MB/s. I've got a large TDP for VMWare deployment and I am familiar with setting parallelism with that product and we see much better individual VM backup performance with that product. Is there anything similar I can tweak for the Plus deployment? I can't find anything in the docs and the current performance makes me quite nervous as we push this product forward into production in the next month. Any assistance or experience with a similar all-virtual deployment would be helpful. Thanks __ Matthew McGeary Service Delivery Manager / Solutions Architect Data Center & Network Management, Nutrien IT T: (306) 933-8921 www.nutrien.com * For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important
Re: Exchange 2010 backup performance
Wanda, did your test show anything? Let me know if you have any questions. Dave Canan TSM Performance ddca...@us.ibm.com -Original Message- From: Prather, Wanda wanda.prat...@icfi.com Sent: 2/10/2014 10:19 AM To: ADSM-L@VM.MARIST.EDU ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Woot, that's cool to know! Thanks Dave. I will run a test tonight and look for a spike. Is RSS something that is enabled in the NIC or in Windows? I know the network guys already had to do some tweaking of the 10G TOE card. Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Dave Canan Sent: Monday, February 10, 2014 1:07 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple
Re: Exchange 2010 backup performance
Hans, What tcp-sizes are you using in AIX and on the nic? When you ran that test on the file of zeros, you had client compression turned off, right? I would love to get 200MB single-thread throughput from my clients. Thanks, -steve -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file
Re: Exchange 2010 backup performance
I'd like to second the comments by Hans -- Windows servers can perform well -- assuming the infrastructure is there to support it. Our TSM server is running 6.2 on a 64-bit Windows 2003 Server. One 1Gb ethernet nic. Our datamover/proxy is running 6.3 client on a Windows 2008 R2 (64-bit) Server. One 1Gb ethernet nic. We are regularly able to stream image backups at 500Mbps (that's megabits-per-second, just to be clear), reading from 4Gb fibre-channel and dumping across a dedicated LAN connection to the TSM server. Sometimes sending for 15-30 seconds at 700-750Mbps, topping out at around 815Mbps. I can run two proxies simultaneously streaming, and the TSM server will be receiving data from multiple image backups at a sustained rate of ~700-750Mbps. Incremental, file-level backups are another story... searching 6 Million files in 4TB and backing up 50GB can take ~2 hours. I would love to see what I can do with 10Gb ethernet, but we're not there yet... Mike On Wed, Feb 12, 2014 at 2:13 PM, Schaub, Steve steve_sch...@bcbst.comwrote: -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. Best regards, Mike RMD IT, x7942
Re: Exchange 2010 backup performance
Hi, I have set send/recv-space at 16MB globally with no and use_isno=0 to override the NICs. Most important is to set TCPWINODWSIZE=0 on the client and TSM server and then just let the OS handle it. I had to experiment with these settings for weeks as we have clients 500km away with high latency(9 ms ping) but with very good throughput(10 Gb fibre.) As always YMMV, network tuning is a mess and I may have just been lucky. Yes, client compression was off. Regards, Hans Chr. On Wed, Feb 12, 2014 at 8:13 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Hans, What tcp-sizes are you using in AIX and on the nic? When you ran that test on the file of zeros, you had client compression turned off, right? I would love to get 200MB single-thread throughput from my clients. Thanks, -steve -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL
Re: Exchange 2010 backup performance
El Domingo 9 de febrero de 2014 19:12, Prather, Wanda wanda.prat...@icfi.com escribió: Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your environment. Thank you, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 02/07/2014 06:04:01 PM: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 02/07/2014 06:06 PM Subject: Exchange 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in the baclient dsm.opt? Or is there anything else I can tweak to improve TSM throughput of a 2010 full backup? Got a 10G network, but Exchange full backup performance not impressive. Thanks for any ideas - links to relevant doc also appreciated! Wanda **Please note new office phone: Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.comhttp://www.icfi.com | 410-868-4872 (m) ICF International | 7125 Thomas Edison Dr., Suite 100, Columbia, Md |443-718-4900 (o)
Re: Exchange 2010 backup performance
Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your
Re: Exchange 2010 backup performance
In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL
Re: Exchange 2010 backup performance
Thank you for sharing that. Will save me banging my head against the wall for nothing! :) -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Schaub, Steve Sent: Monday, February 10, 2014 7:57 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed
Re: Exchange 2010 backup performance
Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important
Re: Exchange 2010 backup performance
Have you examined the network topology altogether. Our Exchange environment has network hops all over the place so our networking guys recommended adding a VLAN tag to both the TSM server and the Exchange DB servers to do backups. In that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the way of the backup stream. Haven't had to tweak performance on Exchange backups at all yet. Of course, now that I say that... SF On 2/10/14 11:16 AM, Prather, Wanda wanda.prat...@icfi.com wrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL
Re: Exchange 2010 backup performance
Hi Sergio! Yep, we had had some other issues with network disconnects, so we just 2 weeks ago added NICS to the Exchange servers and put them on the same segment as the TSM server. No routers, firewalls, hops. Think my next step is a performance trace. W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Sergio O. Fuentes Sent: Monday, February 10, 2014 12:41 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Have you examined the network topology altogether. Our Exchange environment has network hops all over the place so our networking guys recommended adding a VLAN tag to both the TSM server and the Exchange DB servers to do backups. In that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the way of the backup stream. Haven't had to tweak performance on Exchange backups at all yet. Of course, now that I say that... SF On 2/10/14 11:16 AM, Prather, Wanda wanda.prat...@icfi.com wrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.comwrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter
Re: Exchange 2010 backup performance
Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom. ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
Re: Exchange 2010 backup performance
Woot, that's cool to know! Thanks Dave. I will run a test tonight and look for a spike. Is RSS something that is enabled in the NIC or in Windows? I know the network guys already had to do some tweaking of the 10G TOE card. Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Dave Canan Sent: Monday, February 10, 2014 1:07 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU
Re: Exchange 2010 backup performance
Both Wanda http://technet.microsoft.com/en-us/library/gg162712%28v=ws.10%29.aspx -Rick Adamson 904.783.5264 -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Monday, February 10, 2014 1:18 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Woot, that's cool to know! Thanks Dave. I will run a test tonight and look for a spike. Is RSS something that is enabled in the NIC or in Windows? I know the network guys already had to do some tweaking of the 10G TOE card. Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Dave Canan Sent: Monday, February 10, 2014 1:07 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Wanda, one other thing I would like to point out. With the TSM Server being on Windows, one other thing to consider is RSS (Receive Side Scaling). All the processing of IP packets happens on 1 core of the server, and this can be a potential bottleneck. One way to determine this is to watch the TSM server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you enable RSS to spread the packet processing out amongst more cores, but you can specify how many cores and which core numbers to start with. If a perfmon run is done against the Windows Server, it is easy to see if one particular core is doing more worth than the others, and this can be a symptom that RSS may provide a benefit. Dave Canan TSM Performance ddca...@us.ibm.com On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda wanda.prat...@icfi.comwrote: Thank you - forgot to mention this is a Windows TSM server. I am curious that the drive is the bottleneck - a big file of zeros should compress, and give you 200MB/sec on LTO5, yes? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Hans Christian Riksheim Sent: Monday, February 10, 2014 9:04 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance In my experience there is nothing wrong with the TCP stack in Windows. Especially Windows2008R2 performs very well. For a single stream from a 2008R2 client (dsm sel big file of zeroes) to an AIX TSM-server 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup. Bottleneck being the drive. After too much experimenting I have found the critical factor to be to set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as its default is quite OK. YMMV of course. Regards, Hans Chr. On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve steve_sch...@bcbst.com wrote: Wanda, I have fought with this problem myself, and here is what I concluded (at least in our environment, YMMV): 1. Running single-stream backups (one db at a time) you will never see the performance you expect, due to the Windows O/S tcpip stack. I haven't had a chance to stress-test Win2012-R2 yet, but at least through 2008-R2, there seems to be a single-thread constraint that prevents any backup from getting much more than about 20% of the bandwidth. 2. The only way to get around this is to do as Del suggests and parallelize your backups. If you can get 4-6 concurrent jobs running, you can push the network card pretty close to 100%. The catch, as Dell also pointed out, is that you can't run concurrent backups on databases that live on the same disk (since the vss snap is at the disk level). Bottom line is that you would need to divide up your Exchange databases so they are on different disks (or at least, create as many disks as you want to have concurrent backups, then create separate jobs to backup each group). Good luck, Steve Schaub System Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Sunday, February 09, 2014 1:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap
Re: Exchange 2010 backup performance
Del, you are a national treasure! You are very kind to take time to respond. My backups are already very well balanced, I have 2 servers, the DBA's have the DBs split between them so well that they backup almost the same amount of data, and finish within 30 minutes of each other. (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000 disk so should be spiffy speed there.). I tried setting resourceutilization 10 once before, was an impressive failure. The backup appeared to be looping doing VSS snaps (or rather failing to); I think it was doing as you mentioned in 2 below, trying to snap the same LUN multiple times. Will go through the references you included, then open a performance PMR if no improvement. Thank you so much! W -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Friday, February 07, 2014 6:48 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange 2010 backup performance Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your environment. Thank you, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 02/07/2014 06:04:01 PM: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 02/07/2014 06:06 PM Subject: Exchange 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in the baclient dsm.opt? Or is there anything else I can tweak to improve TSM throughput of a 2010 full backup? Got a 10G network, but Exchange full backup performance not impressive. Thanks for any ideas - links to relevant doc also appreciated! Wanda **Please note new office phone: Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.comhttp://www.icfi.com | 410-868-4872 (m) ICF International | 7125 Thomas Edison Dr., Suite 100, Columbia, Md |443-718-4900 (o)
Exchange 2010 backup performance
Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in the baclient dsm.opt? Or is there anything else I can tweak to improve TSM throughput of a 2010 full backup? Got a 10G network, but Exchange full backup performance not impressive. Thanks for any ideas - links to relevant doc also appreciated! Wanda **Please note new office phone: Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.comhttp://www.icfi.com | 410-868-4872 (m) ICF International | 7125 Thomas Edison Dr., Suite 100, Columbia, Md |443-718-4900 (o)
Re: Exchange 2010 backup performance
Hi Wanda, I have a few ideas for you... -- Are you running in a DAG environment? If so, you could do some load balancing between DAG Servers: Most of this in the Exchange book under Managing Exchange Database Availability Group members by using a single policy: http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html The key to load balance when setting up the scheduled backup script is to have a separate invocation of each database. For example: TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE Then, run this command from each of the Exchange servers at or about the same time. -- Here are a few more things to look at: To help with some performance issues, some customers have split their backups into multiple threads or processes in two ways: 1. Increase the value of the RESOURCEUTILIZATION parameter in the DSM.OPT file for the DSMAGENT. Trying setting this to 10. Important: This needs to the DSM.OPT file for the DSMAGENT not the DP/Exchange options file. 2. Split the backups into multiple parallel instances of the TDPEXCC backup execution. i.e. the create separate invocations of DP/Exchange that back up a different set of databases. For example: TDPEXCC BACKUP db1,db2,db3,db4 FULL TDPEXCC BACKUP db5,db6,db7,db8 FULL TDPEXCC BACKUP db9,db10,db11,db12 FULL Put these in separate command files and stagger the launching of them by 10 minutes or so. The key here is that you need to make sure that you don't have any LUNs that appears in more than one invocation. In other words, you don't want to snapshot the same LUN in separate invocations. Note: The integrity check is a Microsoft tool. IBM has no control over the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL program to perform the integrity check. It's a very I/O intensive program that must examine every page of the database file (.EDB) and all log files. -- If none of these help, you should open a PMR to get the performance team to look at your environment. Thank you, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 02/07/2014 06:04:01 PM: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 02/07/2014 06:06 PM Subject: Exchange 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in the baclient dsm.opt? Or is there anything else I can tweak to improve TSM throughput of a 2010 full backup? Got a 10G network, but Exchange full backup performance not impressive. Thanks for any ideas - links to relevant doc also appreciated! Wanda **Please note new office phone: Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.comhttp://www.icfi.com | 410-868-4872 (m) ICF International | 7125 Thomas Edison Dr., Suite 100, Columbia, Md |443-718-4900 (o)
Re: TDP SQL backup performance
On 08/26/2013 10:09 AM, Sven Seefeld wrote: Hi, Can you pos tyour dsm.opt and tdpsql.cfg? There is a lot of tuning that can boost your perfomance. dsm.opt just contains the necessary node and server options, nothing else, tdpsql.cfg mainly lists the default values: Hi, This is a setting I use on a high end plattform with lots of memory. I strongly recommendthat you disable tsm compression (if enabled), and inserta line in tdpsql.cfg 'sqlcompression yes' Play around with numbers. To start with, set stripes = 2, and have a go. Also, make sure your node has a maxnummp=stripes + 2 if you send data to filespool or tapes. LASTPRUNEDate 06/11/2013 12:59:40 SQLAUTHentication INTegrated MOUNTWaitfordata Yes BACKUPMethod LEGACY DIFFESTimate 20 BUFFers 8 BUFFERSIze 8192 STRIPes 8 SQLBUFFers 32 SQLBUFFERSIze 4096 LOGFile C:\var\log\tsm\sql\c02\tdpsql.log LOGPrune 30 LANGuage ENU SQLSERVer xx FROMSQLserver x -- Mvh/Rgds Håkon Phillip Tønder-Keul
Re: TDP SQL backup performance
Hi, Can you pos tyour dsm.opt and tdpsql.cfg? There is a lot of tuning that can boost your perfomance. dsm.opt just contains the necessary node and server options, nothing else, tdpsql.cfg mainly lists the default values: LOCALDSMAgentnode * BACKUPMethod Legacy BackupDestination TSM BUFFers 3 BUFFERSIze 1024 SQLBUFFers 0 SQLBUFFERSIze 1024 STRIPes 1 SQLSERVer SQLAUTHentication SQLuserid MOUNTWaitfordata Yes DIFFESTimate 20 LOGPrune 60 LANGuage ENU Regards, Sven
Re: TDP SQL backup performance
Hi, I have a question about the target device, can an LTO-5 write at 20MB/sec? Perhaps an intermediate landing zone that can write that slow might help. the 1,5 TByte mainly splits into two files (~900 + ~600 GByte, don't ask why) and I don't have enough space in the diskbackuppool for that specific job. But, if I compare the speed with smaller dbs jobs (daily diff ~100 GByte) which fits into the diskpool, they aren't much faster (~25-30 MByte/s). I know that I should reach around 40 MByte/s for the LTO-5 tapes, but I'm pretty lost right now. Will post the dsm.opt and tdpsql.cfg on monday, when the dbas are on duty. Have a nice weekend, Sven
TDP SQL backup performance
Hi, we're experiencing slow performance while backing up a 1,5 TByte MS SQL 2008R2 database with the TDP SQL Agent 6.3. As for DR-scenarios, we're doing legacy backups only. The actual speed is usually around 20 MByte/s, backups go to LTO-5 tape directly. The backup stream has to pass through 3 different firewalls, so we're not quite sure where the bottle neck is located, MS SQL API, TDP Agent or IP-firewall (SAN-Storage and TSM-Server should outperform everything else). Maybe someone can share his performance values with me. Any hints are welcome. Regards, Sven
Re: TDP SQL backup performance
On 08/23/2013 04:07 PM, Sven Seefeld wrote: Hi, we're experiencing slow performance while backing up a 1,5 TByte MS SQL 2008R2 database with the TDP SQL Agent 6.3. As for DR-scenarios, we're doing legacy backups only. The actual speed is usually around 20 MByte/s, backups go to LTO-5 tape directly. The backup stream has to pass through 3 different firewalls, so we're not quite sure where the bottle neck is located, MS SQL API, TDP Agent or IP-firewall (SAN-Storage and TSM-Server should outperform everything else). Maybe someone can share his performance values with me. Any hints are welcome. Regards, Sven Hi, Can you pos tyour dsm.opt and tdpsql.cfg? There is a lot of tuning that can boost your perfomance. -- Mvh/Rgds Håkon Phillip Tønder-Keul Senior Consultant
Re: TDP SQL backup performance
I have a question about the target device, can an LTO-5 write at 20MB/sec? Perhaps an intermediate landing zone that can write that slow might help. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Håkon Phillip Tønder-Keul Sent: Friday, August 23, 2013 12:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] TDP SQL backup performance On 08/23/2013 04:07 PM, Sven Seefeld wrote: Hi, we're experiencing slow performance while backing up a 1,5 TByte MS SQL 2008R2 database with the TDP SQL Agent 6.3. As for DR-scenarios, we're doing legacy backups only. The actual speed is usually around 20 MByte/s, backups go to LTO-5 tape directly. The backup stream has to pass through 3 different firewalls, so we're not quite sure where the bottle neck is located, MS SQL API, TDP Agent or IP-firewall (SAN-Storage and TSM-Server should outperform everything else). Maybe someone can share his performance values with me. Any hints are welcome. Regards, Sven Hi, Can you pos tyour dsm.opt and tdpsql.cfg? There is a lot of tuning that can boost your perfomance. -- Mvh/Rgds Håkon Phillip Tønder-Keul Senior Consultant
Re: TDP SQL backup performance
Everywhere I look shows a lowend of 40-50MB/s as the minimum speed of an LTO5, depending on the vendor. I'm sure compression will also mess with that number. This is probably just classic tape shoe-shining. Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, August 23, 2013 3:23 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] TDP SQL backup performance I have a question about the target device, can an LTO-5 write at 20MB/sec? Perhaps an intermediate landing zone that can write that slow might help. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Håkon Phillip Tønder-Keul Sent: Friday, August 23, 2013 12:08 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] TDP SQL backup performance On 08/23/2013 04:07 PM, Sven Seefeld wrote: Hi, we're experiencing slow performance while backing up a 1,5 TByte MS SQL 2008R2 database with the TDP SQL Agent 6.3. As for DR-scenarios, we're doing legacy backups only. The actual speed is usually around 20 MByte/s, backups go to LTO-5 tape directly. The backup stream has to pass through 3 different firewalls, so we're not quite sure where the bottle neck is located, MS SQL API, TDP Agent or IP-firewall (SAN-Storage and TSM-Server should outperform everything else). Maybe someone can share his performance values with me. Any hints are welcome. Regards, Sven Hi, Can you pos tyour dsm.opt and tdpsql.cfg? There is a lot of tuning that can boost your perfomance. -- Mvh/Rgds Håkon Phillip Tønder-Keul Senior Consultant This message and any attachments (the message) is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. Please note that certain functions and services for BNP Paribas may be performed by BNP Paribas RCC, Inc.
Re: Exchange Legacy 2007 to VSS 2010 backup performance
We recently moved from Exchange 2007 to 2010 on our environment. Just for a reference, we have about 3.3 TB of exchange data. Our TSM server is v5.5.5.0 running on AIX 6.1. The exchange servers have v6.1.3.0 of the TDP client, v6.2.2.0 of the BA client, Windows 2008 R2 SP1 and Exchange 2010 SP1. We have found that the transfer of the data runs about the same once it starts moving data to TSM, but the VSS parts (which are no longer optional) caused our backups to take almost twice as long. We weren't able to find anything in the TDP documentation to tell us why backups were now taking twice as long, however by looking at the IO on the SAN and Exchange logs, we believe we were able to determine what was going on. Feel free to chime in if you think our assessment is incorrect. By default, the TDP causes the exchange software do its own an integrity check of the database every time it does a backup, (either full, incremental or differential). We found that essentially doubles the time the backup takes, because it seems to read/check what it wants to send for the integrity check and then read/send the data to TDP/TSM on another pass. So you are essentially reading all the bits twice for every backup. It seems like a rather inefficient way to run it, but perhaps that's the way it has to be done. There's a flag you can put in so that the integrity check is not called (/SKIPINTEGRITYCHECK), but there is an inherent risk in skipping it. Then again, it didn't do this check for the Legacy backups and we never had a corruption problem with the backups, but YMMV. Currently, a weekly full and daily differentials meet our SLA, and we are toying with the idea of leaving the integrity check on the fulls (and have it take twice as long as it used to) and turning it off for the differentials. Your choice may be different depending on your risk assessment. Test it out for yourself and let us know if your experiences are any different from ours. Ben -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Ian Smith Sent: Wednesday, August 24, 2011 10:12 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance Our Mail/GroupWare service is migrating from Exchange 2007 to 2010 in the next few months. Currently we employ streaming (Legacy) backups across a private 2Gbit bonded private link direct to LTO5 tape and get around 50MByte/s for fulls and around 16-20Mbyte/s for incrementals. In all we have about 20TB of mailstore. I've searched the threads and the user docs, developerworks etc etc but can't find any real figures on how legacy versus vss backups compare. I suspect they will be slower, because of the additional steps in the whole VSS backup protocol but I'm happy to be surprised. If anybody has any real world figures that address the questions below, I'd be truly grateful. 1. Legacy versus VSS-assisted (MS VSS provider) backups of Exchange 2007 to TSM server. 2. VSS-assisted backups of Exchange 2007 versus 2010. This is a big migration for all of us and I'd like to have an idea of what we should expect in testing. Thanks Ian Smith Oxford University/ England. The BCI Email Firewall made the following annotations - *Confidentiality Notice: This E-Mail is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you have received this communication in error, please do not distribute, and delete the original message. Thank you for your compliance. You may contact us at: Blue Cross of Idaho 3000 E. Pine Ave. Meridian, Idaho 83642 1.208.345.4550 -
Re: Exchange Legacy 2007 to VSS 2010 backup performance
For the most part, your description is accurate... but I have a few comments: - Actually... an integrity check was performed for legacy backups, but the Exchange Server did it while it was reading the data, so the penalty was not as bad. If the Exchange server found an integrity problem while running the check, it would fail the backup. - The integrity check for VSS backups is done after the snapshot, and it must read all the pages of the .EDB and .LOG files using an Exchange interface to do it - Microsoft has relaxed their requirement for certain Exchange 2010 environments. They say: Database mobility features, including Database Availability Groups (DAGs), provide more flexible and more reliable database replication. For databases in a DAG that have two or more healthy copies, the database consistency checking step can even be skipped. Source: http://msdn.microsoft.com/en-us/library/dd877010%28v=exchg.140%29.aspx - Since VSS operations use the BA client (DSMAGENT) to perform the backup of files, you can try increasing the RESOURCEUTILIZATION in the DSM.OPT file for the DSMAGENT (not the TDP) to allocate more threads to handle the backup load. - You can also create separate instances for backing up, but... if you do this, they should be snapping separate volumes. If all your databases or logs occupy the same volume, you cannot split them up. In addition, you will want to stagger your backup starts so that the snaps of the different volumes. - Perform backups from the DAG passive copies so that it offloads the impact to production databases. Thanks, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 08/25/2011 12:24:42 PM: From: Ben Bullock bbull...@bcidaho.com To: ADSM-L@vm.marist.edu Date: 08/25/2011 03:37 PM Subject: Re: Exchange Legacy 2007 to VSS 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu We recently moved from Exchange 2007 to 2010 on our environment. Just for a reference, we have about 3.3 TB of exchange data. Our TSM server is v5.5.5.0 running on AIX 6.1. The exchange servers have v6.1.3.0 of the TDP client, v6.2.2.0 of the BA client, Windows 2008 R2 SP1 and Exchange 2010 SP1. We have found that the transfer of the data runs about the same once it starts moving data to TSM, but the VSS parts (which are no longer optional) caused our backups to take almost twice as long. We weren't able to find anything in the TDP documentation to tell us why backups were now taking twice as long, however by looking at the IO on the SAN and Exchange logs, we believe we were able to determine what was going on. Feel free to chime in if you think our assessment is incorrect. By default, the TDP causes the exchange software do its own an integrity check of the database every time it does a backup, (either full, incremental or differential). We found that essentially doubles the time the backup takes, because it seems to read/check what it wants to send for the integrity check and then read/send the data to TDP/TSM on another pass. So you are essentially reading all the bits twice for every backup. It seems like a rather inefficient way to run it, but perhaps that's the way it has to be done. There's a flag you can put in so that the integrity check is not called (/SKIPINTEGRITYCHECK), but there is an inherent risk in skipping it. Then again, it didn't do this check for the Legacy backups and we never had a corruption problem with the backups, but YMMV. Currently, a weekly full and daily differentials meet our SLA, and we are toying with the idea of leaving the integrity check on the fulls (and have it take twice as long as it used to) and turning it off for the differentials. Your choice may be different depending on your risk assessment. Test it out for yourself and let us know if your experiences are any different from ours. Ben -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Ian Smith Sent: Wednesday, August 24, 2011 10:12 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance Our Mail/GroupWare service is migrating from Exchange 2007 to 2010 in the next few months. Currently we employ streaming (Legacy) backups across a private 2Gbit bonded private link direct to LTO5 tape and get around 50MByte/s for fulls and around 16-20Mbyte/s for incrementals. In all we have about 20TB of mailstore. I've searched the threads and the user docs, developerworks etc etc but can't find any real figures on how legacy versus vss backups compare. I suspect they will be slower, because of the additional steps in the whole VSS backup protocol but I'm happy to be surprised. If anybody has any real world figures that address the questions below, I'd be truly grateful. 1. Legacy versus VSS-assisted (MS VSS provider
Re: Exchange Legacy 2007 to VSS 2010 backup performance
Del, Thanks for piping in with the official word, since all my statements were anecdotal in nature. I wasn't too far off though. ;-) I actually like the idea of the integrity check being performed at the same time as the backup and then failing the backup if it finds anything. It just seems more efficient than reading the data twice. Our SAN admins I'm sure would prefer the lesser number of IOPS on the storage. Are there any options/flags I can use to make it behave that way? Thanks, Ben -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Del Hoobler Sent: Thursday, August 25, 2011 2:23 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance For the most part, your description is accurate... but I have a few comments: - Actually... an integrity check was performed for legacy backups, but the Exchange Server did it while it was reading the data, so the penalty was not as bad. If the Exchange server found an integrity problem while running the check, it would fail the backup. - The integrity check for VSS backups is done after the snapshot, and it must read all the pages of the .EDB and .LOG files using an Exchange interface to do it - Microsoft has relaxed their requirement for certain Exchange 2010 environments. They say: Database mobility features, including Database Availability Groups (DAGs), provide more flexible and more reliable database replication. For databases in a DAG that have two or more healthy copies, the database consistency checking step can even be skipped. Source: http://msdn.microsoft.com/en-us/library/dd877010%28v=exchg.140%29.aspx - Since VSS operations use the BA client (DSMAGENT) to perform the backup of files, you can try increasing the RESOURCEUTILIZATION in the DSM.OPT file for the DSMAGENT (not the TDP) to allocate more threads to handle the backup load. - You can also create separate instances for backing up, but... if you do this, they should be snapping separate volumes. If all your databases or logs occupy the same volume, you cannot split them up. In addition, you will want to stagger your backup starts so that the snaps of the different volumes. - Perform backups from the DAG passive copies so that it offloads the impact to production databases. Thanks, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 08/25/2011 12:24:42 PM: From: Ben Bullock bbull...@bcidaho.com To: ADSM-L@vm.marist.edu Date: 08/25/2011 03:37 PM Subject: Re: Exchange Legacy 2007 to VSS 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu We recently moved from Exchange 2007 to 2010 on our environment. Just for a reference, we have about 3.3 TB of exchange data. Our TSM server is v5.5.5.0 running on AIX 6.1. The exchange servers have v6.1.3.0 of the TDP client, v6.2.2.0 of the BA client, Windows 2008 R2 SP1 and Exchange 2010 SP1. We have found that the transfer of the data runs about the same once it starts moving data to TSM, but the VSS parts (which are no longer optional) caused our backups to take almost twice as long. We weren't able to find anything in the TDP documentation to tell us why backups were now taking twice as long, however by looking at the IO on the SAN and Exchange logs, we believe we were able to determine what was going on. Feel free to chime in if you think our assessment is incorrect. By default, the TDP causes the exchange software do its own an integrity check of the database every time it does a backup, (either full, incremental or differential). We found that essentially doubles the time the backup takes, because it seems to read/check what it wants to send for the integrity check and then read/send the data to TDP/TSM on another pass. So you are essentially reading all the bits twice for every backup. It seems like a rather inefficient way to run it, but perhaps that's the way it has to be done. There's a flag you can put in so that the integrity check is not called (/SKIPINTEGRITYCHECK), but there is an inherent risk in skipping it. Then again, it didn't do this check for the Legacy backups and we never had a corruption problem with the backups, but YMMV. Currently, a weekly full and daily differentials meet our SLA, and we are toying with the idea of leaving the integrity check on the fulls (and have it take twice as long as it used to) and turning it off for the differentials. Your choice may be different depending on your risk assessment. Test it out for yourself and let us know if your experiences are any different from ours. Ben -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Ian Smith Sent: Wednesday, August 24, 2011 10:12 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Exchange Legacy
Ang: Re: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance
Ben, not sure if you have counted it in, but have you turned off the mailbox history part? We have an Exchange environment running 4x2TB and the integrity check wasnt really the big issue, but with 19000 users, the mailbox history was. Daniel Sparrman Exist i Stockholm AB Växel: 08-754 98 00 Fax: 08-754 97 30 daniel.sparr...@exist.se http://www.existgruppen.se Posthusgatan 1 761 30 NORRTÄLJE -ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU skrev: - Till: ADSM-L@VM.MARIST.EDU Från: Del Hoobler hoob...@us.ibm.com Sänt av: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU Datum: 08/25/2011 22:22 Ärende: Re: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance For the most part, your description is accurate... but I have a few comments: - Actually... an integrity check was performed for legacy backups, but the Exchange Server did it while it was reading the data, so the penalty was not as bad. If the Exchange server found an integrity problem while running the check, it would fail the backup. - The integrity check for VSS backups is done after the snapshot, and it must read all the pages of the .EDB and .LOG files using an Exchange interface to do it - Microsoft has relaxed their requirement for certain Exchange 2010 environments. They say: Database mobility features, including Database Availability Groups (DAGs), provide more flexible and more reliable database replication. For databases in a DAG that have two or more healthy copies, the database consistency checking step can even be skipped. Source: http://msdn.microsoft.com/en-us/library/dd877010%28v=exchg.140%29.aspx - Since VSS operations use the BA client (DSMAGENT) to perform the backup of files, you can try increasing the RESOURCEUTILIZATION in the DSM.OPT file for the DSMAGENT (not the TDP) to allocate more threads to handle the backup load. - You can also create separate instances for backing up, but... if you do this, they should be snapping separate volumes. If all your databases or logs occupy the same volume, you cannot split them up. In addition, you will want to stagger your backup starts so that the snaps of the different volumes. - Perform backups from the DAG passive copies so that it offloads the impact to production databases. Thanks, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 08/25/2011 12:24:42 PM: From: Ben Bullock bbull...@bcidaho.com To: ADSM-L@vm.marist.edu Date: 08/25/2011 03:37 PM Subject: Re: Exchange Legacy 2007 to VSS 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu We recently moved from Exchange 2007 to 2010 on our environment. Just for a reference, we have about 3.3 TB of exchange data. Our TSM server is v5.5.5.0 running on AIX 6.1. The exchange servers have v6.1.3.0 of the TDP client, v6.2.2.0 of the BA client, Windows 2008 R2 SP1 and Exchange 2010 SP1. We have found that the transfer of the data runs about the same once it starts moving data to TSM, but the VSS parts (which are no longer optional) caused our backups to take almost twice as long. We weren't able to find anything in the TDP documentation to tell us why backups were now taking twice as long, however by looking at the IO on the SAN and Exchange logs, we believe we were able to determine what was going on. Feel free to chime in if you think our assessment is incorrect. By default, the TDP causes the exchange software do its own an integrity check of the database every time it does a backup, (either full, incremental or differential). We found that essentially doubles the time the backup takes, because it seems to read/check what it wants to send for the integrity check and then read/send the data to TDP/TSM on another pass. So you are essentially reading all the bits twice for every backup. It seems like a rather inefficient way to run it, but perhaps that's the way it has to be done. There's a flag you can put in so that the integrity check is not called (/SKIPINTEGRITYCHECK), but there is an inherent risk in skipping it. Then again, it didn't do this check for the Legacy backups and we never had a corruption problem with the backups, but YMMV. Currently, a weekly full and daily differentials meet our SLA, and we are toying with the idea of leaving the integrity check on the fulls (and have it take twice as long as it used to) and turning it off for the differentials. Your choice may be different depending on your risk assessment. Test it out for yourself and let us know if your experiences are any different from ours. Ben -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Ian Smith Sent: Wednesday, August 24, 2011 10:12 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Exchange Legacy 2007 to VSS 2010 backup performance Our Mail/GroupWare service is migrating from Exchange 2007
Re: Exchange Legacy 2007 to VSS 2010 backup performance
Hi Ben, I wish TSM could perform both at the same time, but it is not doable. This is a Microsoft restriction, not a TSM implementation issue. The Microsoft interfaces to check the integrity are at the whole file-level, not at a buffer-level like we would need to perform the backup and integrity check at the same time. We have asked Microsoft for a better interface, but it has not raised high enough in their queue. And with their current emphasis on replication and also allowing customers to skip the integrity check in those DAG environments, I suspect they will not invest more in this and encourage customers to use Exchange 2010 DAG replication to handle their integrity checking. Thanks, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 08/25/2011 04:41:08 PM: From: Ben Bullock bbull...@bcidaho.com To: ADSM-L@vm.marist.edu Date: 08/25/2011 04:57 PM Subject: Re: Exchange Legacy 2007 to VSS 2010 backup performance Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Del, Thanks for piping in with the official word, since all my statements were anecdotal in nature. I wasn't too far off though. ;-) I actually like the idea of the integrity check being performed at the same time as the backup and then failing the backup if it finds anything. It just seems more efficient than reading the data twice. Our SAN admins I'm sure would prefer the lesser number of IOPS on the storage. Are there any options/flags I can use to make it behave that way? Thanks, Ben
Exchange Legacy 2007 to VSS 2010 backup performance
Our Mail/GroupWare service is migrating from Exchange 2007 to 2010 in the next few months. Currently we employ streaming (Legacy) backups across a private 2Gbit bonded private link direct to LTO5 tape and get around 50MByte/s for fulls and around 16-20Mbyte/s for incrementals. In all we have about 20TB of mailstore. I've searched the threads and the user docs, developerworks etc etc but can't find any real figures on how legacy versus vss backups compare. I suspect they will be slower, because of the additional steps in the whole VSS backup protocol but I'm happy to be surprised. If anybody has any real world figures that address the questions below, I'd be truly grateful. 1. Legacy versus VSS-assisted (MS VSS provider) backups of Exchange 2007 to TSM server. 2. VSS-assisted backups of Exchange 2007 versus 2010. This is a big migration for all of us and I'd like to have an idea of what we should expect in testing. Thanks Ian Smith Oxford University/ England.
Tsm server possibly limiting backup performance
Tsm server 6.2.2 OS redhat enterprise linux 6.0 Client tdp for exchange, sqlserver, and backup archive client Client connected via 1 gb ethernet connection. In my experience, I yhave never been able to se throughput of much more than 200 mb/s during a tsm backup, whether tdp or regular client. I wonder, is TSM limiting a single session's bandwidth so as to insure availability for other client sessions which might begin; i.e. prevent one backup stream from monopolizing the server's network connection? I have run ftp and other tests to the server, and achieved throughput of around 800 mb/s over the same connection. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310
Re: Tsm server possibly limiting backup performance
A little more explanation is going to be needed. A 1GB ethernet is not going to give 800mb/s, let alone 200mb/s. 1GB ethernet is only going to provide one direction throughput of 110mb/s. Do you mean a 10gb ethernet connection? From: Lee, Gary D. g...@bsu.edu To: ADSM-L@VM.MARIST.EDU Date: 05/27/2011 09:35 AM Subject:Tsm server possibly limiting backup performance Sent by:ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU Tsm server 6.2.2 OS redhat enterprise linux 6.0 Client tdp for exchange, sqlserver, and backup archive client Client connected via 1 gb ethernet connection. In my experience, I yhave never been able to se throughput of much more than 200 mb/s during a tsm backup, whether tdp or regular client. I wonder, is TSM limiting a single session's bandwidth so as to insure availability for other client sessions which might begin; i.e. prevent one backup stream from monopolizing the server's network connection? I have run ftp and other tests to the server, and achieved throughput of around 800 mb/s over the same connection. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310 - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: Tsm server possibly limiting backup performance
To make sure we're all talking the same language here, can we agree that 'b' means 'bit' and 'B' means 'byte'? This conversation could get very confusing otherwise... /DMc On 27 May 2011, at 15:09, Richard Rhodes rrho...@firstenergycorp.com wrote: A little more explanation is going to be needed. A 1GB ethernet is not going to give 800mb/s, let alone 200mb/s. 1GB ethernet is only going to provide one direction throughput of 110mb/s. Do you mean a 10gb ethernet connection? From: Lee, Gary D. g...@bsu.edu To: ADSM-L@VM.MARIST.EDU Date: 05/27/2011 09:35 AM Subject:Tsm server possibly limiting backup performance Sent by:ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU Tsm server 6.2.2 OS redhat enterprise linux 6.0 Client tdp for exchange, sqlserver, and backup archive client Client connected via 1 gb ethernet connection. In my experience, I yhave never been able to se throughput of much more than 200 mb/s during a tsm backup, whether tdp or regular client. I wonder, is TSM limiting a single session's bandwidth so as to insure availability for other client sessions which might begin; i.e. prevent one backup stream from monopolizing the server's network connection? I have run ftp and other tests to the server, and achieved throughput of around 800 mb/s over the same connection. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310 - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: Tsm server possibly limiting backup performance
I think clarification is needed on mb/s vs. MB/s. a 1gb (gigabit) (note lowercase) connection will certainly do 800 mb/s (megabit/s). it won't, however, do 800 MB/s (megabyte/s) Paul On Fri, May 27, 2011 at 8:09 AM, Richard Rhodes rrho...@firstenergycorp.com wrote: A little more explanation is going to be needed. A 1GB ethernet is not going to give 800mb/s, let alone 200mb/s. 1GB ethernet is only going to provide one direction throughput of 110mb/s. Do you mean a 10gb ethernet connection? From: Lee, Gary D. g...@bsu.edu To: ADSM-L@VM.MARIST.EDU Date: 05/27/2011 09:35 AM Subject:Tsm server possibly limiting backup performance Sent by:ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU Tsm server 6.2.2 OS redhat enterprise linux 6.0 Client tdp for exchange, sqlserver, and backup archive client Client connected via 1 gb ethernet connection. In my experience, I yhave never been able to se throughput of much more than 200 mb/s during a tsm backup, whether tdp or regular client. I wonder, is TSM limiting a single session's bandwidth so as to insure availability for other client sessions which might begin; i.e. prevent one backup stream from monopolizing the server's network connection? I have run ftp and other tests to the server, and achieved throughput of around 800 mb/s over the same connection. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310 - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: Tsm server possibly limiting backup performance
Sorry, thought I had my cases correct. 1 gb (gigabit) ethernet ocnnection. Ftp and other tests yielded 800 megabit performance, Tsm sessions average 200 to 250 megabit. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310 -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Paul Fielding Sent: Friday, May 27, 2011 10:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Tsm server possibly limiting backup performance I think clarification is needed on mb/s vs. MB/s. a 1gb (gigabit) (note lowercase) connection will certainly do 800 mb/s (megabit/s). it won't, however, do 800 MB/s (megabyte/s) Paul On Fri, May 27, 2011 at 8:09 AM, Richard Rhodes rrho...@firstenergycorp.com wrote: A little more explanation is going to be needed. A 1GB ethernet is not going to give 800mb/s, let alone 200mb/s. 1GB ethernet is only going to provide one direction throughput of 110mb/s. Do you mean a 10gb ethernet connection? From: Lee, Gary D. g...@bsu.edu To: ADSM-L@VM.MARIST.EDU Date: 05/27/2011 09:35 AM Subject:Tsm server possibly limiting backup performance Sent by:ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU Tsm server 6.2.2 OS redhat enterprise linux 6.0 Client tdp for exchange, sqlserver, and backup archive client Client connected via 1 gb ethernet connection. In my experience, I yhave never been able to se throughput of much more than 200 mb/s during a tsm backup, whether tdp or regular client. I wonder, is TSM limiting a single session's bandwidth so as to insure availability for other client sessions which might begin; i.e. prevent one backup stream from monopolizing the server's network connection? I have run ftp and other tests to the server, and achieved throughput of around 800 mb/s over the same connection. Gary Lee Senior System Programmer Ball State University phone: 765-285-1310 - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: Tsm server possibly limiting backup performance
The networking itself is just one element in the overall backup operation... You don't say what measurement point you are looking at for your throughput numbers. If it is the Network data transfer rate TSM summary statistic, that reflects TSM interacting with the network layer of the operating system, with affecters such as TSM TCPWindowsize, Path MTU Discovery where that is in effect, etc. Unexpected, elongated network routing can delay things; but your FTP test should have run into that if were a factor...unless conditions are changing depending upon time of day. When a TDP is involved, it can only pass data as fast as the back end application provides it. A traditional B/A backup can be impaired by the paging incited by the amount of memory taken by the Active files list. If standard measurement points don't reveal a cause, a client trace with a representative data set can be illuminating. Richard Sims
Re: vcb backup performance
Steve, VMware has announced the end of availability for VCB. You can see the VMware support policy at http://www.vmware.com/support/policies/lifecycle/vi/eos.html There is a new backup product strategy called 'vStorage APIs for Data Protection (VADP). See http://www.vmware.com/support/policies/lifecycle/vi/eos.html The announcement goes on to say We have also been working with several backup partners to integrate VADP into their solutions to make backup of vSphere Virtual Machines fast, efficient and easy to deploy compared to VCB and other backup solutions. Several of our major backup partners have already released VADP integrated backup products and we expect most of the major backup partners to have VADP integrated backup software by the upcoming feature release of the vSphere platform in 2010. Best wishes, Keith Arbogast Indiana University
VCB backup performance
Hi All Thanks to all those that responded to my time travel query, testing is in progress on that one. One of my customers has an issue with a burgeoning VCB infrastructure. There are two VCB proxies and a lot of VMs being backed up, TSM Server 5.5.3 and client 6.1.2. Most of this is a daily incremental and a monthly full, but there are a couple of big file-servers with lots of small files that get a daily full. VCB backups are single threaded, and we are having trouble getting through them all within the window. TSM doesn't seem to be the problem, we have increased resourceutilization on the proxy without much effect. The bottleneck seems to be the VMWare snapshot/mount/dismount parts of the process, and there is not a lot of visibility of those. 1. Is there any good doc on this process and how to improve its performance? 2. Can the proxy side be effectively parallelized, and if so how? Thanks Steve TSM Admin, Paraparaumu, New Zealand
Re: VCB backup performance
Hi Steve, Been there, done that with VMWare backups. You can parallelize what you are doing now by installing multiple schedulers on your proxy machine. Give it 2 (or more) node names (e.g., PROXYVCB1, PROXYVCB2), install 2 schedulers, 2 dsm.opt files, give them each half the work to do. (Also no reason you can't add more proxy machines.) That will only help the Proxy client drive multiple full backups at once, it won't improve the speed of the VCB fulls, which definitely have performance issues. VMWARE is addressing the VCB performance issue by replacing VCB altogether, so you should be looking ahead to ESX 4.0 or 4.1 for a long-term solution. In VSPHERE (ESX 4.0), VMWare has created a Storage API for their own backup utility called VDR, and for 3rd party vendors to write to (VCB is still supported in 4.0 as well). I have a customer using VDR+TSM with success so far. Like most VM backup products, VDR backs up only to disk. But it's part of ESX 4.0 Enterprise, it's supported by VMWare, gives you incremental and fulls, and DEDUPS, so the disk repository doesn't get all that large. It also thoughtfully keeps the sizes of the resulting files in the repository below 2GB, at least as far as we've seen, so we use TSM subfile backup to back up the repository and get stuff out to tape and vaulted. The VMWare world changes VERY fast. I went to a VMWare UG last week, and the VMWare rep said 4.1 will be out this year, and will NOT support VCB at all any longer. Thus my encouragement to move ahead to something that works a different way. TSM 6.2 will be out in March, and will support the VMWare API. I don't have any doc yet explaining exactly what it will do in practical terms, but it won't be VCB. ..and congrats on the new gig.! Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Steve Harris Sent: Wednesday, March 17, 2010 6:18 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] VCB backup performance Hi All Thanks to all those that responded to my time travel query, testing is in progress on that one. One of my customers has an issue with a burgeoning VCB infrastructure. There are two VCB proxies and a lot of VMs being backed up, TSM Server 5.5.3 and client 6.1.2. Most of this is a daily incremental and a monthly full, but there are a couple of big file-servers with lots of small files that get a daily full. VCB backups are single threaded, and we are having trouble getting through them all within the window. TSM doesn't seem to be the problem, we have increased resourceutilization on the proxy without much effect. The bottleneck seems to be the VMWare snapshot/mount/dismount parts of the process, and there is not a lot of visibility of those. 1. Is there any good doc on this process and how to improve its performance? 2. Can the proxy side be effectively parallelized, and if so how? Thanks Steve TSM Admin, Paraparaumu, New Zealand
System State Backup Performance
IC63094: SYSTEM STATE BACKUP OPTIMIZATION ENHANCEMENT This is a development APAR to improve the efficiency of the method the TSM client uses to query the TSM server database for information about system state files. Additional Keywords: systemstate Local fix Problem summary * USERS AFFECTED: All version 5.5 and 6.1 backup-archive * * clients running on all Windows operating * * systems except Windows XP. * * PROBLEM DESCRIPTION: See ERROR DESCRIPTION * * RECOMMENDATION: Apply fixing level when available. This * * problem is currently projected to be fixed * * in levels 5.5.3 and 6.1.4. Note that this* * is subject to change at the discretion of* * IBM. * * Problem conclusionThe client has been changed so that during system state backup, instead of querying the TSM server for each of the system files one file per transaction, it will now query all of the system files in a single transaction. http://www-01.ibm.com/support/docview.wss?uid=swg1IC63094
Re: backup performance on intel server
I must have under utilized my shift key, you are correct, for every 1Gb, you in theory can get 100MB/sec. I am not sure how slow the LTO4 can write, if it cannot go slow enough there may be some shoe shining. The external disks sound fine, the internals will depend on the config and controllers, but should be good. Jumbo frames will be your friend. A clear network path will be needed. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Stef Coene Sent: Thursday, April 30, 2009 1:49 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] backup performance on intel server On Thursday 30 April 2009, Huebner,Andy,FORT WORTH,IT wrote: 1 Gb Ethernet is about 100Mb/sec Maybe nick picking, but for me: 100Mb = 100 Mbit, 100MB is 100 MByte. LTO4 is about 120Mb/sec + compression Mhh, I was counting 80 MB/s up to 160 MB/s with compression ratio 2:1. A good SCSI or FC drive is good for about 100Mb/sec The disk storage fot the client is an n Series from IBM (16 TB), but also 12 TB local (scsi attached) disks. Assuming you can stream 3 threads across the Ethernet, you will not be able to keep the 3 LTO4 drives busy because of the network. The faster tape drives were not intended to take data from the Ethernet. Assuming some sort of RAID and careful selection of file system backup times the disk can keep up with the network if the read is sequential and the backup software can find the data fast enough. Make sure you have multiple busses. Basically I would look for a PCI-Express system with enough slots for the Ethernet and RAID or HBA controllers. I would be stressed about making the window and since you are using a traditional backup solution the fulls will be tough. The customer was counting on a 24hr or even a 48hr backup window on saterday and sunday. And for the stress, this is solution is pushed from the head quarters in America to the customer in Belgium. They wanted a DS48 from IBM + TSM Thanx for the info. Stef This e-mail (including any attachments) is confidential and may be legally privileged. If you are not an intended recipient or an authorized representative of an intended recipient, you are prohibited from using, copying or distributing the information in this e-mail or its attachments. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message and any attachments. Thank you.
backup performance on intel server
Hi, I have a non-TSM related question, but I think it is the same for all backup software. We need to backup 28 TB for a cutsomer with daily changes of appr 8 TB. If you take a backup window of 8 hr, this is 1 TB/hour. This is also 280 MB/s so 3,5 LTO4 tape streamers (if you take 80MB/s per tape streamer). I can not use SAN backup, so all data has to come in over the network, so I need at least 3 x 1 GB network connections. What kind of intel box running windows can handle this workload? Is the PCI bus of a normal x86 server enough or do I need some high end x86 hardware? Stef PS: the backup software Backup Exec from Symantec
Re: backup performance on intel server
1 Gb Ethernet is about 100Mb/sec LTO4 is about 120Mb/sec + compression A good SCSI or FC drive is good for about 100Mb/sec Assuming you can stream 3 threads across the Ethernet, you will not be able to keep the 3 LTO4 drives busy because of the network. The faster tape drives were not intended to take data from the Ethernet. Assuming some sort of RAID and careful selection of file system backup times the disk can keep up with the network if the read is sequential and the backup software can find the data fast enough. Make sure you have multiple busses. Basically I would look for a PCI-Express system with enough slots for the Ethernet and RAID or HBA controllers. I would be stressed about making the window and since you are using a traditional backup solution the fulls will be tough. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Stef Coene Sent: Thursday, April 30, 2009 1:04 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] backup performance on intel server Hi, I have a non-TSM related question, but I think it is the same for all backup software. We need to backup 28 TB for a cutsomer with daily changes of appr 8 TB. If you take a backup window of 8 hr, this is 1 TB/hour. This is also 280 MB/s so 3,5 LTO4 tape streamers (if you take 80MB/s per tape streamer). I can not use SAN backup, so all data has to come in over the network, so I need at least 3 x 1 GB network connections. What kind of intel box running windows can handle this workload? Is the PCI bus of a normal x86 server enough or do I need some high end x86 hardware? Stef PS: the backup software Backup Exec from Symantec This e-mail (including any attachments) is confidential and may be legally privileged. If you are not an intended recipient or an authorized representative of an intended recipient, you are prohibited from using, copying or distributing the information in this e-mail or its attachments. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message and any attachments. Thank you.
Re: backup performance on intel server
On Thursday 30 April 2009, Huebner,Andy,FORT WORTH,IT wrote: 1 Gb Ethernet is about 100Mb/sec Maybe nick picking, but for me: 100Mb = 100 Mbit, 100MB is 100 MByte. LTO4 is about 120Mb/sec + compression Mhh, I was counting 80 MB/s up to 160 MB/s with compression ratio 2:1. A good SCSI or FC drive is good for about 100Mb/sec The disk storage fot the client is an n Series from IBM (16 TB), but also 12 TB local (scsi attached) disks. Assuming you can stream 3 threads across the Ethernet, you will not be able to keep the 3 LTO4 drives busy because of the network. The faster tape drives were not intended to take data from the Ethernet. Assuming some sort of RAID and careful selection of file system backup times the disk can keep up with the network if the read is sequential and the backup software can find the data fast enough. Make sure you have multiple busses. Basically I would look for a PCI-Express system with enough slots for the Ethernet and RAID or HBA controllers. I would be stressed about making the window and since you are using a traditional backup solution the fulls will be tough. The customer was counting on a 24hr or even a 48hr backup window on saterday and sunday. And for the stress, this is solution is pushed from the head quarters in America to the customer in Belgium. They wanted a DS48 from IBM + TSM Thanx for the info. Stef
Re: Win2K3 backup performance
We have had several instances of slow backup times to Win servers (with 1gb connections) where the throughput looked almost exactly like 100mb connections. As we looked at it, we found that all access to these servers (including backups) was going through a BigIP box with only 100mb connections. Rick Sam Sheppard [EMAIL PROTECTED] .GOV To Sent by: ADSM: ADSM-L@VM.MARIST.EDU Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Win2K3 backup performance 01/31/2008 07:08 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668 - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: Win2K3 backup performance
On Thu, 31 Jan 2008 17:39:49 -0800, Larry Peifer [EMAIL PROTECTED] said: The fastest backup I can get from a Win2003 server, TSM 5.4.0.4 client, to an AIX TSM server, 5.4.0.0 is 17Mb/sec. over GigE with a dedicated 2nd nic for backup only. I'd like to hear from others who are getting better speeds and what tuning can be done. That's 3000 files averaging 100Mb each so it takes about 4.5 hours to transfer about 300 Gig. May be obvious, but have you ruled out throughput bottleneck on the TSM server? Are you writing to a contested disk pool, etc? Or if it's tape, are you in a sour spot w.r.t. streaming, perhaps shoeshining? - Allen S. Rout
Re: Win2K3 backup performance
Also, are you sure you can read the data off of disks any faster? I'm not sure what the equivalent cmd would be on Windows, but on Unix I'd try a dd if=file of=/dev/null bs=128k (or some variant) and see just how fast the data can be read from the drives. Rick Allen S. Rout [EMAIL PROTECTED] Sent by: ADSM:To Dist Stor ADSM-L@VM.MARIST.EDU Manager cc [EMAIL PROTECTED] .EDU Subject Re: Win2K3 backup performance 02/01/2008 09:57 AM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU On Thu, 31 Jan 2008 17:39:49 -0800, Larry Peifer [EMAIL PROTECTED] said: The fastest backup I can get from a Win2003 server, TSM 5.4.0.4 client, to an AIX TSM server, 5.4.0.0 is 17Mb/sec. over GigE with a dedicated 2nd nic for backup only. I'd like to hear from others who are getting better speeds and what tuning can be done. That's 3000 files averaging 100Mb each so it takes about 4.5 hours to transfer about 300 Gig. May be obvious, but have you ruled out throughput bottleneck on the TSM server? Are you writing to a contested disk pool, etc? Or if it's tape, are you in a sour spot w.r.t. streaming, perhaps shoeshining? - Allen S. Rout - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Win2K3 backup performance
We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668
Re: Win2K3 backup performance
Hi Sam, I had a similar problem with a notes server running W2003. I was using the tdp client for lotus notes. My work around was to take a second network interface on the server and lock both the port and the client down to 100/full duplex. This tdp client is the only one I have to run on this second interface and it works fine now. I don't know if you have a hdwe. second interface, but if you do, try adding another ip address and give it a try...TSM allows for the use of more then one hdwe. interface... At any rate kindly let me know what your resolution is as I'm interested... thx jimmyd James Drozynski IBM Pittsburgh Lab 11 Stanwix Street Pittsburgh Pa. 15222 Tel:412-667-4421 Fax:412-667-6975 Tie:989-4421
Re: Win2K3 backup performance
The fastest backup I can get from a Win2003 server, TSM 5.4.0.4 client, to an AIX TSM server, 5.4.0.0 is 17Mb/sec. over GigE with a dedicated 2nd nic for backup only. I'd like to hear from others who are getting better speeds and what tuning can be done. That's 3000 files averaging 100Mb each so it takes about 4.5 hours to transfer about 300 Gig. Sam Sheppard [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 01/31/2008 04:08 PM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject [ADSM-L] Win2K3 backup performance We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668
Re: Win2K3 backup performance
This is what I found to work with our system, with a 1Gb backup network. * TCPWINDOWSIZE 2048 - Maximum setting TCPWindowsize 512 * TCPBUFFSIZE 512 - Maximum setting TCPBuffSize 255 The maximum settings David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Sam Sheppard Sent: Thursday, January 31, 2008 2:09 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Win2K3 backup performance We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668 The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the original message. Thank you.
Re: Win2K3 backup performance
Also set txnbytelimit to themaximum of 2GB (if you have a relatively new client 5.3 or later) and txngroupmax on the server 1024 (though that is not really your issue on this client but might be on clients with many smaller files) Upping txnbytelimit beyond the 25600 default may make a huge difference as you cut down on your transaction mini-commits dramatically... Kelly Lipp CTO, VP Manufacturing STORServer, Inc. 485-B Elkton Drive Colorado Springs, CO 80907 www.storserver.com http://www.storserver.com/ 719-266-8777 From: ADSM: Dist Stor Manager on behalf of David Vargas Sent: Thu 1/31/2008 8:12 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Win2K3 backup performance This is what I found to work with our system, with a 1Gb backup network. * TCPWINDOWSIZE 2048 - Maximum setting TCPWindowsize 512 * TCPBUFFSIZE 512 - Maximum setting TCPBuffSize 255 The maximum settings David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Sam Sheppard Sent: Thursday, January 31, 2008 2:09 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Win2K3 backup performance We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668 The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the original message. Thank you.
Re: Win2K3 backup performance
Hi, Have you recently upgraded the client, and are you using scheduled backups with sessioinitiation=serveronly? From 5.3.5.3 to latest 5.4 client I have an issue with above settings and performance. Ie, with newer clients performace drops with 70-80% when I backup larger files 1Gb. Dsmc and sessioinitiation=clientorserver works fine. //Henrik -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Sam Sheppard Sent: den 1 februari 2008 01:09 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Win2K3 backup performance We have a Windows 2003 client running TSM 5.4.1 backing up to a Solaris 10 TSM server running the 5.4.1 server. The client has several very large files to be backed up (300-400GB). We are finding on a 1.3GB test file that we can only get a throughput of about 10MB/sec (looks like 100Mb speed) even though this client is configured on a GigE VLan. One interesting aspect just discovered is that a restore of the same file got speeds of 37MB/sec, which is about the same as an FTP of the same file from the client to the server. Client, switch, and server all set to 1GB full. At this point, I'm completely mystified as to what might be behind these performance anomalies. I would expect much higher throughput rates on all of these tests, but would be satisfied if the backup would just perform at the same speed as the others. Client options: Server Options: TCPWindowsize 63 TCPWindowsize 131072 TCPBuffsize 32 Thanks Sam Sheppard San Diego Data Processing Corp. (858)-581-9668 --- The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Thank you.
backup performance
Hi list, I´ve started a backup operation at a Linux client using CENT OS (similar to RHat). The operation took 1 hour and 59 minutes to finish, and backed up 5.45GB of data.I think this is kind of slow..considering that the LTO3 tape unit is supposed to be very fast The TSM Server is 5.3.2 running on a W2K box and we have an IBM TotalStorage Ultrium LTO3 (400/800GB) unit connected through a SCSI cable. The linux server (the client) is a virtual machine running on a IBM x-series 366 machine under the VMWare ESX Server. The virtual servers (there are 23) are all configured under this VMWare and are stored in a DS-4300 storage connecte through FO to the x-series machine I am using data compression. Any ideas on how to make this operation faster ??? Thanks. Mario
Re: backup performance
Hot Diggety! Mario Behring was rumored to have written: I´ve started a backup operation at a Linux client using CENT OS (similar to RHat). The operation took 1 hour and 59 minutes to finish, and backed up 5.45GB of data.I think this is kind of slow..considering that the LTO3 tape unit is supposed to be very fast Can you post the end of job statistics? For example, in my dsmsched.log, I have: 05/18/06 22:32:33 --- SCHEDULEREC STATUS BEGIN 05/18/06 22:32:33 Total number of objects inspected: 845,061 05/18/06 22:32:33 Total number of objects backed up: 26,665 05/18/06 22:32:33 Total number of objects updated: 0 05/18/06 22:32:33 Total number of objects rebound: 0 05/18/06 22:32:33 Total number of objects deleted: 0 05/18/06 22:32:33 Total number of objects expired: 1,260 05/18/06 22:32:33 Total number of objects failed: 0 05/18/06 22:32:33 Total number of bytes transferred: 3.11 GB 05/18/06 22:32:33 Data transfer time: 464.32 sec 05/18/06 22:32:33 Network data transfer rate:7,040.43 KB/sec 05/18/06 22:32:33 Aggregate data transfer rate: 2,600.08 KB/sec 05/18/06 22:32:33 Objects compressed by:0% 05/18/06 22:32:33 Elapsed processing time: 00:20:57 05/18/06 22:32:33 --- SCHEDULEREC STATUS END If you can post yours for that backup run, it would help give some insight. Also, do you use a disk pool on the TSM server? For example: TSM client - TSM server (diskpool) - TSM server (tapes) If you do not use a diskpool, you will not be able to feed data to the LTO-3 tape drives fast enough. If that happens, it will do start-and-stop which dramatically slows down performance. -Dan
Re: backup performance
Looks to me based on the calculations if it goes over a little more than 2 hours acceptable. What's acceptable to me might not be acceptable to others. 225 GB per hour, or 62.5 MB/s LTO2 drives can writes 70 ~75 MB/s It's difficult to PD without seeing the whole environment where the bottle neck is, but one area I would check woud be how LTO drives are assigned to HBA card(s) on TSM server. If you have fibre LTO drives, I would check how HBAs are assigned to LTO drives. For example, If you have 1 HBA assigned to LTO drive 1 and 2 another HBA assigned to LTO drive 3,4 When backup occurs, if tapes are mounted from each HBA: drive1 and drive 3, would have a greater performance than say you were to use drive1 and drive 2. Sung Y. Lee ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 03/16/2006 06:46:40 AM: Hi, we use oracle TDP with LTO 2 dirvers for Rman database backup. Normally, the backup for database (450G) take 2h with 2 channels. but some time (intermittent) the backup take more than 2h30. the problem is that in this case only on channel work fine but the other is very slow, so when we do recovery test , the multiplexing not work. any idea please ? Thanks
backup performance
Hi, we use oracle TDP with LTO 2 dirvers for Rman database backup. Normally, the backup for database (450G) take 2h with 2 channels. but some time (intermittent) the backup take more than 2h30. the problem is that in this case only on channel work fine but the other is very slow, so when we do recovery test , the multiplexing not work. any idea please ? Thanks
Re: Network settings and poor backup performance.
Dear Rafa, could you me an invitation to get a gmail account? Thanks --.. -Original Message- From: Rafa [EMAIL PROTECTED] To: ADSM-L@VM.MARIST.EDU Sent: Wed, 30 Nov 2005 12:24:36 -0500 Subject: Re: [ADSM-L] Network settings and poor backup performance. On 11/30/05, Ray Louvier [EMAIL PROTECTED] wrote: Can anyone give the technical reason why auto detect on Server NICs cause such horrible performance for TSM server backups and restores. We This sounds like a network problem, not a TSM one. Quick way to prove it: try a backup/restore to another server on another location. I had a similar problem recently. My TSM servers are connected with gigabit ethernet interfaces to cisco switches. This cisco connected to another cisco switch where there was a 100Mbit connected server. Performance was dismal and the server interface showed CRC galore. The NIC on the server and the switch port were set to autodetect and both reported being 100mbit full duplex. The only way to solve this issue was to shutdown the interface and bring it back up again so they would renegotiate. Both still reported 100Mbit full but the CRC went away. This very same thing happened on two other servers. I don't know why the hocus pocus fixed it since nothing seems to have been changed but it worked. Advice: check for CRC. ___ Try the New Netscape Mail Today! Virtually Spam-Free | More Storage | Import Your Contact List http://mail.netscape.com
Network settings and poor backup performance.
Can anyone give the technical reason why auto detect on Server NICs cause such horrible performance for TSM server backups and restores. We have our TSM server NIC set to 100MB Full also our switch Port set to 100 Full but we have users with sorted networks such as 1 GB switches plugged into 100 M wall jacks and travel across several routers., well you get my meaning. These people or demanding to know why they take 8 days to restore 6 GB. They have their server NIC set to Auto detect, They have a 1 GB switch that the server and several others are plugged into because they want hi speed between these servers. But this switch plugs into a normal 100 meg wall jack to get to our data center. I have instructed them to put the NIC to 100 Full and possibly move off swith but they want a technical reason why. They are developers. -- This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Network settings and poor backup performance.
ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 11/30/2005 10:00:06 AM: Can anyone give the technical reason why auto detect on Server NICs cause such horrible performance for TSM server backups and restores. sound of hollow laughter This is a bone of network contention (no pun intended), particularly with Cisco networks. It is *not* a TSM problem. (As usual, TSM is the World's Best IT Infrastructure Problem Finder.) There are any number of whitepapers, Cisco and otherwise, that insist that all NICs in all networks have speed and duplex settings hardset to the best speed suited to the network hardware. The problem with autonegotiate is that, unless *every* network device in the chain is set to autonegotiate, the entire chain runs at 10/half. -- Mark Stapleton ([EMAIL PROTECTED]) -- Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation. ==
Re: Network settings and poor backup performance.
Because nic switch vendors never seem to implement the autodetect spec in the same and/or correct way. Depending on the manufacturer of the switch/nic, sometimes you have to use auto on both, sometimes you have to hardcode both, sometimes you have to hard code one but set the other to auto. There's very little rhymm, reason, or predictability. You just have find out what usually works with your particular combination of equipment. This makes it very difficult if you don't have standardized hardware in your environment, and have to figure this out for every individual combination of equipment. The exception to this is if you've got a gigabit nic plugging into a gigabit switch. That link virtually always has to be set to auto on both ends if you want to get gigabit speed. [EMAIL PROTECTED] 11/30/2005 10:00 AM Can anyone give the technical reason why auto detect on Server NICs cause such horrible performance for TSM server backups and restores. We have our TSM server NIC set to 100MB Full also our switch Port set to 100 Full but we have users with sorted networks such as 1 GB switches plugged into 100 M wall jacks and travel across several routers., well you get my meaning. These people or demanding to know why they take 8 days to restore 6 GB. They have their server NIC set to Auto detect, They have a 1 GB switch that the server and several others are plugged into because they want hi speed between these servers. But this switch plugs into a normal 100 meg wall jack to get to our data center. I have instructed them to put the NIC to 100 Full and possibly move off swith but they want a technical reason why. They are developers. -- This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message. Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Network settings and poor backup performance.
On 11/30/05, Ray Louvier [EMAIL PROTECTED] wrote: Can anyone give the technical reason why auto detect on Server NICs cause such horrible performance for TSM server backups and restores. We This sounds like a network problem, not a TSM one. Quick way to prove it: try a backup/restore to another server on another location. I had a similar problem recently. My TSM servers are connected with gigabit ethernet interfaces to cisco switches. This cisco connected to another cisco switch where there was a 100Mbit connected server. Performance was dismal and the server interface showed CRC galore. The NIC on the server and the switch port were set to autodetect and both reported being 100mbit full duplex. The only way to solve this issue was to shutdown the interface and bring it back up again so they would renegotiate. Both still reported 100Mbit full but the CRC went away. This very same thing happened on two other servers. I don't know why the hocus pocus fixed it since nothing seems to have been changed but it worked. Advice: check for CRC.
Re: Network settings and poor backup performance.
== On Wed, 30 Nov 2005 10:20:59 -0600, Mark Stapleton [EMAIL PROTECTED] said: sound of hollow laughter This is a bone of network contention (no pun intended), particularly with Cisco networks. It is *not* a TSM problem. (As usual, TSM is the World's Best IT Infrastructure Problem Finder.) What he said, but louder. I have a nearly autonomic response to new clients; very very often the initial backup goes very slowly, I suggest they check duplex settings, and things get going. - Allen S. Rout
Re: Slow backup performance
Hi Troy, Thanks for responding. Checked nic/switch both set to 100FD. Currently the server is at Netware 6.5SP2. Not sure what tsaup?? we're runing but the following are the revision numbers for the nlm's mentioned TSAFS.NLM 6.50.09 SMDR.NLM 6.54.01 TCP.NLM 6.57.03 TCPIP.NLM 6.57.06 The tsa.cfg file is thus, pretty close to the defaults used. Read Buffer Size: 65536 Read Threads Per Job: 4 Read Thread Allocation: 100 Read Ahead Throttle: 2 Cache Memory Threshold: 10 Disable Cluster: no We've currently set ResourceUtilization to 6 Couldn't find where the TCP Delayed Acknowlegement setting is applied? Is it a Novell OS option or a TSM server/client option? Thanks in advance Cameron Troy Frank [EMAIL PROTECTED] WISC.EDU To Sent by: ADSM: ADSM-L@VM.MARIST.EDU Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Re: Slow backup performance 12/10/2005 11:42 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU It would also be helpful to know what revision levels your server is at. Do you have any 6.5 service packs applied? Any of the TSA or TCP updates? What settings are you using when you load tsafs (look in sys:\etc\sms\tsa.cfg), and are you using ResourceUtilization or any other performance setttings in your dsm.opt? You might also want to check your server's tcp settings to make sure that TCP Delayed Ackknowlegement is off. As has been mentioned already, it's also very possible that there is something wrong in the nic/switch setup. There is no always right answer as to how they should be configged. Some nic/switch combos work best with both hardcoded, some with both on auto, and some with one hardcoded/one on auto. I would also make sure you're running the newest version of the nic driver. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 10/11/2005 6:30:04 AM Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM:cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside
Re: Slow backup performance
It looks like all your tsa/tcp modules are a little out of date. If possible, I'd get nw65sp4a installed on it. Alternately, you could just run the tsa5up18 tcp659j updates. Your tsa.cfg file looks pretty standard, and I've never really had trouble with it unless Cache Memory Threshold is set over 20/25%. The Delayed Acknowledgement parameter can be set a couple different ways. Either a set delayed acknowledgement = off/on at the commandline, or through the remote manager portal for the server (click set parameters, then communications). If you do it from the commandline, the setting won't stick after reboot unless you add it to the autoexec.ncf. The other thing I noticed is that you've got resourceutilization set to 6 in dsm.opt. Usually there's not much reason to set it any more than #tape drives+1. I've experienced problems here trying to set it to anything more than 4. ymmv, as always. If the problem is disk-system performance, you will want to verify that you're running the latest drivers for your raid/scsi adapter. Also be sure to update the controller's firmware to the latest version. Here's some netware settings to look at in Remote Manager under set parameters/disk Auto scan for devices ON Auto load of CDM modules ON Snapshot Allocate Block Count 5000 Sequential Elevator Depth 6000 (the default is way to low at 8) Enable IO Handicap Attribute OFF Mirrored Devices Are Out Of Sync Message Frequency 28 Remirror Block Size 1 Concurrent Remirror Requests 32 Ignore Partition Ownership OFF Ignore Disk Geometry OFF Multi-path Support OFF Enable Hardware Write Back ON (Depending on your controller, you might want to experiment with this OFF) Enable Disk Read After Write Verify OFF (make sure this is off) Auto LFVMount ON Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 10/18/2005 1:57:22 AM Hi Troy, Thanks for responding. Checked nic/switch both set to 100FD. Currently the server is at Netware 6.5SP2. Not sure what tsaup?? we're runing but the following are the revision numbers for the nlm's mentioned TSAFS.NLM 6.50.09 SMDR.NLM 6.54.01 TCP.NLM 6.57.03 TCPIP.NLM 6.57.06 The tsa.cfg file is thus, pretty close to the defaults used. Read Buffer Size: 65536 Read Threads Per Job: 4 Read Thread Allocation: 100 Read Ahead Throttle: 2 Cache Memory Threshold: 10 Disable Cluster: no We've currently set ResourceUtilization to 6 Couldn't find where the TCP Delayed Acknowlegement setting is applied? Is it a Novell OS option or a TSM server/client option? Thanks in advance Cameron Troy Frank [EMAIL PROTECTED] WISC.EDU To Sent by: ADSM: ADSM-L@VM.MARIST.EDU Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Re: Slow backup performance 12/10/2005 11:42 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU It would also be helpful to know what revision levels your server is at. Do you have any 6.5 service packs applied? Any of the TSA or TCP updates? What settings are you using when you load tsafs (look in sys:\etc\sms\tsa.cfg), and are you using ResourceUtilization or any other performance setttings in your dsm.opt? You might also want to check your server's tcp settings to make sure that TCP Delayed Ackknowlegement is off. As has been mentioned already, it's also very possible that there is something wrong in the nic/switch setup. There is no always right answer as to how they should be configged. Some nic/switch combos work best with both hardcoded, some with both on auto, and some with one hardcoded/one on auto. I would also make sure you're running the newest version of the nic driver. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 10/11/2005 6:30:04 AM Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client
Re: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance
Leigh, Many thanks for your clarifications, they helped a lot ! Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: Wednesday, 12 October, 2005 12:03 To: ADSM-L@VM.MARIST.EDU Subject: Re: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance Arnaud, I believe that things changed with Windows2000 and above. This is the note from the manual Windows 2000 and Windows XP provide a larger TCP receive window size when communicating with hosts that also provide this support, known as RFC1323. In these environments, a value greater than 63 may be useful. Refer to Description of Windows 2000 TCP Features, Microsoft knowledge base, article 224829, for details regarding TCP features in Windows 2000. I think that the maximums are as follows TCPBUFFSIZE 512 TCPWINDOWSIZE 2048 TXNBYTELIMIT2097152 I personally use the maximums, but there are always trade-offs. Increasing the buffer size and window size will mean that you require more resource and the overhead of retransmissions when an error is encountered is that much higher. However, with today's network speeds and reliability, coupled with the specification of the latest Intel hardware, I believe the trade-off is worth it. The TXNBYTELIMIT setting to the maximum is a recommendation for better performance when writing straight to LTO1 tape technology (or for that matter any other tape technology that cannot adapt it's tape streaming speed) Using the maximums has worked well for me in the environments I have used them in. This doesn't necessarily mean it's gospel. Also bear in mind that your TSM server has to support the above (ie RFC1323). Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of PAC Brion Arnaud Sent: 12 October 2005 10:00 To: ADSM-L@VM.MARIST.EDU Subject: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance David,(others ?) Like many people here, I'm in a perpetual quest of new tricks for speeding up the backups in our shop, and thought I could try using the TCPWindowsize and TCPBuffsize values you provided in your post for some testing. To my surprise, the backup speed increased dramatically (50 % gain)as when compared to the standard settings I was previously using (TCPW 63, TCPB 31). However I'm wondering how this is possible, as TSM user manual states that the TCPWindowsize max value for Windows based systems is limited to 63 ! Do you (or anybody) have an explanation on how this is possible, and maybe some clarifications about potential side effects of such TCP settings ? Many thanks in advance ... Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of David Vargas Sent: Wednesday, 12 October, 2005 02:29 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance Don't know if this will help, but I thought I'd share what I found when I was having problems with our Exchange TDP backup performance. In the dsm.opt file I added - TCPWindowsize255 TCPBuffSize 127 My backups went from 800K/sec to 30MB/sec and backup time went from 3 days to 2.6 hours for a total of 290GB of data on a 1Gb network. There were a couple of other settings that I changed, but this one is what made it fly. I still think it should be faster... You may need to adjust the numbers for optimal performance in your environment. - David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: Tuesday, October 11, 2005 1:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8
Re: Slow backup performance
David,(others ?) Like many people here, I'm in a perpetual quest of new tricks for speeding up the backups in our shop, and thought I could try using the TCPWindowsize and TCPBuffsize values you provided in your post for some testing. To my surprise, the backup speed increased dramatically (50 % gain)as when compared to the standard settings I was previously using (TCPW 63, TCPB 31). However I'm wondering how this is possible, as TSM user manual states that the TCPWindowsize max value for Windows based systems is limited to 63 ! Do you (or anybody) have an explanation on how this is possible, and maybe some clarifications about potential side effects of such TCP settings ? Many thanks in advance ... Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of David Vargas Sent: Wednesday, 12 October, 2005 02:29 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance Don't know if this will help, but I thought I'd share what I found when I was having problems with our Exchange TDP backup performance. In the dsm.opt file I added - TCPWindowsize255 TCPBuffSize 127 My backups went from 800K/sec to 30MB/sec and backup time went from 3 days to 2.6 hours for a total of 290GB of data on a 1Gb network. There were a couple of other settings that I changed, but this one is what made it fly. I still think it should be faster... You may need to adjust the numbers for optimal performance in your environment. - David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: Tuesday, October 11, 2005 1:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM: cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently
Re: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance
Arnaud, I believe that things changed with Windows2000 and above. This is the note from the manual Windows 2000 and Windows XP provide a larger TCP receive window size when communicating with hosts that also provide this support, known as RFC1323. In these environments, a value greater than 63 may be useful. Refer to Description of Windows 2000 TCP Features, Microsoft knowledge base, article 224829, for details regarding TCP features in Windows 2000. I think that the maximums are as follows TCPBUFFSIZE 512 TCPWINDOWSIZE 2048 TXNBYTELIMIT2097152 I personally use the maximums, but there are always trade-offs. Increasing the buffer size and window size will mean that you require more resource and the overhead of retransmissions when an error is encountered is that much higher. However, with today's network speeds and reliability, coupled with the specification of the latest Intel hardware, I believe the trade-off is worth it. The TXNBYTELIMIT setting to the maximum is a recommendation for better performance when writing straight to LTO1 tape technology (or for that matter any other tape technology that cannot adapt it's tape streaming speed) Using the maximums has worked well for me in the environments I have used them in. This doesn't necessarily mean it's gospel. Also bear in mind that your TSM server has to support the above (ie RFC1323). Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of PAC Brion Arnaud Sent: 12 October 2005 10:00 To: ADSM-L@VM.MARIST.EDU Subject: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance David,(others ?) Like many people here, I'm in a perpetual quest of new tricks for speeding up the backups in our shop, and thought I could try using the TCPWindowsize and TCPBuffsize values you provided in your post for some testing. To my surprise, the backup speed increased dramatically (50 % gain)as when compared to the standard settings I was previously using (TCPW 63, TCPB 31). However I'm wondering how this is possible, as TSM user manual states that the TCPWindowsize max value for Windows based systems is limited to 63 ! Do you (or anybody) have an explanation on how this is possible, and maybe some clarifications about potential side effects of such TCP settings ? Many thanks in advance ... Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of David Vargas Sent: Wednesday, 12 October, 2005 02:29 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance Don't know if this will help, but I thought I'd share what I found when I was having problems with our Exchange TDP backup performance. In the dsm.opt file I added - TCPWindowsize255 TCPBuffSize 127 My backups went from 800K/sec to 30MB/sec and backup time went from 3 days to 2.6 hours for a total of 290GB of data on a 1Gb network. There were a couple of other settings that I changed, but this one is what made it fly. I still think it should be faster... You may need to adjust the numbers for optimal performance in your environment. - David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: Tuesday, October 11, 2005 1:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM: cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow
Re: [SPAM: 6.500] Re: [ADSM-L] Slow backup performance
Cameron If I understand your scenario correctly, you have 5GB worth of data, with approx 80,000 files and very little change data. In my experience 80,000 files is small and the catalogue that would be sent from the TSM server would be small also. Can you watch the client session during an incremental backup and determine how much data (catalogue information) is SENT to the client and how long this takes. Also from the dsmsched.log, how long is it between the backup starting the first entry of '* Processed 500 files *'. This will give you an idea of the time that it takes to transmit the catalogue from the TSM server to the client and scan the first 500 files. If this is greater than one hour, I would still say that your network is at fault. Also, if you are concerned about the file scan times or performance, why don't you try a selective backup of a volume to a dummy nodename. This would negate the whole filescan performance issue and give you true network performance. You can always delete the node and associated data after testing. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 11 October 2005 12:30 To: ADSM-L@VM.MARIST.EDU Subject: [SPAM: 6.500] Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM: cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME
Re: Slow backup performance
It would also be helpful to know what revision levels your server is at. Do you have any 6.5 service packs applied? Any of the TSA or TCP updates? What settings are you using when you load tsafs (look in sys:\etc\sms\tsa.cfg), and are you using ResourceUtilization or any other performance setttings in your dsm.opt? You might also want to check your server's tcp settings to make sure that TCP Delayed Ackknowlegement is off. As has been mentioned already, it's also very possible that there is something wrong in the nic/switch setup. There is no always right answer as to how they should be configged. Some nic/switch combos work best with both hardcoded, some with both on auto, and some with one hardcoded/one on auto. I would also make sure you're running the newest version of the nic driver. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 10/11/2005 6:30:04 AM Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM:cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any
Re: Slow backup performance
Leigh, Many thanks for your clarifications, they helped a lot ! Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: Wednesday, 12 October, 2005 12:03 To: ADSM-L@VM.MARIST.EDU Subject: Re: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance Arnaud, I believe that things changed with Windows2000 and above. This is the note from the manual Windows 2000 and Windows XP provide a larger TCP receive window size when communicating with hosts that also provide this support, known as RFC1323. In these environments, a value greater than 63 may be useful. Refer to Description of Windows 2000 TCP Features, Microsoft knowledge base, article 224829, for details regarding TCP features in Windows 2000. I think that the maximums are as follows TCPBUFFSIZE 512 TCPWINDOWSIZE 2048 TXNBYTELIMIT2097152 I personally use the maximums, but there are always trade-offs. Increasing the buffer size and window size will mean that you require more resource and the overhead of retransmissions when an error is encountered is that much higher. However, with today's network speeds and reliability, coupled with the specification of the latest Intel hardware, I believe the trade-off is worth it. The TXNBYTELIMIT setting to the maximum is a recommendation for better performance when writing straight to LTO1 tape technology (or for that matter any other tape technology that cannot adapt it's tape streaming speed) Using the maximums has worked well for me in the environments I have used them in. This doesn't necessarily mean it's gospel. Also bear in mind that your TSM server has to support the above (ie RFC1323). Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of PAC Brion Arnaud Sent: 12 October 2005 10:00 To: ADSM-L@VM.MARIST.EDU Subject: [SPAM: 10.200] Re: [ADSM-L] Slow backup performance David,(others ?) Like many people here, I'm in a perpetual quest of new tricks for speeding up the backups in our shop, and thought I could try using the TCPWindowsize and TCPBuffsize values you provided in your post for some testing. To my surprise, the backup speed increased dramatically (50 % gain)as when compared to the standard settings I was previously using (TCPW 63, TCPB 31). However I'm wondering how this is possible, as TSM user manual states that the TCPWindowsize max value for Windows based systems is limited to 63 ! Do you (or anybody) have an explanation on how this is possible, and maybe some clarifications about potential side effects of such TCP settings ? Many thanks in advance ... Cheers. Arnaud ** Panalpina Management Ltd., Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 Direct: +41 (61) 226 19 78 e-mail: [EMAIL PROTECTED] ** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of David Vargas Sent: Wednesday, 12 October, 2005 02:29 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance Don't know if this will help, but I thought I'd share what I found when I was having problems with our Exchange TDP backup performance. In the dsm.opt file I added - TCPWindowsize255 TCPBuffSize 127 My backups went from 800K/sec to 30MB/sec and backup time went from 3 days to 2.6 hours for a total of 290GB of data on a 1Gb network. There were a couple of other settings that I changed, but this one is what made it fly. I still think it should be faster... You may need to adjust the numbers for optimal performance in your environment. - David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: Tuesday, October 11, 2005 1:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8
Re: Slow backup performance
Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM:cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any help on this would be appreciated Regards Cameron
Re: Slow backup performance
Do check the network settings and make sure you're up to date on TCP/IP patches for NetWare too as they can make a difference. I've got a 915GB data volume on NW6.5 SP4 with 764,000 files and it only took a 5.3.0 client 41 minutes to inspect it for what that's worth. Andrew Ferris Network Support Analyst iCAPTURE Research Centre University of British Columbia [EMAIL PROTECTED] 10/11/2005 4:30 am Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM:cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any help on this would be appreciated Regards Cameron
Re: Slow backup performance
Don't know if this will help, but I thought I'd share what I found when I was having problems with our Exchange TDP backup performance. In the dsm.opt file I added - TCPWindowsize255 TCPBuffSize 127 My backups went from 800K/sec to 30MB/sec and backup time went from 3 days to 2.6 hours for a total of 290GB of data on a 1Gb network. There were a couple of other settings that I changed, but this one is what made it fly. I still think it should be faster... You may need to adjust the numbers for optimal performance in your environment. - David -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: Tuesday, October 11, 2005 1:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Slow backup performance Guys, Thanks for the info, I'll double check the network settings. Though I have changed the COMMtimeout value from 180secs to 1800 secs as suggested by Richard Simms, which has stopped the session restart error messages and infact the backup seems to be error free. As for the speed, I'm still leaning towards a tsm netware client/ Netware OS issue. I say this because last night I performed a test backup on a 10gig volume (50% full) that hadn't had any changes and it still took 10 hours to scan only 8 odd files. The Client itself hovered around 87-89% utilisation. When this server was netware 6.0 it used to flatline on 100% when performing a backup with the same slow performance. We thought once we'd upgraded to 6.5 with the additional memory required with 6.5, that performance would improve. Unfortunately it hasn't yet. Has anyone experienced this before? Regards Cameron Richard van Denzel [EMAIL PROTECTED] To NL ADSM-L@VM.MARIST.EDU Sent by: ADSM: cc Dist Stor Manager Subject [EMAIL PROTECTED] Re: Slow backup performance .EDU 07/10/2005 07:25 PM Please respond to ADSM: Dist Stor Manager [EMAIL PROTECTED] .EDU Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced
Re: Slow backup performance
If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any help on this would be appreciated Regards Cameron
Re: Slow backup performance
You can also use iperf to check your network performance, see the links: http://dast.nlanr.net/Projects/Iperf/ http://sourceforge.net/projects/iperf Or just do a search on Google for iperf. Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any help on this would be appreciated Regards Cameron
Re: Slow backup performance
Oops, I don't know if there is an iperf for Netware. Do other clients have the same problems from the network the NW server is on? Also you did not specify what kind of network is between server and client(s) (100Mb/1000Mb)? Met vriendelijke groet, With kind regards, Richard van Denzel. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Leigh Reed Sent: vrijdag 7 oktober 2005 10:37 To: ADSM-L@VM.MARIST.EDU Subject: Re: Slow backup performance If you have already checked this and it is very obvious to you, please don't be offended, but it has been a very common occurrence throughout the life of network backups. Have you checked that your NIC is hard coded to 100MB Full Duplex and your switch is also the same. You could also check if your switch is showing any CRC errors. Lastly, try an FTP from client server to TSM server to establish if the network is performing outside of TSM. Again, if I'm teaching you to suck hen produce, apologies. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Cameron Ambrose Sent: 07 October 2005 07:42 To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Hello, We're currently experiencing backup failures/extreme slowness on a Netware 6.5 SP2 server TSM Client 5.3.0 TSM Server 5.2.0 TSA's up to date as recommended by Client Doco Client Logs 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:43:19 ANS1809W Session is lost; initializing session reopen procedure. 10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to 'readRtn'. 10/05/2005 14:44:15 ANS1999E Incremental processing of 'SYS:' stopped. 10/05/2005 14:48:24 ANS1999E Incremental processing of 'INF:' stopped. 10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP connection failure Server Logs 10/05/05 14:39:53 ANR0481W Session 2177 for node SERVERNAME (NetWare) terminated - client did not respond within 180 seconds. 10/05/05 14:41:10 ANR0406I Session 2178 started for node SERVERNAME (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)). Currently we have had a backup running for 24 hours and all that has been transfered is 4.19 gig and 255,000 files scanned. Has anyone else experienced similar issue's Any help on this would be appreciated Regards Cameron
Re: Novell backup performance
Thanks Again! Kurt and David Kurt Beyers wrote: Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESSxx.xx.xx.xx TCPSERVERADDRESSxx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIRSYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000
Re: Novell backup performance
Kurt, I sent this information off to our Netware admin as a possible fix for some issues we're having, and she cannot even locate the setting. I tried a Google search with no luck, and a search of Novell's website didn't return anything to me. I'm not a Netware person at all so I have no idea what to even suggest to her. The systems in question are on Netware 6.5--do you know if the option still exists there, and if so, how I can guide her to it? Thanks much! Kathleen _ Kathleen Hallahan Freddie Mac Distributed Backups 703-450-3317 Timothy Hughes [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 09/16/2005 08:05 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Novell backup performance Thanks Again! Kurt and David Kurt Beyers wrote: Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESSxx.xx.xx.xx TCPSERVERADDRESSxx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIRSYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000
Re: Novell backup performance
Open the server's remote management website (http://servername:8008), click the Set Parameters link, then the Communications link. It will be in there. You can also set it at the console with a command something like set tcp delayed acknowlegement = off. I don't believe the setting stays permanently that way, however (unless you add it to the autoexec.ncf). Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/16/2005 9:27 AM Kurt, I sent this information off to our Netware admin as a possible fix for some issues we're having, and she cannot even locate the setting. I tried a Google search with no luck, and a search of Novell's website didn't return anything to me. I'm not a Netware person at all so I have no idea what to even suggest to her. The systems in question are on Netware 6.5--do you know if the option still exists there, and if so, how I can guide her to it? Thanks much! Kathleen _ Kathleen Hallahan Freddie Mac Distributed Backups 703-450-3317 Timothy Hughes [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 09/16/2005 08:05 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Novell backup performance Thanks Again! Kurt and David Kurt Beyers wrote: Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Novell backup performance
Thanks--I'll pass this along to her! _ Kathleen Hallahan Freddie Mac Distributed Backups 703-450-3317 Troy Frank [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 09/16/2005 11:19 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Novell backup performance Open the server's remote management website (http://servername:8008), click the Set Parameters link, then the Communications link. It will be in there. You can also set it at the console with a command something like set tcp delayed acknowlegement = off. I don't believe the setting stays permanently that way, however (unless you add it to the autoexec.ncf). Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/16/2005 9:27 AM Kurt, I sent this information off to our Netware admin as a possible fix for some issues we're having, and she cannot even locate the setting. I tried a Google search with no luck, and a search of Novell's website didn't return anything to me. I'm not a Netware person at all so I have no idea what to even suggest to her. The systems in question are on Netware 6.5--do you know if the option still exists there, and if so, how I can guide her to it? Thanks much! Kathleen _ Kathleen Hallahan Freddie Mac Distributed Backups 703-450-3317 Timothy Hughes [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 09/16/2005 08:05 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Novell backup performance Thanks Again! Kurt and David Kurt Beyers wrote: Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached
Re: Novell backup performance
Troy answered the question already. These changes are required to make the setting persistent. It can be changed from the CLI with the command 'SET TCP DELAYED ACKNOWLEDGEMENT = OFF ' but would be lost at the next restart of the server Kurt Van: ADSM: Dist Stor Manager namens Troy Frank Verzonden: vr 16/09/2005 17:19 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance Open the server's remote management website (http://servername:8008), click the Set Parameters link, then the Communications link. It will be in there. You can also set it at the console with a command something like set tcp delayed acknowlegement = off. I don't believe the setting stays permanently that way, however (unless you add it to the autoexec.ncf). Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/16/2005 9:27 AM Kurt, I sent this information off to our Netware admin as a possible fix for some issues we're having, and she cannot even locate the setting. I tried a Google search with no luck, and a search of Novell's website didn't return anything to me. I'm not a Netware person at all so I have no idea what to even suggest to her. The systems in question are on Netware 6.5--do you know if the option still exists there, and if so, how I can guide her to it? Thanks much! Kathleen _ Kathleen Hallahan Freddie Mac Distributed Backups 703-450-3317 Timothy Hughes [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 09/16/2005 08:05 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Novell backup performance Thanks Again! Kurt and David Kurt Beyers wrote: Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies
Re: Novell backup performance
When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESSxx.xx.xx.xx TCPSERVERADDRESSxx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIRSYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000
Re: Novell backup performance
Check as well that the parameter TCP_DELAYED_ACKNOWLEDGEMENT=OFF on the Novell server. Switching it from the default 'ON' to OFF improved the backups times with a factor of 3 at a customer. best regards; Kurt Van: ADSM: Dist Stor Manager namens David E Ehresman Verzonden: do 15/09/2005 20:25 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] Novell backup performance When we had a similar problem Servergraph/TSM told us the bottleneck was the network I/O. Upgrading the TSM server to a gig ethernet card and switch dramatically improved the backup times of our Novel clients. [EMAIL PROTECTED] 09/12/05 1:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESSxx.xx.xx.xx TCPSERVERADDRESSxx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIRSYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000
Re: Novell backup performance
As always, review the TSM Performance Tuning Guide. TCPWindowsize is a classic network tuning knob which can have large benefits where right value is chosen - which may require experimentation. Richard Sims
Re: Novell backup performance
Thanks mark Stapleton, Mark wrote: From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Kinder So, to follow up, should RESOURCEUTILIZATION never be set higher than the number of NetWare volumes being backed up? And in the case of a server with a single NetWare volume being backed up, is there a way to establish multiple backup sessions to improve backup times? I've traversed the (seemingly) appropriate documentation and only succeeded in confusing myself. I think I know how to do it in Unix/Linux, but that obviously doesn't help here. Yes, you can multithread the backups in a different way. Create another nodename, a second dsm.opt file using that nodename, and include and exclude lines like (assuming there are two NetWare volumes sys: and vol1: dsm.opt copy1 nodename nwmachine-a exclude *:/.../* include sys:/.../* dsm.opt copy2 nodename nwmachine-b exclude *:/.../* include vol1:/.../* This can be done with three or more volumes as well. You can then easily create two threads for your backups (and your restores) without the use of resourceutilization, multiple mount points, etc. -- Mark Stapleton ([EMAIL PROTECTED]) IBM Certified Advanced Deployment Professional Tivoli Storage Management Solutions 2005 IBM Certified Advanced Technical Expert (CATE) AIX Office 262.521.5627
FW: Novell backup performance
Yes. -- Mark Stapleton ([EMAIL PROTECTED]) IBM Certified Advanced Deployment Professional Tivoli Storage Management Solutions 2005 IBM Certified Advanced Technical Expert (CATE) AIX Office 262.521.5627 -Original Message- From: Kevin Kinder [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 13, 2005 8:42 AM To: Stapleton, Mark Subject: Re: Novell backup performance Mark, Thanks for the reply. One final question. Can I use the same strategy if I have just one volume on the server? For example: DSM1.opt Exclude *:/.../* Include SYS:Apps/.../* DSM2.opt Exclude *:/.../* Include SYS:Data/.../* [EMAIL PROTECTED] 9/12/05 5:40 PM From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Kinder So, to follow up, should RESOURCEUTILIZATION never be set higher than the number of NetWare volumes being backed up? And in the case of a server with a single NetWare volume being backed up, is there a way to establish multiple backup sessions to improve backup times? I've traversed the (seemingly) appropriate documentation and only succeeded in confusing myself. I think I know how to do it in Unix/Linux, but that obviously doesn't help here. Yes, you can multithread the backups in a different way. Create another nodename, a second dsm.opt file using that nodename, and include and exclude lines like (assuming there are two NetWare volumes sys: and vol1: dsm.opt copy1 nodename nwmachine-a exclude *:/.../* include sys:/.../* dsm.opt copy2 nodename nwmachine-b exclude *:/.../* include vol1:/.../* This can be done with three or more volumes as well. You can then easily create two threads for your backups (and your restores) without the use of resourceutilization, multiple mount points, etc. -- Mark Stapleton ([EMAIL PROTECTED]) IBM Certified Advanced Deployment Professional Tivoli Storage Management Solutions 2005 IBM Certified Advanced Technical Expert (CATE) AIX Office 262.521.5627
Novell backup performance
Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESSxx.xx.xx.xx TCPSERVERADDRESSxx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIRSYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000
Re: Novell backup performance
The only thing in the dsm.opt that jumps out at me is the MEMORYEFFICIENTBACKUP. That will definitely slow down backups, and should only be used if the client being backed up really is memory-deficient. The other thing is, RESOURCEUTILIZATION tends to have dimishing returns. I've found that setting to anything more than 4or5 can have negative consequences (especially if the client is memory-deficient). This is especially true since netware boxes tend to not have more than 2 or 3 filesystems on a server. Other reasons could be bad network transfer rate to that server, slow disk subsystem, and/or slow cpu. If possible, I'd make sure the OS is current on all patches, all the device drivers have been updated to newest versions, and double/triple check for network speed/duplex mismatches. I'd check through the dsmsched.log and look at the end-of-backup summaries. There's usually some info in there that'll point to either network/disks/cpu. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/12/2005 12:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Novell backup performance
So, to follow up, should RESOURCEUTILIZATION never be set higher than the number of NetWare volumes being backed up? And in the case of a server with a single NetWare volume being backed up, is there a way to establish multiple backup sessions to improve backup times? I've traversed the (seemingly) appropriate documentation and only succeeded in confusing myself. I think I know how to do it in Unix/Linux, but that obviously doesn't help here. [EMAIL PROTECTED] 9/12/05 2:10:06 PM The only thing in the dsm.opt that jumps out at me is the MEMORYEFFICIENTBACKUP. That will definitely slow down backups, and should only be used if the client being backed up really is memory-deficient. The other thing is, RESOURCEUTILIZATION tends to have dimishing returns. I've found that setting to anything more than 4or5 can have negative consequences (especially if the client is memory-deficient). This is especially true since netware boxes tend to not have more than 2 or 3 filesystems on a server. Other reasons could be bad network transfer rate to that server, slow disk subsystem, and/or slow cpu. If possible, I'd make sure the OS is current on all patches, all the device drivers have been updated to newest versions, and double/triple check for network speed/duplex mismatches. I'd check through the dsmsched.log and look at the end-of-backup summaries. There's usually some info in there that'll point to either network/disks/cpu. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/12/2005 12:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Novell backup performance
Well, the old saying is, never say never... There are exceptions to what I said. Generally you only need to set resourceutilization to = volumes +1. However, what if you're backing up 2 volumes, sys: vol1: , but have 5 tape drives available in your library? To do multi-session restores (using all 5 drives), you'd need to set the MOUNTLIMIT of the client to 5, and set RESOURCEUTILIZATION to at least 5. There are probably other exceptions also, that I just don't know about. In my case, setting resouceutilization to anything 4 tended to make netware backups flaky. Since setting it to 4 was plenty good in my environment, I didn't worry too much about making it work with higher values. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/12/2005 1:21 PM So, to follow up, should RESOURCEUTILIZATION never be set higher than the number of NetWare volumes being backed up? And in the case of a server with a single NetWare volume being backed up, is there a way to establish multiple backup sessions to improve backup times? I've traversed the (seemingly) appropriate documentation and only succeeded in confusing myself. I think I know how to do it in Unix/Linux, but that obviously doesn't help here. [EMAIL PROTECTED] 9/12/05 2:10:06 PM The only thing in the dsm.opt that jumps out at me is the MEMORYEFFICIENTBACKUP. That will definitely slow down backups, and should only be used if the client being backed up really is memory-deficient. The other thing is, RESOURCEUTILIZATION tends to have dimishing returns. I've found that setting to anything more than 4or5 can have negative consequences (especially if the client is memory-deficient). This is especially true since netware boxes tend to not have more than 2 or 3 filesystems on a server. Other reasons could be bad network transfer rate to that server, slow disk subsystem, and/or slow cpu. If possible, I'd make sure the OS is current on all patches, all the device drivers have been updated to newest versions, and double/triple check for network speed/duplex mismatches. I'd check through the dsmsched.log and look at the end-of-backup summaries. There's usually some info in there that'll point to either network/disks/cpu. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/12/2005 12:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me
Re: Novell backup performance
Troy, Thanks again! Troy Frank wrote: The only thing in the dsm.opt that jumps out at me is the MEMORYEFFICIENTBACKUP. That will definitely slow down backups, and should only be used if the client being backed up really is memory-deficient. The other thing is, RESOURCEUTILIZATION tends to have dimishing returns. I've found that setting to anything more than 4or5 can have negative consequences (especially if the client is memory-deficient). This is especially true since netware boxes tend to not have more than 2 or 3 filesystems on a server. Other reasons could be bad network transfer rate to that server, slow disk subsystem, and/or slow cpu. If possible, I'd make sure the OS is current on all patches, all the device drivers have been updated to newest versions, and double/triple check for network speed/duplex mismatches. I'd check through the dsmsched.log and look at the end-of-backup summaries. There's usually some info in there that'll point to either network/disks/cpu. Troy Frank Network Services University of Wisconsin Medical Foundation 608.829.5384 [EMAIL PROTECTED] 9/12/2005 12:43 PM Hello, We have a Novell client who's backup is taking to long We added the Resourceutilization line (8) to the dsm.opt file but it's still taking long. Does anyone have any suggestions on what else I might take a look at or do that would help improve the backup time. The backup starts around midnight. Here is the DSM.OPT file * Setting Options for the TCP/IP Communication Method TCPCLIENTADDRESS xx.xx.xx.xx TCPSERVERADDRESS xx.xx.xx.xx (tsm server ip) TCPPORT 1500 HTTPPORT 1581 errorlogretention 14 schedlogretention 7 Setting Nodename NODENAME X ERRORLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmerror.log SCHEDLOGNAME SYS:Tivoli\TSM\CLIENT\BA\dsmsched.log PASSWORDDIR SYS:Tivoli\TSM\CLIENT\BA\password * EXCLUDE directory:.O=TIVOLI.* * INCLUDE directory:.O=TIVOLI.OU=Development.* DOMAIN ALL-LOCAL -NDS -SYS: -VOL1: * DOMAIN ALL-LOCAL * NetWare general excludes * - * EXCLUDE sys:\system\secaudit.log * EXCLUDE sys:\system\events.log * EXCLUDE sys:\system\system.log * EXCLUDE sys:\system\btrieve.trn * EXCLUDE SYS:\system\tsa\err$hst.* * EXCLUDE sys:\system\tsa\err$log.* * EXCLUDE sys:\system\tsa\skip$log.* * EXCLUDE sys:\system\tsa\tsa$temp.* * EXCLUDE sys:\system\sys$log.err * EXCLUDE sys:\_swap_.mem * EXCLUDE sys:\vol$log.err * EXCLUDE sys:\tts$log.err exclude.dir poa:RESTORE exclude.dir nvol1:RESTORE exclude.dir mta:RESTORE exclude.dir gwia:RESTORE * NetWare Memory management option MEMORYEFFICIENTBACKUP YES * NWEXITNLMP NO managedservices schedule webclient * managedservices schedule schedmode prompted passwordaccess generate * NWUSER server_1_name\user\:password * NWUSER server_2_name\user:password * NWUSER nds_tree_name\user:password NWPWFILE YES * Resouce Utilization specifies the number of back end connections resourceutilization 8 Quiet * VERBOSE Thanks for any help, suggestions or advice! Novell Client 5.2.2 TSM Server 5.3.1.4 AIX RS/6000 Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Novell backup performance
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Kinder So, to follow up, should RESOURCEUTILIZATION never be set higher than the number of NetWare volumes being backed up? And in the case of a server with a single NetWare volume being backed up, is there a way to establish multiple backup sessions to improve backup times? I've traversed the (seemingly) appropriate documentation and only succeeded in confusing myself. I think I know how to do it in Unix/Linux, but that obviously doesn't help here. Yes, you can multithread the backups in a different way. Create another nodename, a second dsm.opt file using that nodename, and include and exclude lines like (assuming there are two NetWare volumes sys: and vol1: dsm.opt copy1 nodename nwmachine-a exclude *:/.../* include sys:/.../* dsm.opt copy2 nodename nwmachine-b exclude *:/.../* include vol1:/.../* This can be done with three or more volumes as well. You can then easily create two threads for your backups (and your restores) without the use of resourceutilization, multiple mount points, etc. -- Mark Stapleton ([EMAIL PROTECTED]) IBM Certified Advanced Deployment Professional Tivoli Storage Management Solutions 2005 IBM Certified Advanced Technical Expert (CATE) AIX Office 262.521.5627
Solaris backup performance degradation, fyi
Something I just ran across: Solaris implementations may be experiencing degraded performance due to debugging having been left on in the TSM device driver, as shipped. Do an IBM site search on IC40302 for the recent APAR and helpful Technote. Richard Sims
Domino backup performance
Hi all, Windows 2000 TSM server 5.2.1 clients 5.1 - 5.2 Domino 6.0.3 soon to be 6.5.4 I'm in a bit of quandry, and I would like another angle on it. The best performance I can squeeze out of our Domino backups is 10MB/s, where normal backups over the same network reach 40 MB/s. A quarter of the speed of normal backups! That's pretty terrible.. We have 2 domino servers which backup their txnlogs every 3 hours during the week. Every weekend on Saturday at 20:00 we run full DB backups of both machines. When the backup starts there is usually only one drive (of the three) available, and since we don't have collocation on in the receiving pool, both sessions take hold of the one tape and take turns to write data to it. In all my years of working with TSM, I've never seen it do this before..but that's besides the point. Later on during the Domino backup, the other drives become free, and I'm pretty sure I saw a session leave the drive-being-shared-by-two-sessions and take it's own. Is anyone able to explain the process TSM goes through when it's deciding what to do in this kind of situation? At any rate, my backup speed of one-quarter the maximum possible speed, is not good and I would like to hear some suggestions about how I can go about reducing the time it takes to backup our 900GB Domino Servers - approx 13 hours. I've been through the 'backup to disk then migrate off to next-pool' motions, but am unable to acquire the disk I'd need. Soon we will have another drive to play with, so I will turn on collocation in the primary Domino tape pool and expect each session to take a drive. BUT, will this make any difference at all? I'm beginning to think that it's the actual size of the databases and txnlogs which is bottlenecking our performance - is the LTO2 drive going to be having a hard time stopping at the end of each txnlog/maildb and starting again for the next one? I'm not totally au fait with the ins and outs of the LTO mechanism, Thank for your suggestions, in advance. Regards, Matthew Aviva plc Registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ Registered in England Number 02468686 www.aviva.com This message and any attachments are confidential. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachment from your system. Also, if you are not the intended recipient you must not copy this message or attachment or disclose the contents to any other person.
Re: Backup Performance
I changed resourcesutilization from 5 back to 3, then backup performance is still very good. I guess the root cause is the tcpwindowsize that HP-UX does not support 128. Thanks for everyone's help!!! 06/25/05 22:40:13 --- SCHEDULEREC STATUS BEGIN 06/25/05 22:40:13 Total number of objects inspected: 376,929 06/25/05 22:40:13 Total number of objects backed up: 897 06/25/05 22:40:13 Total number of objects updated: 0 06/25/05 22:40:13 Total number of objects rebound: 0 06/25/05 22:40:13 Total number of objects deleted: 0 06/25/05 22:40:13 Total number of objects expired: 58 06/25/05 22:40:13 Total number of objects failed: 0 06/25/05 22:40:13 Total number of bytes transferred:181.80 GB 06/25/05 22:40:13 LanFree data bytes: 181.80 GB 06/25/05 22:40:13 Data transfer time:1,150.47 sec 06/25/05 22:40:13 Network data transfer rate:165,701.22 KB/sec 06/25/05 22:40:13 Aggregate data transfer rate: 17,931.77 KB/sec 06/25/05 22:40:13 Objects compressed by:0% 06/25/05 22:40:13 Elapsed processing time: 02:57:11 06/25/05 22:40:13 --- SCHEDULEREC STATUS END 06/25/05 22:40:13 --- SCHEDULEREC OBJECT END 06/25/05 19:45:00 On 6/24/05, Sung Y Lee [EMAIL PROTECTED] wrote: Np.. Glad you were able to resolve the issue. Now that's pretty good backup speed. Sung Y. Lee William [EMAIL PROTECTED] William [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 06/23/2005 10:19 PM Please respond to William [EMAIL PROTECTED] To ADSM-L@VM.MARIST.EDU ADSM-L@VM.MARIST.EDU cc Subject Re: Backup Performance Thanks Sung. I did not find any lanfree related issue. As I mentioned in previous post. I changed 2 parameters: 1. resouceutilization from current 3 to 5 2. tcpwindowsize from current 128 to 63 Now the backup looks very good. 06/23/05 20:26:13 --- SCHEDULEREC STATUS BEGIN 06/23/05 20:26:13 Total number of objects inspected: 376,443 06/23/05 20:26:13 Total number of objects backed up: 749 06/23/05 20:26:13 Total number of objects updated: 0 06/23/05 20:26:13 Total number of objects rebound: 0 06/23/05 20:26:13 Total number of objects deleted: 0 06/23/05 20:26:13 Total number of objects expired:473 06/23/05 20:26:13 Total number of objects failed: 0 06/23/05 20:26:13 Total number of bytes transferred:58.45 GB 06/23/05 20:26:13 LanFree data bytes: 54.56 GB 06/23/05 20:26:13 Data transfer time: 317.63 sec 06/23/05 20:26:13 Network data transfer rate:192,974.37 KB/sec 06/23/05 20:26:13 Aggregate data transfer rate: 23,679.02 KB/sec 06/23/05 20:26:13 Objects compressed by:0% 06/23/05 20:26:13 Elapsed processing time: 00:43:08 On 6/23/05, Sung Y Lee [EMAIL PROTECTED] wrote: Looking at the config, this appears to be lanfree backup client. If the lanfree is broken, then normally TSM client will default to tcpip over lan. If this is so, then I do see TSM storage agent version mismatch. From what I understand, TSM server code and TSM storage agent should match. Is it possible that Lanfree was put in place to over come possible network performance issue? Check the activity log for q actlog begind=-1 s=lanfree. If you are not getting any lanfree for this client or any other clients then I would examine the SAN/Switch configuration. Sometimes I have seen where recycle of storage agent can help with this. Thanks, Sung Y. Lee ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 06/23/2005 02:54:38 PM: I have a performance issue here, any input would be greatly appreciated. TSM Server: 5.3 on AIX 5.3 TSM Client: 5.2.3 on HP-UX B.11.11 TSM Storag Agent 5.2.3 Tape Library: IBM 3584 LTO2 with 12 drives The backup throughput is not good. 06/17/05 06:59:51 Total number of objects inspected: 375,305 06/17/05 06:59:51 Total number of bytes transferred:137.57 GB 06/17/05 06:59:51 Elapsed processing time: 11:16:41 06/18/05 05:15:41 Total number of objects inspected: 375,766 06/18/05 05:15:41 Total number of bytes transferred:120.24 GB 06/18/05 05:15:41 Elapsed processing time: 09:32:08 06/19/05 06:37:26 Total number of objects inspected: 375,840 06/19/05 06:37:26 Total number of bytes transferred:142.51 GB 06/19/05 06:37:26 Elapsed processing time: 10:54:18 06/20/05 06:04:59 Total number of objects inspected: 375,881 06/20/05 06:04:59 Total number of bytes transferred:122.82 GB 06/20/05 06:04:59 Elapsed processing time: 10:21:51 06/21/05 04:58:34 Total number of objects inspected: 376,030 06/21/05 04:58:34 Total number of bytes transferred
Backup Performance
I have a performance issue here, any input would be greatly appreciated. TSM Server: 5.3 on AIX 5.3 TSM Client: 5.2.3 on HP-UX B.11.11 TSM Storag Agent 5.2.3 Tape Library: IBM 3584 LTO2 with 12 drives The backup throughput is not good. 06/17/05 06:59:51 Total number of objects inspected: 375,305 06/17/05 06:59:51 Total number of bytes transferred:137.57 GB 06/17/05 06:59:51 Elapsed processing time: 11:16:41 06/18/05 05:15:41 Total number of objects inspected: 375,766 06/18/05 05:15:41 Total number of bytes transferred:120.24 GB 06/18/05 05:15:41 Elapsed processing time: 09:32:08 06/19/05 06:37:26 Total number of objects inspected: 375,840 06/19/05 06:37:26 Total number of bytes transferred:142.51 GB 06/19/05 06:37:26 Elapsed processing time: 10:54:18 06/20/05 06:04:59 Total number of objects inspected: 375,881 06/20/05 06:04:59 Total number of bytes transferred:122.82 GB 06/20/05 06:04:59 Elapsed processing time: 10:21:51 06/21/05 04:58:34 Total number of objects inspected: 376,030 06/21/05 04:58:34 Total number of bytes transferred:123.83 GB 06/21/05 04:58:34 Elapsed processing time: 09:15:27 06/22/05 06:44:12 Total number of objects inspected: 376,119 06/22/05 06:44:12 Total number of bytes transferred:138.77 GB 06/22/05 06:44:12 Elapsed processing time: 11:01:06 06/23/05 13:45:24 Total number of objects inspected: 376,478 06/23/05 13:45:24 Total number of bytes transferred:291.25 GB 06/23/05 13:45:24 Elapsed processing time: 18:02:19 dsm.sys SErvername tsm COMMmethod TCPip TCPPort1500 HTTPport 1581 WEBPORTS 1582 1583 TCPServeraddress 10.12.1.20 node sm54 passwordaccess generate Schedmode prompted errorlogname /opt/tivoli/tsm/client/ba/bin/dsmerror.log errorlogretention 30 schedlogname /opt/tivoli/tsm/client/ba/bin/dsmsched.log schedlogretention 7 resourceutilization 3 tcpwindowsize 128 tcpbuffsize 64 TCPNodelay Yes largecommbuffers no compression no enablelanfree yes LANFREECommmethod TCPIP LANFREETCPPort 1500 TxnByteLimit 2097152 Except I found one warning message from dsmerror.log, nothing else: 06/17/05 19:43:32 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535 06/18/05 19:43:07 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535 06/19/05 19:43:07 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535 06/20/05 19:43:06 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535 06/21/05 19:43:05 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535 06/22/05 19:43:04 SetSocketOptions(): Warning. The TCP window size defined to ADSM is not supported by your system. It will be to set default size - 65535
