When I'm doing my tests, I'm running the job only on a single client. I noticed some high latencies on the guest using the built in tools when running a test job yesterday, so I'm going to pursue your ideas further. Thanks for your feedback! -- Jack Forester, Jr. Sr. Data Protection Administrator Global Technology Services - AHS Mylan, Inc. 5005 Greenbag Road Morgantown, WV 26501
jack.fores...@mylan.com Phone: +1.304.554.6039 Cell: +1.412.805.5313 "Martin, Jonathan" <jmart...@intersil.com> Sent by: veritas-bu-boun...@mailman.eng.auburn.edu 07/07/2010 02:29 PM To jack.fores...@mylan.com, veritas-bu@mailman.eng.auburn.edu cc Subject Re: [Veritas-bu] Performance tuning Windows 2008 client Are you hitting multiple VMs simultaneously on the same datastore? Is the speed better when you only run one backup at a time? We’ve identified serious performance issues related to requesting to many random I/Os from a raid group as the same time. Backing up multiple VMs and rebooting multiple VMs on the same datastore can cause serious latency spikes. Latency is a bit tricky to track down, but if you open the VC, increase metrics gather to maximum, and select the host the VM is on you can see the latency for any given lun. Anything sustained higher than 20ms is a performance problem waiting to happen. Ideally we shoot for <10ms. Let me know if you need more information or help tracking down these metrics. -Jonathan From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of jack.fores...@mylan.com Sent: Wednesday, July 07, 2010 1:56 PM To: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Performance tuning Windows 2008 client > Do these contain lots and lots of little files? If so, have you considered FlashBackup? The backup is 19GB over 35,000 files. That's pretty typical. It took 17 hours to run. That's not typical. We hope to upgrade to NBU7 later this year and take advantage of the new features for backing up VMs > Do you have enough memory on these guests. We've seen some issues where one of our VMware admins decided to give all of the new guests VERY little memory, forgetting that after he provisioned the guest he was supposed to increase it to a reasonable size. The guests all have 4GB memory. > If you do network exerciser from the guest to the media server, what sort of performance do you get? For example, have you tried something like an FTP or a simple file copy directly from the guest to the media server? I copied a folder containing 215MB of files from the guest to the media server. It took nearly two minutes. A lot slower than I'd have expected for a 1Gb LAN, so to have a basis of comparason, I copied 206MB of files from a 2k8 server running on bare metal to the same media server, and that copy took about half the time. Not stellar performance, but still better than the VM. This is all repeatable. What I've observed is that at the start of the backup, we get good performance, but it slows after 20 minutes or so. The activity also seems to happen in bursts with periods of good speeds followed by periods of poor performance. I'm a UNIX administrator by trade, so troubleshooting Windows issues is a bit foreign to me. -- Jack Forester, Jr. Sr. Data Protection Administrator Global Technology Services - AHS Mylan, Inc. 5005 Greenbag Road Morgantown, WV 26501 jack.fores...@mylan.com Phone: +1.304.554.6039 Cell: +1.412.805.5313 "Ed Wilts" <ewi...@ewilts.org> Sent by: veritas-bu-boun...@mailman.eng.auburn.edu 07/07/2010 10:50 AM To jack.fores...@mylan.com cc veritas-bu@mailman.eng.auburn.edu Subject Re: [Veritas-bu] Performance tuning Windows 2008 client On Wed, Jul 7, 2010 at 9:28 AM, <jack.fores...@mylan.com> wrote: We're observing some significant performance issues with some of our Windows 2008 SP2 clients. Backing up to a DD880 VTL, one client in particular is running at just over 300KB/sec. Others are running 2-3MB/sec. The clients in question are all virtual machines running on VMWare ESX 3.5.0 238493. Our Windows 2008 clients running on bare metal are performing well. Our NetBackup environment is version 6.5.3 on the master/media server and the client. We've tried tuning the net buffer size, but that made no difference. Are there other things we can try? Do these contain lots and lots of little files? If so, have you considered FlashBackup? Do you have enough memory on these guests. We've seen some issues where one of our VMware admins decided to give all of the new guests VERY little memory, forgetting that after he provisioned the guest he was supposed to increase it to a reasonable size. If you do network exerciser from the guest to the media server, what sort of performance do you get? For example, have you tried something like an FTP or a simple file copy directly from the guest to the media server? .../Ed Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE ewi...@ewilts.org Linkedin We have a case open with Symantec, but I thought I'd check here to see if anyone else has run into this issue. -- Jack Forester, Jr. Sr. Data Protection Administrator Global Technology Services - AHS Mylan, Inc. 5005 Greenbag Road Morgantown, WV 26501 jack.fores...@mylan.com Phone: +1.304.554.6039 Cell: +1.412.805.5313 ============================================================================== CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. ============================================================================== _______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu _______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ============================================================================== CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. ============================================================================== _______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ============================================================================== CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. ==============================================================================
_______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
