Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
Shortly speaking - try to disable compression and things can speed up dramatically. The reason: You have to take in the consideration that client-side TSM compression is single-threaded and thus is limited by the (single) processor power. And your Domino server might be busy enough. 566kB/s compressed data rate on low-freq Pentium II/III (even on 4-way server) might be the best you can squeeze from the processor. The best compression rate I've ever seen was about 3 MB/s compressed per processor on 750 MHz RS64-IV (RS/6000 M80). SMP can be utilized only if you increase resource utilization in dsm.opt Zlatko Krastev IT Consultant Barbara Andrews [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 24.09.2002 22:05 Please respond to ADSM: Dist Stor Manager To: [EMAIL PROTECTED] cc: Subject:HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ? ... 5) I have very little knowledge of networking. Could someone comment on the throughput values we are seeing? Our max of 566 Kb/sec is no where near the 100 Mb/sec I have been told our ethernet connection is.
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
Check your network settings on your server. Some network cards do not Auto-negotiate with certain switches. If everything is set to Auto-negotiate, set the network card and the port of the switch it is plugged into to 100 MB/s Full duplex. I was getting the same throughput on some of my TSM servers until I did that and then I started to get 8-9 Megabyte/sec performance. Your performance may vary. Thank You, Matt Martinez Workgroup Support Analyst IDEXX Laboratories, Inc 207-856-0656 [EMAIL PROTECTED] -Original Message- From: Barbara Andrews [mailto:[EMAIL PROTECTED]] Sent: Tuesday, September 24, 2002 3:06 PM To: [EMAIL PROTECTED] Subject:HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ? Up until a few days ago, we were backing up only small amounts of data for several nodes each night. Then an emergency request came in to start nightly incremental backups using the TSM Domino TDP for a Domino mail server that does not do transaction logging. (Note: The Domino support staff estimates implementation of transaction logging 3-4 weeks from now I wish it were sooner!) All of a sudden we went from a total backup of no more than 11GB per night to more than double that, with almost 20-23 GB per night coming from that one new node alone. This node's data is sent over a 100 MB ethernet connection. Even with client compression turned on, the best throughput rate we've seen is 566 Kb/sec., and that was when the backup was run on a Sunday when there is very little activity on all systems involved (client, server, network). When other backups are running during the night, the best we've seen is 368.72 Kb/sec. This results in the backup for this one node taking at least 17 hours to run to completion. Before the addition of this new node, I had all my processes (expiration, reclamation, migration, backup storage pool) running separately and not at the same time as client backup sessions. Now we no longer have enough time in the day to do this. I need to change the way my processes run, but I have several questions I need answered before I do this. If anyone can help me with the following questions, I would appreciate it: 1) What are the implications of the process generated when I issue backup stg backuppool offsite-pool running at the same time that a node is currently in session backing up to the backuppool storage pool? 1a) Will the backup stg process finish during the node's session? 2) Do we need to be in a state where there are no active processes and/or backup/restore sessions running while the TSM database backup is running? 2a) What kind of state would the TSM database be in if it was restored from a backup that had other processes or backup/restore sessions running at the time it was taken? 3) Is there any problem with running an expire inventory while any of the following are running?: - a node backup session - a node restore session - migration - reclamation 4) Is there a problem running migration and reclamation at the same time if they both access the same pool? I would have thought the only problem might be a wait for a tape while one process is using it, but that this probably would only occur occasionally. I have a reclamation process running that caused a message Removable volume 30050 is required for space reclamation. A migration process is running which requires writing to the same tape pool that 300050 is in, however 300050 displays as last being written to the day before. 5) I have very little knowledge of networking. Could someone comment on the throughput values we are seeing? Our max of 566 Kb/sec is no where near the 100 Mb/sec I have been told our ethernet connection is.
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Barbara Andrews Even with client compression turned on, the best throughput rate we've seen is 566 Kb/sec., and that was when the backup was run on a Sunday when there is very little activity on all systems involved (client, server, network). When other backups are running during the night, the best we've seen is 368.72 Kb/sec. This results in the backup for this one node taking at least 17 hours to run to completion. I've often said that TSM is one of the finest problem finders in the tech industry today. It will show you bus problems, adapter bottlenecks, network missettings, and SAN misconfigurations faster than anything else in the market. Your network problems doubtless existed prior to the change in your backup schema; you just didn't see it because you didn't stress the network with it. I would say that you need to check the speed and duplex settings for *all* network hardware in the line of the problem--the Notes server's NIC, the switch port that it's plugged into, the switch port that the TSM server is plugged into, and the NIC for the TSM server. They should (in most cases) all be set to 100 full duplex for a 100mb Ethernet network. -- Mark Stapleton ([EMAIL PROTECTED]) Certified TSM consultant Certified AIX system engineer MCSE
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
I've already seen one reply to check the NIC settings - those will get you every time. An easy check that can really pay off. Another thing to check is to see if the client is CPU or I/O bound. If your CPU utilization is at 100% (or near it), client side compression can slow you down. If the DASD is I/O bound, you'll need to address that area to improve performance. To answer the questions: 1. The source storagepool is examined at the beginning of the backup stgpool process, so any data that comes in after the process starts is not going to be copied. That is as dangerous as it sounds - you may not have all the data you need offsite. 1a. Yes, it will finish, since it doesn't know about the data the client has been sending. 2. Sort of the same as question 1 above, but the database backup will pick up changes as it goes (after the end of the backup process). However, if you have a process running that doesn't complete/commit it's changes, they may not get backed up. It's a crapshoot, and can be dangerous, so I'd recommend you avoid it if you can. 2a. The backup will have the image as it was when the backup was taken. If there is data being copied from tape to tape, and the old, now scratch, tape is reused between the TSM database backup and the restore, the data can be overwritten. See reuse delay for ways to close this exposure. 3. Expire Inventory can run parallel to anything else. With reclamation/migration, you may expire a file you just spent time reclaiming, so if you can avoid it, you should, but then there are only 24 hours in the day... 4. No, you can run migration and reclamation at the same time, and you're right - you may fight for tapes/drives. With the message you are getting - does the tape get mounted? I think that message just means that TSM wants the tape - it's just an FYI. 5. Yes, that is VERY slow. Your actual performance may vary depending on a lot of factors, but that is definitely slow. Nick Cassimatis [EMAIL PROTECTED] Today is the tomorrow of yesterday.
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
Around here, we call TSM backups our Computer/Server/Node/Client Stress Tester. I totally agree Mark, we currently used TSM backup stress tester to test our Gigabyte interface and it did show pros and cons of the setup of the GB network. Thank You, Bill Rosette Data Center/IS/Papa Johns International WWJD |-+ | | Mark Stapleton | | | stapleto@BERBEE.| | | COM | | | Sent by: ADSM: | | | Dist Stor| | | Manager | | | [EMAIL PROTECTED]| | | .EDU| | || | || | | 09/24/2002 03:45 | | | PM | | | Please respond to| | | ADSM: Dist Stor | | | Manager | | || |-+ --| | | | To: [EMAIL PROTECTED] | | cc: | | Subject: Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup | |sessions ? | --| From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Barbara Andrews Even with client compression turned on, the best throughput rate we've seen is 566 Kb/sec., and that was when the backup was run on a Sunday when there is very little activity on all systems involved (client, server, network). When other backups are running during the night, the best we've seen is 368.72 Kb/sec. This results in the backup for this one node taking at least 17 hours to run to completion. I've often said that TSM is one of the finest problem finders in the tech industry today. It will show you bus problems, adapter bottlenecks, network missettings, and SAN misconfigurations faster than anything else in the market. Your network problems doubtless existed prior to the change in your backup schema; you just didn't see it because you didn't stress the network with it. I would say that you need to check the speed and duplex settings for *all* network hardware in the line of the problem--the Notes server's NIC, the switch port that it's plugged into, the switch port that the TSM server is plugged into, and the NIC for the TSM server. They should (in most cases) all be set to 100 full duplex for a 100mb Ethernet network. -- Mark Stapleton ([EMAIL PROTECTED]) Certified TSM consultant Certified AIX system engineer MCSE
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
Another thing to consider is that each time an email comes in to the Domino server, some user file is getting changed. If a file is changed while TSM is backing it up, it starts over (depending on your settings). This bit us when we started backing up our Domino server. Since you are using the TDP (which we weren't at first) this might not be an issue, but it's worth looking into. Nancy Reeves Technical Support, Wichita State University [EMAIL PROTECTED] 316-978-3860
Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ?
Mark; run the command netstat -v on the backup interface and take a look at - Drop packets - Recieve errors - Transmit errors - CRC errors - DMA overruns - The recieve and transmit queue size. - Any collisions (single or otherwise) Like others said, Hard code all the devices involved to 100 MB (no auto_negotiate). I believe this change will require a reboot of the system. - Try and get used to the different switches of the netstat command - Also, run a traceroute (tracert in the case of NT) from both sides and see if you are using the proper route. If not you will need to change routing table. Hope this helps. Regards; Hussein M. Abdirahman AIX administrator IrwinToy LTD. Toronto - Canada -Original Message- From: Mark Stapleton [mailto:[EMAIL PROTECTED]] Sent: Tuesday, September 24, 2002 3:45 PM To: [EMAIL PROTECTED] Subject: Re: HELP!... What processes can/can't run simultaneously and what throttles throughput on backup sessions ? From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED]]On Behalf Of Barbara Andrews Even with client compression turned on, the best throughput rate we've seen is 566 Kb/sec., and that was when the backup was run on a Sunday when there is very little activity on all systems involved (client, server, network). When other backups are running during the night, the best we've seen is 368.72 Kb/sec. This results in the backup for this one node taking at least 17 hours to run to completion. I've often said that TSM is one of the finest problem finders in the tech industry today. It will show you bus problems, adapter bottlenecks, network missettings, and SAN misconfigurations faster than anything else in the market. Your network problems doubtless existed prior to the change in your backup schema; you just didn't see it because you didn't stress the network with it. I would say that you need to check the speed and duplex settings for *all* network hardware in the line of the problem--the Notes server's NIC, the switch port that it's plugged into, the switch port that the TSM server is plugged into, and the NIC for the TSM server. They should (in most cases) all be set to 100 full duplex for a 100mb Ethernet network. -- Mark Stapleton ([EMAIL PROTECTED]) Certified TSM consultant Certified AIX system engineer MCSE
