HELP!!! EMERGENCY!!!
I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
How are your disk pools? Full or still have room? What is your tape status? Do you have other communications problems? (Slow printing, slow terminals, slow FTP...) -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED] Sent: Friday, 04 April 2003 12:13 PM To: [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- Information from the mail header --- Sender: ADSM: Dist Stor Manager [EMAIL PROTECTED] Poster: Joni Moyer [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- - I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
Disk pools aren't full. Tape drives are available. There are plenty of tapes. The db is 88% full. Recovery log is 7% utilized. I did have the following message, but it said the server would continue: ANR0390W A server database deadlock situation has been encountered: lock request for transaction 0:6.517663675 will be denied to resolve the deadlock. ANRD DFUTIL(734): Lock acquisition (isLock) failed for bitfile root 0. ANR2183W AFMOVE3551: Transaction 0:6.517663675 was aborted. Any suggestions!!! I really need help! Should I contact support? If I do it'll take 3 hours to get a response... Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338 Kai Hintze [EMAIL PROTECTED] To: [EMAIL PROTECTED] TSONS.COM cc: Sent by: ADSM: Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] .EDU 04/04/2003 02:19 PM Please respond to ADSM: Dist Stor Manager How are your disk pools? Full or still have room? What is your tape status? Do you have other communications problems? (Slow printing, slow terminals, slow FTP...) -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED] Sent: Friday, 04 April 2003 12:13 PM To: [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- Information from the mail header --- Sender: ADSM: Dist Stor Manager [EMAIL PROTECTED] Poster: Joni Moyer [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- - I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
Joni, I had this happen on TSM 4.2.1.9 a long time ago, my only option was to bounce the TSM server. I did not have as many clients active only 10 or so. Sorry but that was my quickest fix at that time. No other way to free these sessions up that I know of. - Brian Brian L. Nick Systems Technician - Storage Solutions The Phoenix Companies Inc. 100 Bright Meadow Blvd Enfield CT. 06082-1900 E-MAIL: [EMAIL PROTECTED] PHONE: (860)403-2281 Joni Moyer [EMAIL PROTECTED]To: [EMAIL PROTECTED] HMARK.COM cc: Sent by: ADSM:Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] ST.EDU 04/04/03 02:22 PM Please respond to ADSM: Dist Stor Manager Disk pools aren't full. Tape drives are available. There are plenty of tapes. The db is 88% full. Recovery log is 7% utilized. I did have the following message, but it said the server would continue: ANR0390W A server database deadlock situation has been encountered: lock request for transaction 0:6.517663675 will be denied to resolve the deadlock. ANRD DFUTIL(734): Lock acquisition (isLock) failed for bitfile root 0. ANR2183W AFMOVE3551: Transaction 0:6.517663675 was aborted. Any suggestions!!! I really need help! Should I contact support? If I do it'll take 3 hours to get a response... Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338 Kai Hintze [EMAIL PROTECTED] To: [EMAIL PROTECTED] TSONS.COM cc: Sent by: ADSM: Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] .EDU 04/04/2003 02:19 PM Please respond to ADSM: Dist Stor Manager How are your disk pools? Full or still have room? What is your tape status? Do you have other communications problems? (Slow printing, slow terminals, slow FTP...) -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED] Sent: Friday, 04 April 2003 12:13 PM To: [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- Information from the mail header --- Sender: ADSM: Dist Stor Manager [EMAIL PROTECTED] Poster: Joni Moyer [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- - I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
AW: HELP!!! EMERGENCY!!!
quick ideas: Maybe your primary storage pool is full and the sessions are in MediaWait state? look at Q ACTL on server and dsmsched.log and dsmerror.log on clients Juraj -Ursprüngliche Nachricht- Von: Joni Moyer [mailto:[EMAIL PROTECTED] Gesendet: Freitag, 04. April 2003 21:13 An: [EMAIL PROTECTED] Betreff: HELP!!! EMERGENCY!!! I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
I have seen a deadlock situation when trying to delete objects on the same FS. Are any of your sessions trying to backup the same fs names/files? Have you tried killing off the sessions? Is 100 client sessions at one time common? Are there any helpful messages in the individual client's dsmsched log? Have you have any network interruptions lately? Sometimes the big hammer approach is quickest (killing the server) but doesn't always point to the cause. Good luck, Mary Vraspir -Original Message- From: Salak Juraj [mailto:[EMAIL PROTECTED] Sent: Friday, April 04, 2003 1:22 PM To: [EMAIL PROTECTED] Subject: AW: HELP!!! EMERGENCY!!! quick ideas: Maybe your primary storage pool is full and the sessions are in MediaWait state? look at Q ACTL on server and dsmsched.log and dsmerror.log on clients Juraj -Ursprüngliche Nachricht- Von: Joni Moyer [mailto:[EMAIL PROTECTED] Gesendet: Freitag, 04. April 2003 21:13 An: [EMAIL PROTECTED] Betreff: HELP!!! EMERGENCY!!! I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
If you want support to look at the problem, take a svc dump. Then cycle your tsm server to clear the deadlock. David [EMAIL PROTECTED] 04/04/03 02:22PM Disk pools aren't full. Tape drives are available. There are plenty of tapes. The db is 88% full. Recovery log is 7% utilized. I did have the following message, but it said the server would continue: ANR0390W A server database deadlock situation has been encountered: lock request for transaction 0:6.517663675 will be denied to resolve the deadlock. ANRD DFUTIL(734): Lock acquisition (isLock) failed for bitfile root 0. ANR2183W AFMOVE3551: Transaction 0:6.517663675 was aborted. Any suggestions!!! I really need help! Should I contact support? If I do it'll take 3 hours to get a response... Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338 Kai Hintze [EMAIL PROTECTED] To: [EMAIL PROTECTED] TSONS.COM cc: Sent by: ADSM: Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] .EDU 04/04/2003 02:19 PM Please respond to ADSM: Dist Stor Manager How are your disk pools? Full or still have room? What is your tape status? Do you have other communications problems? (Slow printing, slow terminals, slow FTP...) -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED] Sent: Friday, 04 April 2003 12:13 PM To: [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- Information from the mail header --- Sender: ADSM: Dist Stor Manager [EMAIL PROTECTED] Poster: Joni Moyer [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- - I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
Joni, I had left you a v-mail, thought you would like a quick response. We also run OS/390 r2.10, and have several TSM servers. When we were still on 4.1 code we had this happen frequently. IBM was never able to find a cause or solution. However, when we upgraded to 4.2 and now 5.1 the problem went away. The only thing we could do when this happens, was as Brian said, bounce the server. Some of the symptoms you should see, processes will 'hang' also, they will show 'current file' as a size of 0 bytes, some sessions may also show this. The TSM server will continue to accept sessions and start new processes, it will not move any data however, and the sessions and processes will hang around until you stop/start the server. You will want to upgrade your code ASAP to avoid this. Sorry for no better answer, but this is the best we could ever come up with. Ryan Miller Principal Financial Group Tivoli Certified Consultant Tivoli Storage Manager v4.1 -Original Message- From: Brian L. Nick [mailto:[EMAIL PROTECTED] Sent: Friday, April 04, 2003 1:25 PM To: [EMAIL PROTECTED] Subject: Re: HELP!!! EMERGENCY!!! Joni, I had this happen on TSM 4.2.1.9 a long time ago, my only option was to bounce the TSM server. I did not have as many clients active only 10 or so. Sorry but that was my quickest fix at that time. No other way to free these sessions up that I know of. - Brian Brian L. Nick Systems Technician - Storage Solutions The Phoenix Companies Inc. 100 Bright Meadow Blvd Enfield CT. 06082-1900 E-MAIL: [EMAIL PROTECTED] PHONE: (860)403-2281 Joni Moyer [EMAIL PROTECTED]To: [EMAIL PROTECTED] HMARK.COM cc: Sent by: ADSM:Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] ST.EDU 04/04/03 02:22 PM Please respond to ADSM: Dist Stor Manager Disk pools aren't full. Tape drives are available. There are plenty of tapes. The db is 88% full. Recovery log is 7% utilized. I did have the following message, but it said the server would continue: ANR0390W A server database deadlock situation has been encountered: lock request for transaction 0:6.517663675 will be denied to resolve the deadlock. ANRD DFUTIL(734): Lock acquisition (isLock) failed for bitfile root 0. ANR2183W AFMOVE3551: Transaction 0:6.517663675 was aborted. Any suggestions!!! I really need help! Should I contact support? If I do it'll take 3 hours to get a response... Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338 Kai Hintze [EMAIL PROTECTED] To: [EMAIL PROTECTED] TSONS.COM cc: Sent by: ADSM: Subject: Re: HELP!!! EMERGENCY!!! Dist Stor Manager [EMAIL PROTECTED] .EDU 04/04/2003 02:19 PM Please respond to ADSM: Dist Stor Manager How are your disk pools? Full or still have room? What is your tape status? Do you have other communications problems? (Slow printing, slow terminals, slow FTP...) -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED] Sent: Friday, 04 April 2003 12:13 PM To: [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- Information from the mail header --- Sender: ADSM: Dist Stor Manager [EMAIL PROTECTED] Poster: Joni Moyer [EMAIL PROTECTED] Subject: HELP!!! EMERGENCY!!! -- - I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: HELP!!! EMERGENCY!!!
Joni If you have any processes using server-to-server communications running concurrently with the backups, try cancelling the processes. This in itself will probably not free things up but if you then try cancel all the sessions created by the server-to-server communications, things may resume. You may have to do this on both source and destination servers. Worked for me in the past on 3.7 and 4.2. Regards Neil Schofield Visit our web site to find out about our award winning Cool Schools community campaign at http://www.yorkshirewater.com/yorkshirewater/schools.html The information in this e-mail is confidential and may also be legally privileged. The contents are intended for recipient only and are subject to the legal notice available at http://www.keldagroup.com/email.htm Yorkshire Water Services Limited Registered Office Western House Halifax Road Bradford BD6 2SZ Registered in England and Wales No 2366682
Re: HELP!!! EMERGENCY!!!
Joni, I have this happen with TSM 4.1.5 on AIX. You may or may not like the answer. Stop and restart the TSM server. This is the only way to clear up the sessions. Sias Get your own 800 number Voicemail, fax, email, and a lot more http://www.ureach.com/reg/tag On, Joni Moyer ([EMAIL PROTECTED]) wrote: I am running TSM 4.1.5 on OS/390 2.10 I am getting calls from clients about slow backups, restores,etc. on TSM. I went out and looked and there are 100 sessions going for different clients, but they aren't ending. Some are in the IDLEW status, but a majority of them are in the run status. What should I look into? Thanks! Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: recovery log filling up rapidly: Please help: EMERGENCY!!!! !
I've seen the log fill in Normal if you have a client running that's not sending commits often enough. Clients with NICs set to Autonegotiate, then ending up at 100/Half Duplex (or 10/Half duplex, a couple of times) will pin the log due to not getting to a commit point. Add in expiration on an active TSM server, and your log can fill up fast. Also, I believe the log doesn't commit during a database backup operation, so if that takes an extended period of time (wait for mount point, then large database to send), you can get the log pretty full. Nick Cassimatis Today is the tomorrow of yesterday.
Re: recovery log filling up rapidly: Please help: EMERGENCY!!!! !
One possibility, which I have seen at clients, is that the database is on very slow disk. Orville L. Lantto Datatrend Technologies, Inc. (http://www.datatrend.com) IBM Premier Business Partner 121 Cheshire Lane, Suite 700 Minnetonka, MN 55305 Email: [EMAIL PROTECTED] CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. Raghu S [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 01/16/2003 06:52 AM Please respond to ADSM: Dist Stor Manager To: [EMAIL PROTECTED] cc: Subject:Re: recovery log filling up rapidly: Please help: EMERGENCY ! Hi Joni Its strange to see recovery log filling up to 5GB level in NORMAL mode ( you mentioned that in ur previous mail). In NORMALMODE recovery log doesn't keep transactions, it deletes transaction as soon as it committed to database.Thats why only Point-in-Retsore ( till the last consistent database backup) possible with recovery log in NORMAL mode. You can't use database backup trigger with recovery log in NORMAL mode. with TSM 5.x Recovery Log max size is 13GB. I can't figure out the reason for this behaviour.Let me know if you could find the reason. Thanks regds Raghu -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 15, 2003 7:02 AM To: [EMAIL PROTECTED] Subject: recovery log filling up rapidly: Please help: EMERGENCY! My recovery log is filling up at a rapid rate. I am running TSM at 4.1.3 on the mainframe. It is 4.6 GB and my DB is 48 GB with 46 GB in use. I am not running expiration of the inventory yet due to a previous deletion of a nodes data, could that be the cause? I noticed that the log is reaching about 95% full before I run another full backup. I usually run a full DB backup every day at 4:30. What can I do? I can't increase the recovery log because I guess on this version it can only go to 5 GB? What version is this no longer true for? I really need help and I can't find anyone from IBM to return my calls. My previous problem with the recovery log is no longer true. It does now reset back to 0% after a full backup, but I am running out of space. I do have the log fully extended and with the 5 GB limit I am stuck. Also, I can't set up a DB backup trigger because I am on the mainframe and that feature of TSM is not available. Thank you in advance for any help you can give me Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
recovery log filling up rapidly: Please help: EMERGENCY!!!!!
My recovery log is filling up at a rapid rate. I am running TSM at 4.1.3 on the mainframe. It is 4.6 GB and my DB is 48 GB with 46 GB in use. I am not running expiration of the inventory yet due to a previous deletion of a nodes data, could that be the cause? I noticed that the log is reaching about 95% full before I run another full backup. I usually run a full DB backup every day at 4:30. What can I do? I can't increase the recovery log because I guess on this version it can only go to 5 GB? What version is this no longer true for? I really need help and I can't find anyone from IBM to return my calls. My previous problem with the recovery log is no longer true. It does now reset back to 0% after a full backup, but I am running out of space. I do have the log fully extended and with the 5 GB limit I am stuck. Also, I can't set up a DB backup trigger because I am on the mainframe and that feature of TSM is not available. Thank you in advance for any help you can give me Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: recovery log filling up rapidly: Please help: EMERGENCY!!!!!
no when you do not expiration you db grows but the log may grow with un-backed up databasr. try to backup your database more frequently or increase your db. or you may change your log type. Burak Demircan CEO / ITT MERCEDES-BENZ TURK A.S. [EMAIL PROTECTED] tel:+90 212 482 35 00 (4676) fax :+90 212 481 11 54 [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 15.01.2003 14:07 Please respond to ADSM-L To: [EMAIL PROTECTED] cc: Subject: recovery log filling up rapidly: Please help: EMERGENCY! My recovery log is filling up at a rapid rate. I am running TSM at 4.1.3 on the mainframe. It is 4.6 GB and my DB is 48 GB with 46 GB in use. I am not running expiration of the inventory yet due to a previous deletion of a nodes data, could that be the cause? I noticed that the log is reaching about 95% full before I run another full backup. I usually run a full DB backup every day at 4:30. What can I do? I can't increase the recovery log because I guess on this version it can only go to 5 GB? What version is this no longer true for? I really need help and I can't find anyone from IBM to return my calls. My previous problem with the recovery log is no longer true. It does now reset back to 0% after a full backup, but I am running out of space. I do have the log fully extended and with the 5 GB limit I am stuck. Also, I can't set up a DB backup trigger because I am on the mainframe and that feature of TSM is not available. Thank you in advance for any help you can give me Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
Re: recovery log filling up rapidly: Please help: EMERGENCY!!!! !
Hi Joni, I have been there. I am on the Mainframe also. The DB backup trigger does work. I started to do an incremental DB backup just before the backup would start in the evening. It helped me out a lot. The version you are on 4.1.x has the log limit of 5GB. Version 4.2 and above it is 11GB. IF YOUR LOG FILLs YOU LOCK UP. Then you will have to extend your log, make sure NOTHING ELSE BY THE DB BACKUP RUNS, then let everything start up. I personally would stop absolutely everything but the DB backup right now! Do a CANCEL PROC on that deletion you have running. Stop the backups if you can, stop the migrations or any other services and pray that you have enough log space left to do the DB backup. MAtt -Original Message- From: Joni Moyer [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 15, 2003 7:02 AM To: [EMAIL PROTECTED] Subject:recovery log filling up rapidly: Please help: EMERGENCY! My recovery log is filling up at a rapid rate. I am running TSM at 4.1.3 on the mainframe. It is 4.6 GB and my DB is 48 GB with 46 GB in use. I am not running expiration of the inventory yet due to a previous deletion of a nodes data, could that be the cause? I noticed that the log is reaching about 95% full before I run another full backup. I usually run a full DB backup every day at 4:30. What can I do? I can't increase the recovery log because I guess on this version it can only go to 5 GB? What version is this no longer true for? I really need help and I can't find anyone from IBM to return my calls. My previous problem with the recovery log is no longer true. It does now reset back to 0% after a full backup, but I am running out of space. I do have the log fully extended and with the 5 GB limit I am stuck. Also, I can't set up a DB backup trigger because I am on the mainframe and that feature of TSM is not available. Thank you in advance for any help you can give me Joni Moyer Systems Programmer [EMAIL PROTECTED] (717)975-8338
