TSM server crash
Hi All, OS: W2K3 TSM Version 5.4 My recolvery log ran full and TSM is down at the moment. I only managed to format a new logvol and dbvol using dsmfmt. This is where i am stuck!! Thanks in advance.. +-- |This was sent by jerome.swa...@computacenter.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +--
Re: TSM server crash
You can try to put in script commands DEFINE LOGVOLUME and EXTEND LOG for formatted log volume and run: DSMSERV runfile script Good luck! Grigori G. Solonovitch Senior Technical Architect Information Technology Bank of Kuwait and Middle East http://www.bkme.com Phone: (+965) 2231-2274 Mobile: (+965) 99798073 E-Mail: g.solonovi...@bkme.com Please consider the environment before printing this Email -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of matrixdot Sent: Monday, December 28, 2009 3:08 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] TSM server crash Hi All, OS: W2K3 TSM Version 5.4 My recolvery log ran full and TSM is down at the moment. I only managed to format a new logvol and dbvol using dsmfmt. This is where i am stuck!! Thanks in advance.. +-- |This was sent by jerome.swa...@computacenter.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- Please consider the environment before printing this Email. This email message and any attachments transmitted with it may contain confidential and proprietary information, intended only for the named recipient(s). If you have received this message in error, or if you are not the named recipient(s), please delete this email after notifying the sender immediately. BKME cannot guarantee the integrity of this communication and accepts no liability for any damage caused by this email or its attachments due to viruses, any other defects, interception or unauthorized modification. The information, views, opinions and comments of this message are those of the individual and not necessarily endorsed by BKME.
Re: TSM server crash
Hi, See Server Utilities in the Admin Reference Guide, looking for the dsmserv extend log part... -- Best regards / Cordialement / مع تحياتي Erwann SIMON Le 28/12/2009 13:08, matrixdot a écrit : Hi All, OS: W2K3 TSM Version 5.4 My recolvery log ran full and TSM is down at the moment. I only managed to format a new logvol and dbvol using dsmfmt. This is where i am stuck!! Thanks in advance.. +-- |This was sent by jerome.swa...@computacenter.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +--
Re: TSM server crash
Thanks found it!! -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Erwann Simon Sent: Monday, December 28, 2009 4:55 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] TSM server crash Hi, See Server Utilities in the Admin Reference Guide, looking for the dsmserv extend log part... -- Best regards / Cordialement / مع تحياتي Erwann SIMON Le 28/12/2009 13:08, matrixdot a écrit : Hi All, OS: W2K3 TSM Version 5.4 My recolvery log ran full and TSM is down at the moment. I only managed to format a new logvol and dbvol using dsmfmt. This is where i am stuck!! Thanks in advance.. +-- |This was sent by jerome.swa...@computacenter.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- ** COMPUTACENTER PLC is registered in England and Wales with the registered number 03110569. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW COMPUTACENTER (UK) Limited is registered in England and Wales with the registered number 01584718. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW COMPUTACENTER (Mid-Market) Limited is registered in England and Wales with the registered number 3434654. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW COMPUTACENTER (FMS) Limited is registered in England and Wales with the registered number 3798091. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW The contents of this email are intended for the named addressee only. It contains information which may be confidential and which may also be privileged. Unless you are the named addressee (or authorised to receive mail for the addressee) you may not copy or use it, or disclose it to anyone else. If you receive it in error please notify us immediately and then destroy it. Computacenter information is available from: http://www.computacenter.com **
TSM server crash
Hello, MY TSM server has crashed.This is my library Manager also for 8 Client TSM servers.Can anyone give me the step by step procedure how can i recover this server to original condition. -- Best Regards Ashish Sharma ST Microelectronics Ltd. 919717003853
Re: TSM server crash
When posting to a technical mailing list such as this, include details in your postings. The TSM Problem Determination Guide manual a solid starting point for pursuing problems, in conjunction with the Administration Guide manual. Richard Sims
Re: TSM server crash
If your server is down, I strongly recommend you open a severity 1 call to Tivoli Support. On Fri, Jul 31, 2009 at 11:45 AM, ashish sharma ashishsharm...@gmail.comwrote: Hello, MY TSM server has crashed.This is my library Manager also for 8 Client TSM servers.Can anyone give me the step by step procedure how can i recover this server to original condition. -- Best Regards Ashish Sharma ST Microelectronics Ltd. 919717003853
Re: TSM server crash
Have you tried to restart TSM? Does it fail on startup? What errors? TSM Version? OS? Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of ashish sharma Sent: Friday, July 31, 2009 10:45 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] TSM server crash Hello, MY TSM server has crashed.This is my library Manager also for 8 Client TSM servers.Can anyone give me the step by step procedure how can i recover this server to original condition. -- Best Regards Ashish Sharma ST Microelectronics Ltd. 919717003853 This e-mail (including any attachments) is confidential and may be legally privileged. If you are not an intended recipient or an authorized representative of an intended recipient, you are prohibited from using, copying or distributing the information in this e-mail or its attachments. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message and any attachments. Thank you.
TSM server crash
I wonder if anyone else has had a similar issue... We have a TSM server (5.2) running on a Windows 2000 server. This runs on an IBM x346 hardware. This server also has IBM Storage Management software on it which has been happily running on there for around a year. Today I also installed IBM Director on this server. To complete the installation we have to reboot the server. Upon reboot the server now crashes just as it finishes starting up Windows with unexpected_kernel_mode_trap on the blue screen of death... If we boot up the server in safe mode it boots up OK however we cannot get any TSM services to run, and it won't let me un-install IBM Director. We took all the hard disks out of this x346 and put them in another x346 to boot up but the same error occurred. As we can see the TSM data storage disks (the D and the E drive) when we boot up in safe mode we would rather not blow everything away by re-installing Windows and having to recreate TSM from scratch and restore all the data from the tapes. I realise that this is mainly a Windows issue now, but if anyone else was in a similar situation with their TSM server I would like to know if you were able to get around it. Thanks Paul Paul Dudley ANL IT Operations Dept. ANL Container Line [EMAIL PROTECTED] ANL DISCLAIMER This e-mail and any file attached is confidential, and intended solely to the named addressees. Any unauthorised dissemination or use is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by return e-mail from your system. Please do not copy, use or make reference to it for any purpose, or disclose its contents to any person.
Re: TSM server crash
On Jul 26, 2006, at 7:06 AM, Paul Dudley wrote: ...Upon reboot the server now crashes just as it finishes starting up Windows with unexpected_kernel_mode_trap ... Search on that condition for possible causes, one of which is summarized in very recent IBM Technote 1241015. Richard Simshttp://people.bu.edu/rbs
Re: [SPAM: 4.300] [ADSM-L] TSM server crash
Paul A search of the IBM website with the error, returns the following Technote http://www-1.ibm.com/support/docview.wss?uid=swg21241015 It's not strictly related to IBM Director, but you never know. I have had problems with Anti-Virus s/w kernel stack settings in the past, however, again not strictly the scenario that you have. You may have to disable the anti-virus service in Recovery Console, which will allow you to boot and then you can change the registry setting. http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2002071208532048?OpenDocumentExpandSection=2Src=w Hope this is of some use. Leigh -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Paul Dudley Sent: 26 July 2006 12:07 To: ADSM-L@VM.MARIST.EDU Subject: [SPAM: 4.300] [ADSM-L] TSM server crash I wonder if anyone else has had a similar issue... We have a TSM server (5.2) running on a Windows 2000 server. This runs on an IBM x346 hardware. This server also has IBM Storage Management software on it which has been happily running on there for around a year. Today I also installed IBM Director on this server. To complete the installation we have to reboot the server. Upon reboot the server now crashes just as it finishes starting up Windows with unexpected_kernel_mode_trap on the blue screen of death... If we boot up the server in safe mode it boots up OK however we cannot get any TSM services to run, and it won't let me un-install IBM Director. We took all the hard disks out of this x346 and put them in another x346 to boot up but the same error occurred. As we can see the TSM data storage disks (the D and the E drive) when we boot up in safe mode we would rather not blow everything away by re-installing Windows and having to recreate TSM from scratch and restore all the data from the tapes. I realise that this is mainly a Windows issue now, but if anyone else was in a similar situation with their TSM server I would like to know if you were able to get around it. Thanks Paul Paul Dudley ANL IT Operations Dept. ANL Container Line [EMAIL PROTECTED] ANL DISCLAIMER This e-mail and any file attached is confidential, and intended solely to the named addressees. Any unauthorised dissemination or use is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by return e-mail from your system. Please do not copy, use or make reference to it for any purpose, or disclose its contents to any person.
Re: export nodes causes TSM server crash
John, Thanks for the script to check if any exports are running prior to starting the next batch job of 15 exports. It is currently running to check if it makes life easier for TSM. I managed to get another crash of the TSM server when the export of the first 15 nodes was started. No other activities were being performed on the TSM server except the Tivoli Operational Reporting tool that monitors each hour the TSM activities. I've stopped the reporting service as well to see if this has anything to do with it. No news from IBM support so far, I'll update the thread if I know more later on. regards, Kurt From: ADSM: Dist Stor Manager on behalf of John Monahan Sent: Mon 2/27/2006 22:36 To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] export nodes causes TSM server crash I agree that the TSM server shouldn't ever crash, but just because it shouldn't crash doesn't necessarily mean you should try to run 75 or 100 or 1000 exports concurrently either. Until a fix is produced, I would just limit your concurrent exports to what you know works without committing a self-imposed denial of service attack on your TSM server. Here is what I would do with your scripts that have the exports separated into groups of 15 nodes each: 1. Kick off the first one as is. 2. Modify all the other scripts to first check for any export processes still running, and if there are, then have those scripts reschedule themselves. ie: select * from processes where upper(process)='EXPORT NODE' if (rc_ok) goto reschedule run next set of export node commands here exit :reschedule del sched thisschedname type=a def sched thisschedname type=a cmd=run thisscriptname active=yes startt=NOW+0:30 perunits=onetime exit __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 02:57 PM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: export nodes causes TSM server crash John, The export of just 15 nodes was tested earlier on. It contained the larger nodes already. At that time, the TSM server just was slowly (high CPU consumption and a lot of disk I/O which is normal of course). It worked fine. The export of all of the nodes at the same time causes an immediate crash of the TSM server. I did not mean to do the export at once but did not notice that the parallel/serial commands would not work as the exports are started in the background. So I changed the script to work in groups of 15 nodes. The export of the nodes in groups of 15 caused a new crash when the last group export was started. A few of the earlier exports were still running at that time, the nodes in the latest group export were rather small nodes. A support call was logged of course. The question is what causes the TSM server crash. Except the PK_EXCEPTION and PK_THREAD messages in the application log, nothing else is found. Just have to wait for some new from the labs at this time. And will contact them tomorrow again. regards, Kurt Van: ADSM: Dist Stor Manager namens John Monahan Verzonden: ma 2/27/2006 20:15 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] export nodes causes TSM server crash Let me see if I understand you correctly. The export works fine when only 15 nodes are running, but after 2 hours when the second set of 15 nodes kicks in (while some from the first group of 15 are stilli running) that is when your server crashes? Or does your server crash with only 15 nodes running an export? __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 05:35 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity
export nodes causes TSM server crash
Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt
Re: export nodes causes TSM server crash
Hi, You confuse me, what are you trying to achieve? Monthly exports could easily be replaced with archives or backup sets. //Henrik -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Kurt Beyers Sent: den 27 februari 2006 12:35 To: ADSM-L@VM.MARIST.EDU Subject: export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt --- The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Thank you.
Re: export nodes causes TSM server crash
Henrik, Too many servers over LAN/WAN are in the picture to take a monthly archive in just a weekend. And because there are also many TDP backups, the export was chosen instead of the backupset (only possible for a TSM BA client). The export could be taken relative to a certain data too. A PMR has been opened at TSM support too of course. regards, Kurt From: ADSM: Dist Stor Manager on behalf of Henrik Wahlstedt Sent: Mon 2/27/2006 13:57 To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] export nodes causes TSM server crash Hi, You confuse me, what are you trying to achieve? Monthly exports could easily be replaced with archives or backup sets. //Henrik -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Kurt Beyers Sent: den 27 februari 2006 12:35 To: ADSM-L@VM.MARIST.EDU Subject: export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt --- The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Thank you.
Re: export nodes causes TSM server crash
Let me see if I understand you correctly. The export works fine when only 15 nodes are running, but after 2 hours when the second set of 15 nodes kicks in (while some from the first group of 15 are stilli running) that is when your server crashes? Or does your server crash with only 15 nodes running an export? __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 05:35 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt
Re: export nodes causes TSM server crash
The server should not crash. Ever. End of story. Call support. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of John Monahan Sent: Monday, February 27, 2006 2:15 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: export nodes causes TSM server crash Let me see if I understand you correctly. The export works fine when only 15 nodes are running, but after 2 hours when the second set of 15 nodes kicks in (while some from the first group of 15 are stilli running) that is when your server crashes? Or does your server crash with only 15 nodes running an export? __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 05:35 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt
Re: export nodes causes TSM server crash
John, The export of just 15 nodes was tested earlier on. It contained the larger nodes already. At that time, the TSM server just was slowly (high CPU consumption and a lot of disk I/O which is normal of course). It worked fine. The export of all of the nodes at the same time causes an immediate crash of the TSM server. I did not mean to do the export at once but did not notice that the parallel/serial commands would not work as the exports are started in the background. So I changed the script to work in groups of 15 nodes. The export of the nodes in groups of 15 caused a new crash when the last group export was started. A few of the earlier exports were still running at that time, the nodes in the latest group export were rather small nodes. A support call was logged of course. The question is what causes the TSM server crash. Except the PK_EXCEPTION and PK_THREAD messages in the application log, nothing else is found. Just have to wait for some new from the labs at this time. And will contact them tomorrow again. regards, Kurt Van: ADSM: Dist Stor Manager namens John Monahan Verzonden: ma 2/27/2006 20:15 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] export nodes causes TSM server crash Let me see if I understand you correctly. The export works fine when only 15 nodes are running, but after 2 hours when the second set of 15 nodes kicks in (while some from the first group of 15 are stilli running) that is when your server crashes? Or does your server crash with only 15 nodes running an export? __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 05:35 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt
Re: export nodes causes TSM server crash
I agree that the TSM server shouldn't ever crash, but just because it shouldn't crash doesn't necessarily mean you should try to run 75 or 100 or 1000 exports concurrently either. Until a fix is produced, I would just limit your concurrent exports to what you know works without committing a self-imposed denial of service attack on your TSM server. Here is what I would do with your scripts that have the exports separated into groups of 15 nodes each: 1. Kick off the first one as is. 2. Modify all the other scripts to first check for any export processes still running, and if there are, then have those scripts reschedule themselves. ie: select * from processes where upper(process)='EXPORT NODE' if (rc_ok) goto reschedule run next set of export node commands here exit :reschedule del sched thisschedname type=a def sched thisschedname type=a cmd=run thisscriptname active=yes startt=NOW+0:30 perunits=onetime exit __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 02:57 PM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: export nodes causes TSM server crash John, The export of just 15 nodes was tested earlier on. It contained the larger nodes already. At that time, the TSM server just was slowly (high CPU consumption and a lot of disk I/O which is normal of course). It worked fine. The export of all of the nodes at the same time causes an immediate crash of the TSM server. I did not mean to do the export at once but did not notice that the parallel/serial commands would not work as the exports are started in the background. So I changed the script to work in groups of 15 nodes. The export of the nodes in groups of 15 caused a new crash when the last group export was started. A few of the earlier exports were still running at that time, the nodes in the latest group export were rather small nodes. A support call was logged of course. The question is what causes the TSM server crash. Except the PK_EXCEPTION and PK_THREAD messages in the application log, nothing else is found. Just have to wait for some new from the labs at this time. And will contact them tomorrow again. regards, Kurt Van: ADSM: Dist Stor Manager namens John Monahan Verzonden: ma 2/27/2006 20:15 Aan: ADSM-L@VM.MARIST.EDU Onderwerp: Re: [ADSM-L] export nodes causes TSM server crash Let me see if I understand you correctly. The export works fine when only 15 nodes are running, but after 2 hours when the second set of 15 nodes kicks in (while some from the first group of 15 are stilli running) that is when your server crashes? Or does your server crash with only 15 nodes running an export? __ John Monahan Consultant Infrastructure Solutions Group Computech Resources, Inc. Office: 952-833-0930 ext 109 Cell: 952-221-6938 http://www.computechresources.com Kurt Beyers [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 02/27/2006 05:35 AM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject export nodes causes TSM server crash Hello everybody, I've got a TSM server 5.3.2.2 running on Windows2003 Enterprise Edition SP1 (7 GB RAM, Xeon 3,2 GHz CPU) that has about 100 TSM clients defined. Each month an export of each TSM node with the active backup data will be taken to disk (DS4100 with SATA disks of 250 GB). The disk storage pool that contains the backups is on the DS4100 too. I've scheduled the export of the TSM nodes past weekend with a few scripts. I first tried to launch just one script that took the export in blocks of 15 nodes using the PARALLEL and SERIAL commands. However as the export is started in the background, all of the 75 exporst were started immediately. This causes a TSM server crash. After restarting the TSM server, no error logs are found in the activity log. Except that no more than 16 commands can be started in one PARALLEL statement. The last normal message about the export is written in the log and then the next message are when the server is started again. I've split up then the export myself in a script where the export of 15 nodes was started and 4 administrative schedules were defined that triggered the export of 15 additional nodes every 2 hours later on. The TSM server crashed once more. Is this a know feature when the export of a lot of nodes is started? Am I overseeing some parameters here? Can the export be started in a better way using TSM scripting? An export server instead of an 'export node' for each TSM node is not an option as then the impot of one node would take too much time. thanks in advance, Kurt
Re: TSM Server crash
On Mon, 15 Nov 2004, Richard Sims wrote: Any ideas what I should do now, other than restore the database? Rainer - You probably have a database inconsistency...which goes back some days in time such that a db restore may be futile. (See if there is a server error log file - dsmserv.err - which server failures want to produce.) Yes, there is, and a dsmserv.dmp file. Your best course of action would probably be to contact TSM Support for analysis of your database problem. Thanks, Richard. I had the slim hope that someone might suggest some immediate remedy. Rainer Schöpf
TSM Server crash
This evening, our TSM server crashed (again). I've seen this very error several times, with version 5.2.2.0. When this happened I restored the database from the latest backup and everything was okay, for a while at least. I upgraded to 5.2.3.0 recently, and it's the first time it happened with 5.2.3.0 Version information: TSM for Windows, Version 5, Release 2, Level 3.0 on Windows 2000 Server SP4 with all the latest patches from Windows Update. Error message in console.log: ANRD pkthread.c(835): ThreadId50 Run-time assertion failed: bitOff = 0, Thread 50, File dballoc.c, Line 388. Callchain: 104EAD79 outTextf()+1529 - 10006649 pkLogicAbort()+39 - Error message in Windows Event Log: TSM Server Diagnostic: ANRD: ADSM Exception Information: file = pkthread.c, line = 2253,Code = e001, Address = 77E9BBF3~ When I try to restart the server, I get this error message in console.log: Tivoli Storage Manager for Windows Version 5, Release 2, Level 3.0 Licensed Materials - Property of IBM (C) Copyright IBM Corporation 1990,2003. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation. ANR0990I Server restart-recovery in progress. ANR0200I Recovery log assigned capacity is 1024 megabytes. ANR0201I Database assigned capacity is 4096 megabytes. ANR0306I Recovery log volume mount in progress. ANRD dballoc.c(1363): ThreadId0 Zero bit count mismatch for SMP page addr 153600; Zero Bits = 1, HeaderZeroBits = 2. Callchain: 104EAD79 outTextf()+1529 - 101A53B6 tsmInitializeServer()+19DB86 - 00480001 Unknown - ANR0361E Database initialization failed: error initializing database page allocator. I cannot look into the activity log until I restore the database, but I think it was running a scheduled backup from a Windows client when the error occured. Any ideas what I should do now, other than restore the database? Thanks in advance, Rainer Schöpf
Re: TSM Server crash
... Any ideas what I should do now, other than restore the database? Rainer - You probably have a database inconsistency...which goes back some days in time such that a db restore may be futile. (See if there is a server error log file - dsmserv.err - which server failures want to produce.) Your best course of action would probably be to contact TSM Support for analysis of your database problem. Richard Sims
Windows TSM server Crash
Hi all I have a big problem Is there a step by step procedure when a Windows 2000 TSM server crash I have to rebuild mine and I dont know where to start .. I have full backup of the serer .. but since it is dead ... I'm kindda stuck Thanks Luc Beaudoin Administrateur Réseau / Network Administrator/TSM Hopital General Juif S.M.B.D. Tel: (514) 340-8222 ext:8254
Re: Windows TSM server Crash
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Luc Beaudoin Is there a step by step procedure when a Windows 2000 TSM server crash I have to rebuild mine and I dont know where to start .. I have full backup of the serer .. but since it is dead ... I'm kindda stuck ...which is a good reason why a documented DR solution *must* contain *at least* documentaiton on how to restore the TSM server. Rule #1 of disaster recovery: Practice it, before you have to. -- Mark Stapleton ([EMAIL PROTECTED]) Berbee Information Networks Office 262.521.5627
Re: Windows TSM server Crash
Thanks Mark ... That is exactly what I'm doing now ... in my lab I still don't know where to start ... my TSM server is down, I formatted it ... installed win2k OS ... Now .. what are the steps ... re-install TSM server, I dont have a clue .. I tested (and documented) windows server restore, SQL servers DB restore ... Domain controller restore ... NOW I'm stuck with TSM server restore thanks again Luc Stapleton, Mark [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 2004-11-03 10:50 AM Please respond to ADSM: Dist Stor Manager To: [EMAIL PROTECTED] cc: Subject:Re: Windows TSM server Crash From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Luc Beaudoin Is there a step by step procedure when a Windows 2000 TSM server crash I have to rebuild mine and I dont know where to start .. I have full backup of the serer .. but since it is dead ... I'm kindda stuck ...which is a good reason why a documented DR solution *must* contain *at least* documentaiton on how to restore the TSM server. Rule #1 of disaster recovery: Practice it, before you have to. -- Mark Stapleton ([EMAIL PROTECTED]) Berbee Information Networks Office 262.521.5627
Re: Windows TSM server Crash
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Luc Beaudoin That is exactly what I'm doing now ... in my lab I still don't know where to start ... my TSM server is down, I formatted it ... installed win2k OS ... Now .. what are the steps ... re-install TSM server, I dont have a clue .. I tested (and documented) windows server restore, SQL servers DB restore ... Domain controller restore ... NOW I'm stuck with TSM server restore Read Wanda Prather's response to your initial request; it's on the money as to what you need to do. The point I was making is that you really should have had a DR drill, prior to your actual disaster. If you had, you would now have documentation telling everything you need to do, step by step. Sorry for the pickle you're in now; good luck to you. -- Mark Stapleton ([EMAIL PROTECTED]) Berbee Information Networks Office 262.521.5627
Re: Windows TSM server Crash
Thanks again Mark But maybe I'm not clear enought ... let me try to put it simple I am doing a Disaster recovery plan. I made a lab with three win2k servers(DC, File server/SQL, TSM server) and 1 win2k workstation I made documentation and procedure for those disasters: - Restore a win2000 server (file server) - Restore a win2000 server with SQL 2000 - Restore a win2000 server that is a domain controller The one that I have problem with is - Restore a windows 2000 server with TSM SERVER on it (THE TSM SERVER) To re-install the OS on it .. no problem ... but How can I restore the data if the THE TSM SERVER is dead ... what R the steps ... exemple : 1. install TSM SERVER software 2. configure drive or library 3. import node or volumes .. 4 ... etc .. that is what I'm looking for .. the steps after the OS is up to put THE TSM SERVER back on-line with THE TSM DB, POLICY DOMAINS, NODES, SCHEDULES hope it's more clear thanks Luc
Re: Windows TSM server Crash
The chapter I pointed you to the other day includes information on restoring the TSM server. My recommendation would be to: 1) Read the chapter I mentioned. Section Recovering Your Server Using Database and Storage Pool Backups covers the restore scenario, but the entire chapter is worth reading to understand your options for protecting the server as well. 2) Attempt the restore per the info in the book. 3) If you continue to have trouble, post the *exact* steps you took, as well as the results that demonstrate the problem you are having. That will give list members a lot more info to help you with. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence. ADSM: Dist Stor Manager [EMAIL PROTECTED] wrote on 11/03/2004 09:46:31: Thanks again Mark But maybe I'm not clear enought ... let me try to put it simple I am doing a Disaster recovery plan. I made a lab with three win2k servers(DC, File server/SQL, TSM server) and 1 win2k workstation I made documentation and procedure for those disasters: - Restore a win2000 server (file server) - Restore a win2000 server with SQL 2000 - Restore a win2000 server that is a domain controller The one that I have problem with is - Restore a windows 2000 server with TSM SERVER on it (THE TSM SERVER) To re-install the OS on it .. no problem ... but How can I restore the data if the THE TSM SERVER is dead ... what R the steps ... exemple : 1. install TSM SERVER software 2. configure drive or library 3. import node or volumes .. 4 ... etc .. that is what I'm looking for .. the steps after the OS is up to put THE TSM SERVER back on-line with THE TSM DB, POLICY DOMAINS, NODES, SCHEDULES hope it's more clear thanks Luc
Re: Windows TSM server Crash
Hi, If your ITSM server is running, give it the Prepare command (help prepare). This will give you all the info you need to recover your server. Some basic steps - Install OS; - Install ITSM server (level at least same as before); - Configure / Format DB en Logs. Size is as before or bigger; - Configure devices; - Configure (restore volhist) - RESTORE your ITSM database from last Full and Incremental DB back volumes. It isn't the complete work, some steps. Regards, Karel -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] Behalf Of Luc Beaudoin Sent: woensdag 3 november 2004 17:47 To: [EMAIL PROTECTED] Subject: Re: Windows TSM server Crash Thanks again Mark But maybe I'm not clear enought ... let me try to put it simple I am doing a Disaster recovery plan. I made a lab with three win2k servers(DC, File server/SQL, TSM server) and 1 win2k workstation I made documentation and procedure for those disasters: - Restore a win2000 server (file server) - Restore a win2000 server with SQL 2000 - Restore a win2000 server that is a domain controller The one that I have problem with is - Restore a windows 2000 server with TSM SERVER on it (THE TSM SERVER) To re-install the OS on it .. no problem ... but How can I restore the data if the THE TSM SERVER is dead ... what R the steps ... exemple : 1. install TSM SERVER software 2. configure drive or library 3. import node or volumes .. 4 ... etc .. that is what I'm looking for .. the steps after the OS is up to put THE TSM SERVER back on-line with THE TSM DB, POLICY DOMAINS, NODES, SCHEDULES hope it's more clear thanks Luc
TSM-Server crash : Help
Hi all, since 3 days my TSM-Server (AIX 4.3.3 , TSM 4.1.4) crashs at night. In dsmserv.err the following messages are written : 11/21/2002 01:31:06 ANR7834S Thread 72 (tid 482c) terminating on signal 11 (Seg mentation violation). 11/21/2002 01:31:06 ANR7834S GPR 0: 0x, 1: 0x36a586b0, 2: 0x30199d48, 3: 0x35ba5ed0 11/21/2002 01:31:06 ANR7834S GPR 4: 0x, 5: 0x, 6: 0x0001, 7: 0x1000b7db 11/21/2002 01:31:06 ANR7834S GPR 8: 0xb7db, 9: 0x, 10: 0xf0218fd4, 11: 0x3634d384 11/21/2002 01:31:06 ANR7834S GPR 12: 0x10242198, 13: 0x, 14: 0x0001, 15: 0x0001 11/21/2002 01:31:06 ANR7834S GPR 16: 0x, 17: 0x, 18: 0x, 19: 0x 11/21/2002 01:31:06 ANR7834S GPR 20: 0x, 21: 0x, 22: 0x0004, 23: 0x300c3a38 11/21/2002 01:31:06 ANR7834S GPR 24: 0x, 25: 0x0001, 26: 0x, 27: 0x35ba5ed0 11/21/2002 01:31:06 ANR7834S GPR 28: 0x35cf5ef0, 29: 0x300027a8, 30: 0x35cf5ef0, 31: 0x35cf5dc0 11/21/2002 01:31:06 ANR7834S IAR: 0x102421b0 LR: 0x10242198 CONTEXT: 0x36a58330 11/21/2002 01:31:06 ANR7833S Server thread 1 terminated in response to program abort. 11/21/2002 01:31:06 ANR7833S Server thread 2 terminated in response to program abort. In actlog the last entries are errors-messages : 11/21/02 01:30:14 ANR0406I Session 1545 started for node TENTAX (OpenVMS) (Tcp/Ip 148.192.120.14(1025)). 11/21/02 01:30:20 ANR0444W Protocol error on session 1101 for node TENTAX (OpenVMS) - out-of-sequence verb (type (Unknown)) received. 11/21/02 01:30:21 ANR0484W Session 1101 for node TENTAX (OpenVMS) terminated - protocol violation detected. 11/21/02 01:30:30 ANR0480W Session 1102 for node TENTAX (OpenVMS) terminated - connection with client severed. 11/21/02 01:30:47 ANR0480W Session 1157 for node TENTAX (OpenVMS) terminated - connection with client severed. 11/21/02 01:31:05 ANR0444W Protocol error on session 1545 for node TENTAX (OpenVMS) - out-of-sequence verb (type ArchQryResp1) received. Can anybody give me some help. Thanks Chris
Re: TSM-Server crash : Help
Christoph, what were the functions in the trace-back? Anything like this: 08/01/2002 16:58:43 ANRD Trace-back of called functions: 08/01/2002 16:58:43 ANRD 0x000100077208 pkFree 08/01/2002 16:58:43 ANRD 0x0001006B0138 SmDoEventLog 08/01/2002 16:58:43 ANRD 0x0001006AC9C0 SmNodeSession 08/01/2002 16:58:43 ANRD 0x0001006999C8 HandleNodeSession 08/01/2002 16:58:43 ANRD 0x000100699C0C DoNodeGeneral 08/01/2002 16:58:43 ANRD 0x000100697008 smExecuteSession 08/01/2002 16:58:43 ANRD 0x00010008A1D8 SessionThread 08/01/2002 16:58:43 ANRD 0x00010007B728 StartThread 08/01/2002 16:58:43 ANRD 0x7EC1F8A0 *UNKNOWN* 08/01/2002 16:58:43 ANRD 0x00010007B620 StartThread In which case, we have seen this after a client upgrade - in certain circumstances the upgraded client will send invalid event information as it starts, which will crash _any_ server except the latest v5.1.x. Regards, -=Dave=- +44 (0) 20 7608 7140 A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1 |-+--- | | Christoph Pilgram | | | [EMAIL PROTECTED]| | | ELHEIM.COM | | | Sent by: ADSM: Dist Stor Manager | | | [EMAIL PROTECTED] | | | | | | | | | 11/21/2002 08:34 AM | | | Please respond to ADSM: Dist Stor | | | Manager| | | | |-+--- ---| | | | To: [EMAIL PROTECTED] | | cc: | | Subject: TSM-Server crash : Help | ---| Hi all, since 3 days my TSM-Server (AIX 4.3.3 , TSM 4.1.4) crashs at night. In dsmserv.err the following messages are written : 11/21/2002 01:31:06 ANR7834S Thread 72 (tid 482c) terminating on signal 11 (Seg mentation violation). 11/21/2002 01:31:06 ANR7834S GPR 0: 0x, 1: 0x36a586b0, 2: 0x30199d48, 3: 0x35ba5ed0 11/21/2002 01:31:06 ANR7834S GPR 4: 0x, 5: 0x, 6: 0x0001, 7: 0x1000b7db 11/21/2002 01:31:06 ANR7834S GPR 8: 0xb7db, 9: 0x, 10: 0xf0218fd4, 11: 0x3634d384 11/21/2002 01:31:06 ANR7834S GPR 12: 0x10242198, 13: 0x, 14: 0x0001, 15: 0x0001 11/21/2002 01:31:06 ANR7834S GPR 16: 0x, 17: 0x, 18: 0x, 19: 0x 11/21/2002 01:31:06 ANR7834S GPR 20: 0x, 21: 0x, 22: 0x0004, 23: 0x300c3a38 11/21/2002 01:31:06 ANR7834S GPR 24: 0x, 25: 0x0001, 26: 0x, 27: 0x35ba5ed0 11/21/2002 01:31:06 ANR7834S GPR 28: 0x35cf5ef0, 29: 0x300027a8, 30: 0x35cf5ef0, 31: 0x35cf5dc0 11/21/2002 01:31:06 ANR7834S IAR: 0x102421b0 LR: 0x10242198 CONTEXT: 0x36a58330 11/21/2002 01:31:06 ANR7833S Server thread 1 terminated in response to program abort. 11/21/2002 01:31:06 ANR7833S Server thread 2 terminated in response to program abort. In actlog the last entries are errors-messages : 11/21/02 01:30:14 ANR0406I Session 1545 started for node TENTAX (OpenVMS) (Tcp/Ip 148.192.120.14(1025)). 11/21/02 01:30:20 ANR0444W Protocol error on session 1101 for node TENTAX (OpenVMS) - out-of-sequence verb (type (Unknown)) received. 11/21/02 01:30:21 ANR0484W Session 1101 for node TENTAX (OpenVMS) terminated - protocol violation detected. 11/21/02 01:30:30 ANR0480W Session 1102 for node TENTAX (OpenVMS) terminated - connection with client severed. 11/21/02 01:30:47 ANR0480W Session 1157 for node TENTAX (OpenVMS) terminated - connection with client severed. 11/21/02 01:31:05 ANR0444W Protocol error on session 1545 for node TENTAX (OpenVMS) - out-of-sequence verb (type ArchQryResp1) received. Can anybody give me some help. Thanks Chris
Re: TSM server crash when backing up a LINUX
Our TSM server is 3.7.4.6 and Linux client is 4.1.2.0, everytime when we start to backup the Linux, the TSM server crashes immediately. Here are a cut from dsmserv.err. Any idea? Back in January of 1999 much the same was happening, as the client sent the Linux version number to the server and that number was overly long, like 2.1.131. Be sure to report this server defect to TSM Support: no communication from any client - or any other entity attempting to contact the server - should cause the server to crash. Servers need to be able to protect themselves on the network. Richard Sims, BU
