Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
Steve, We have seen this issue extensively on Solaris 9/NBU 5.1 MPx - the resolution for us was a complex one involving making sure the /etc/system settings specify enough shared memory (we have ours at half of the physical RAM) and message queue sizes (we have ours set to "set msgsys:msginfo_msgmnb=524288", the highest Symantec recommends) and monitoring the 'ipcs' output on a five minute basis. We found that we were effectively killing the scheduler at certain times during the week due to the sheer number of jobs set to start at any given time. We have also encountered behavior where the scheduler completely missed (error code 196) a vast number of jobs and this pointed us to a bp.conf setting on our master (CLIENT_CONNECT_TIMEOUT was set to 3600, in combination with enough clients defined in backup policies that do not exist, causes the scheduler to hang/choke/freeze in such a manner that the number of 196s is shocking - once we put this back down to a setting of "300", all was right with our world). A fix for a memory leak? I wish it was that simply for us - we keep getting the "upgrade to 6.5" to fix our problem and after the exhaustive experience with the scheduler in 5.1, I am convinced it is better in 6.5, just not sure what else might be worse -Doug From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew Agle Sent: Sunday, December 30, 2007 12:06 PM To: Hudson, Steve Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging I've seen that action with solaris 10/NBU5.1MP5. We ended up finding a memory leak with the bpsched module, sent a memory dump to Veritas and they provided an update to us. I suggest the next time it happens that you get a memory dump and send it in. Matthew On Dec 19, 2007 11:35 AM, Hudson, Steve <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote: We have seen at least 4 times in the last week where all Jobs Hang and it looks like BPSCHED goes away. We must then use the Kill -9 command on the Solaris 8 host to kill everything as the bp.kill_all and netbackup stop commands are ineffective. Anyone else seen this behavior in 5.1 MP5 ??? Steven R. Hudson Sysadmin - Enterprise Storage Iron Mountain 745 Atlantic Avenue Boston MA 02111 Phone: (617) 535-2849 [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> The information contained in this email message and its attachments is intended only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet is not a secure communications medium. If you are requesting or have requested the transmittal of personal data, as defined in applicable privacy laws by means of email or in an attachment to email you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not he intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or coping or disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email and delete the original message. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu<mailto:Veritas-bu@mailman.eng.auburn.edu> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- Matthew C. Agle, MCSA [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> Define Trouble: One hundred users standing up in their cubes asking "what happened" after you change a setting ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
I've seen that action with solaris 10/NBU5.1MP5. We ended up finding a memory leak with the bpsched module, sent a memory dump to Veritas and they provided an update to us. I suggest the next time it happens that you get a memory dump and send it in. Matthew On Dec 19, 2007 11:35 AM, Hudson, Steve <[EMAIL PROTECTED]> wrote: > We have seen at least 4 times in the last week where all Jobs Hang and it > looks like BPSCHED goes away. We must then use the Kill -9 command on the > Solaris 8 host to kill everything as the bp.kill_all and netbackup stop > commands are ineffective. Anyone else seen this behavior in 5.1 MP5 ??? > > > > Steven R. Hudson > > Sysadmin - Enterprise Storage > > Iron Mountain > > 745 Atlantic Avenue > > Boston MA 02111 > > Phone: (617) 535-2849 > > > > [EMAIL PROTECTED] > > > > -- > > * The information contained in this email message and its attachments is > intended only for the private and confidential use of the recipient(s) named > above, unless the sender expressly agrees otherwise. Transmission of email > over the Internet is not a secure communications medium. If you are > requesting or have requested the transmittal of personal data, as defined in > applicable privacy laws by means of email or in an attachment to email you > must select a more secure alternate means of transmittal that supports your > obligations to protect such personal data. If the reader of this message is > not he intended recipient and/or you have received this email in error, you > must take no action based on the information in this email and you are > hereby notified that any dissemination, misuse or coping or disclosure of > this communication is strictly prohibited. If you have received this > communication in error, please notify us immediately by email and delete the > original message. * > > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > > -- Matthew C. Agle, MCSA [EMAIL PROTECTED] Define Trouble: One hundred users standing up in their cubes asking "what happened" after you change a setting ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
In my previous job, we had this problem (and they probably still do). We were running on a Solaris-9 master server. We had over 1600 clients and over 1300 policies. The jobs were running 7x24. We upped the values in the /etc/system file to the maximum for what Paul pointed out below. All of this was of limited use as it would still hang up about once a week or so. The solution is to go to 6.x because of the new scheduler. --stuart liddle -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin Piszcz Sent: Wednesday, December 19, 2007 10:15 AM To: Paul Keating Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging Ahh yes he is using Solaris 8--definitely make sure you tune appropriately, with 10, most of the parameters are not needed. Justin. On Wed, 19 Dec 2007, Paul Keating wrote: > There used to be a technote that no logner exists. http://seer.support.veritas.com/docs/268122.htm : Message Queue parameters: On some UNIX platforms with NetBackup configurations, it can be necessary to increase the system's message queue resources to avoid bpsched hangs. For example, the following changes may need to be made to the /etc/system file: set msgsys:msginfo_msgmap=500 set msgsys:msginfo_msgmnb=65536 set msgsys:msginfo_msgssz=16 set msgsys:msginfo_msgseg=8192 set msgsys:msginfo_msgtql=500 Also, the below technote may be of interest. http://seer.support.veritas.com/docs/274544.htm -- > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf > Of Justin Piszcz > Sent: December 19, 2007 12:50 PM > To: Hudson, Steve > Cc: veritas-bu@mailman.eng.auburn.edu > Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging > > > I have not seen that before but I have only used 5.1MP4 and 5.1MP6 in > production, not 5.1MP5. If you open a case with Symantec one > of the first > things they may ask you to do is upgrade to 5.1MP6, is that possible? > > Justin. > > On Wed, 19 Dec 2007, Hudson, Steve wrote: > > > We have seen at least 4 times in the last week where all > Jobs Hang and > > it looks like BPSCHED goes away. We must then use the Kill > -9 command on > > the Solaris 8 host to kill everything as the bp.kill_all > and netbackup > > stop commands are ineffective. Anyone else seen this > behavior in 5.1 MP5 > > ??? > > > > > > > > Steven R. Hudson > > > > Sysadmin - Enterprise Storage > > > > Iron Mountain > > > > 745 Atlantic Avenue > > > > Boston MA 02111 > > > > Phone: (617) 535-2849 > > > > > > > > [EMAIL PROTECTED] > > > > > > > > > > > > The information contained in this email message and its attachments > is intended > > only for the private and confidential use of the recipient(s) named > above, unless the sender expressly agrees otherwise. Transmission > of email over the Internet > > is not a secure communications medium. If you are requesting or > have requested > > the transmittal of personal data, as defined in applicable privacy > laws by means > > of email or in an attachment to email you must select a more > secure alternate means of transmittal that supports your > obligations to protect such personal data. If the reader of this > message is not he intended recipient and/or you have received this > email in error, you must take no action based on the information in > this email and you are hereby notified that any dissemination, > misuse or coping or disclosure of this communication is strictly > prohibited. If you have received > > this communication in error, please notify us immediately by email > and delete the original message. > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > La version française suit le texte anglais. This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so. Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
Ahh yes he is using Solaris 8--definitely make sure you tune appropriately, with 10, most of the parameters are not needed. Justin. On Wed, 19 Dec 2007, Paul Keating wrote: There used to be a technote that no logner exists. http://seer.support.veritas.com/docs/268122.htm : Message Queue parameters: On some UNIX platforms with NetBackup configurations, it can be necessary to increase the system's message queue resources to avoid bpsched hangs. For example, the following changes may need to be made to the /etc/system file: set msgsys:msginfo_msgmap=500 set msgsys:msginfo_msgmnb=65536 set msgsys:msginfo_msgssz=16 set msgsys:msginfo_msgseg=8192 set msgsys:msginfo_msgtql=500 Also, the below technote may be of interest. http://seer.support.veritas.com/docs/274544.htm -- -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin Piszcz Sent: December 19, 2007 12:50 PM To: Hudson, Steve Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging I have not seen that before but I have only used 5.1MP4 and 5.1MP6 in production, not 5.1MP5. If you open a case with Symantec one of the first things they may ask you to do is upgrade to 5.1MP6, is that possible? Justin. On Wed, 19 Dec 2007, Hudson, Steve wrote: > We have seen at least 4 times in the last week where all Jobs Hang and > it looks like BPSCHED goes away. We must then use the Kill -9 command on > the Solaris 8 host to kill everything as the bp.kill_all and netbackup > stop commands are ineffective. Anyone else seen this behavior in 5.1 MP5 > ??? > > > > Steven R. Hudson > > Sysadmin - Enterprise Storage > > Iron Mountain > > 745 Atlantic Avenue > > Boston MA 02111 > > Phone: (617) 535-2849 > > > > [EMAIL PROTECTED] > > > > > > The information contained in this email message and its attachments is intended > only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet > is not a secure communications medium. If you are requesting or have requested > the transmittal of personal data, as defined in applicable privacy laws by means > of email or in an attachment to email you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not he intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or coping or disclosure of this communication is strictly prohibited. If you have received > this communication in error, please notify us immediately by email and delete the original message. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu La version française suit le texte anglais. This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so. Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion, utilisation ou copie de ce courriel ou des renseignements qu'il contient par une personne autre que le ou les destinataires désignés est interdite. Si vous recevez ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre ordinateur toute copie du courriel reçu. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
There used to be a technote that no logner exists. http://seer.support.veritas.com/docs/268122.htm : Message Queue parameters: On some UNIX platforms with NetBackup configurations, it can be necessary to increase the system's message queue resources to avoid bpsched hangs. For example, the following changes may need to be made to the /etc/system file: set msgsys:msginfo_msgmap=500 set msgsys:msginfo_msgmnb=65536 set msgsys:msginfo_msgssz=16 set msgsys:msginfo_msgseg=8192 set msgsys:msginfo_msgtql=500 Also, the below technote may be of interest. http://seer.support.veritas.com/docs/274544.htm -- > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf > Of Justin Piszcz > Sent: December 19, 2007 12:50 PM > To: Hudson, Steve > Cc: veritas-bu@mailman.eng.auburn.edu > Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging > > > I have not seen that before but I have only used 5.1MP4 and 5.1MP6 in > production, not 5.1MP5. If you open a case with Symantec one > of the first > things they may ask you to do is upgrade to 5.1MP6, is that possible? > > Justin. > > On Wed, 19 Dec 2007, Hudson, Steve wrote: > > > We have seen at least 4 times in the last week where all > Jobs Hang and > > it looks like BPSCHED goes away. We must then use the Kill > -9 command on > > the Solaris 8 host to kill everything as the bp.kill_all > and netbackup > > stop commands are ineffective. Anyone else seen this > behavior in 5.1 MP5 > > ??? > > > > > > > > Steven R. Hudson > > > > Sysadmin - Enterprise Storage > > > > Iron Mountain > > > > 745 Atlantic Avenue > > > > Boston MA 02111 > > > > Phone: (617) 535-2849 > > > > > > > > [EMAIL PROTECTED] > > > > > > > > > > > > The information contained in this email message and its attachments > is intended > > only for the private and confidential use of the recipient(s) named > above, unless the sender expressly agrees otherwise. Transmission > of email over the Internet > > is not a secure communications medium. If you are requesting or > have requested > > the transmittal of personal data, as defined in applicable privacy > laws by means > > of email or in an attachment to email you must select a more > secure alternate means of transmittal that supports your > obligations to protect such personal data. If the reader of this > message is not he intended recipient and/or you have received this > email in error, you must take no action based on the information in > this email and you are hereby notified that any dissemination, > misuse or coping or disclosure of this communication is strictly > prohibited. If you have received > > this communication in error, please notify us immediately by email > and delete the original message. > ___ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > La version française suit le texte anglais. This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so. Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion, utilisation ou copie de ce courriel ou des renseignements qu'il contient par une personne autre que le ou les destinataires désignés est interdite. Si vous recevez ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre ordinateur toute copie du courriel reçu. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
No probably not as we are looking to go to 6.5.1 right after the first of the year... -Original Message- From: Justin Piszcz [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 19, 2007 12:50 PM To: Hudson, Steve Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging I have not seen that before but I have only used 5.1MP4 and 5.1MP6 in production, not 5.1MP5. If you open a case with Symantec one of the first things they may ask you to do is upgrade to 5.1MP6, is that possible? Justin. On Wed, 19 Dec 2007, Hudson, Steve wrote: > We have seen at least 4 times in the last week where all Jobs Hang and > it looks like BPSCHED goes away. We must then use the Kill -9 command on > the Solaris 8 host to kill everything as the bp.kill_all and netbackup > stop commands are ineffective. Anyone else seen this behavior in 5.1 MP5 > ??? > > > > Steven R. Hudson > > Sysadmin - Enterprise Storage > > Iron Mountain > > 745 Atlantic Avenue > > Boston MA 02111 > > Phone: (617) 535-2849 > > > > [EMAIL PROTECTED] > > > > > > The information contained in this email message and its attachments is intended > only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet > is not a secure communications medium. If you are requesting or have requested > the transmittal of personal data, as defined in applicable privacy laws by means > of email or in an attachment to email you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not he intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or coping or disclosure of this communication is strictly prohibited. If you have received > this communication in error, please notify us immediately by email and delete the original message. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NBU 5.1 MP5 all jobs hanging
I have not seen that before but I have only used 5.1MP4 and 5.1MP6 in production, not 5.1MP5. If you open a case with Symantec one of the first things they may ask you to do is upgrade to 5.1MP6, is that possible? Justin. On Wed, 19 Dec 2007, Hudson, Steve wrote: > We have seen at least 4 times in the last week where all Jobs Hang and > it looks like BPSCHED goes away. We must then use the Kill -9 command on > the Solaris 8 host to kill everything as the bp.kill_all and netbackup > stop commands are ineffective. Anyone else seen this behavior in 5.1 MP5 > ??? > > > > Steven R. Hudson > > Sysadmin - Enterprise Storage > > Iron Mountain > > 745 Atlantic Avenue > > Boston MA 02111 > > Phone: (617) 535-2849 > > > > [EMAIL PROTECTED] > > > > > > The information contained in this email message and its attachments is intended > only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet > is not a secure communications medium. If you are requesting or have requested > the transmittal of personal data, as defined in applicable privacy laws by means > of email or in an attachment to email you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not he intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or coping or disclosure of this communication is strictly prohibited. If you have received > this communication in error, please notify us immediately by email and delete the original message. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] NBU 5.1 MP5 all jobs hanging
We have seen at least 4 times in the last week where all Jobs Hang and it looks like BPSCHED goes away. We must then use the Kill -9 command on the Solaris 8 host to kill everything as the bp.kill_all and netbackup stop commands are ineffective. Anyone else seen this behavior in 5.1 MP5 ??? Steven R. Hudson Sysadmin - Enterprise Storage Iron Mountain 745 Atlantic Avenue Boston MA 02111 Phone: (617) 535-2849 [EMAIL PROTECTED] The information contained in this email message and its attachments is intended only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet is not a secure communications medium. If you are requesting or have requested the transmittal of personal data, as defined in applicable privacy laws by means of email or in an attachment to email you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not he intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or coping or disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email and delete the original message.___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
