[Veritas-bu] Image Cleanup hangs
All: Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u's thrown in for good measure. Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs. Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn't notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn't work Friday so the rest of us didn't know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn't submit jobs because the nbperm daemon wouldn't stay up. We found a document that said nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: 063008.bpjobd.db , bpjobd.act.db , bpjobd.db , jobid , jobid.lock pempersist2. The bad news was that it cleared out the activity log for the 2 weeks which we can live with. So, that's where we are, anyone know how to take care of the image cleanup hang? Thanks in advance.. Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com ** ** LEGAL DISCLAIMER ** ** This E-mail message and any attachments may contain legally privileged, confidential or proprietary information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution or copying of this E-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this E-mail message from your computer. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Image Cleanup hangs
This can be because of a corrupt image. We also faced same issue where the image cleanup was getting stuck at a particular client. We moved the client's images to a temp location and restarted the netbackup services. Image cleanup auto restarted and completed successfully. On Wed, Jun 8, 2011 at 7:36 AM, BeDour, Wayne wbed...@lear.com wrote: All: Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u’s thrown in for good measure. Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs. Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn’t notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn’t work Friday so the rest of us didn’t know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn’t submit jobs because the nbperm daemon wouldn’t stay up. We found a document that said “nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active”. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: * ** 063008.bpjobd.db* ,*** bpjobd.act.db , bpjobd.db*** * , jobid , jobid.lock pempersist2. * The bad news was that it cleared out the activity log for the 2 weeks which we can live with. So, that’s where we are, anyone know how to take care of the image cleanup hang? Thanks in advance.. Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: *wbed...@lear.com* wbed...@lear.com ** ** LEGAL DISCLAIMER ** ** This E-mail message and any attachments may contain legally privileged, confidential or proprietary information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution or copying of this E-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this E-mail message from your computer. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Image Cleanup hangs
Sounds to me like image cleanup doesn't like something with a particular client's image. As you know, when /usr/openv fills up, nasty things can happen, including corrupt/incomplete catalog entries. I couldn't find a specific technote that addresses this in my quick search, but this one may help you. http://www.symantec.com/business/support/index?page=contentid=TECH55691key=15143actp=LIST It deals with bpdbm core dumping when it encounters the bad image, which would be a good thing to verify (though I think you would already know that). Basically, take a look through BPDBM log to see if you can find the specific image that is causing issues. You can also run image cleanup for a specific client by running bpimage -cleanup -client offending client name, which may help the log searching if nothing else. Once you find the image you will have to clean it up manually. Be careful when cleaning it up, as you don't want to remove any good entries. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. BeDour, Wayne wbed...@lear.com Sent by: veritas-bu-boun...@mailman.eng.auburn.edu 06/08/2011 09:37 AM To veritas-bu@mailman.eng.auburn.edu cc Subject [Veritas-bu] Image Cleanup hangs All: Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u’s thrown in for good measure. Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs. Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn’t notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn’t work Friday so the rest of us didn’t know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn’t submit jobs because the nbperm daemon wouldn’t stay up. We found a document that said “nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active”. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: 063008.bpjobd.db , bpjobd.act.db , bpjobd.db , jobid , jobid.lock pempersist2. The bad news was that it cleared out the activity log for the 2 weeks which we can live with. So, that’s where we are, anyone know how to take care of the image cleanup hang? Thanks in advance.. Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com ** ** LEGAL DISCLAIMER ** ** This E-mail message and any attachments may contain legally privileged, confidential or proprietary information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution or copying of this E-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this E-mail message from your computer. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Image Cleanup hangs
I have been looking in the bpdbm log and have found a record that has the following; 10:26:25.419 [23129] 16 delete_expired_backups: Bad image header: sfd-BV2105-linux-dd_1306900839_INCR, error: file read failed (13) 10:26:25.419 [23129] 2 IsCatalogCleanupTerminated: Terminated = 1 Looks to me like that is the problem image. Now what I can’t find is a doc going through how to remove just one image. Anyone have or know of a document that has instructions to remove one image? Thanks Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com From: rusty.ma...@sungard.com [mailto:rusty.ma...@sungard.com] Sent: Wednesday, June 08, 2011 11:30 AM To: BeDour, Wayne Cc: veritas-bu@mailman.eng.auburn.edu; veritas-bu-boun...@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Image Cleanup hangs Sounds to me like image cleanup doesn't like something with a particular client's image. As you know, when /usr/openv fills up, nasty things can happen, including corrupt/incomplete catalog entries. I couldn't find a specific technote that addresses this in my quick search, but this one may help you. http://www.symantec.com/business/support/index?page=contentid=TECH55691key=15143actp=LIST http://www.symantec.com/business/support/index?page=contentid=TECH55691key=15143actp=LIST It deals with bpdbm core dumping when it encounters the bad image, which would be a good thing to verify (though I think you would already know that). Basically, take a look through BPDBM log to see if you can find the specific image that is causing issues. You can also run image cleanup for a specific client by running bpimage -cleanup -client offending client name, which may help the log searching if nothing else. Once you find the image you will have to clean it up manually. Be careful when cleaning it up, as you don't want to remove any good entries. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. BeDour, Wayne wbed...@lear.com Sent by: veritas-bu-boun...@mailman.eng.auburn.edu 06/08/2011 09:37 AM To veritas-bu@mailman.eng.auburn.edu cc Subject [Veritas-bu] Image Cleanup hangs All: Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u’s thrown in for good measure. Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs. Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn’t notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn’t work Friday so the rest of us didn’t know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn’t submit jobs because the nbperm daemon wouldn’t stay up. We found a document that said “nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active”. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: 063008.bpjobd.db , bpjobd.act.db , bpjobd.db , jobid , jobid.lock pempersist2. The bad news was that it cleared out the activity log for the 2 weeks which we can live with. So, that’s where we are, anyone know how to take care of the image cleanup hang? Thanks in advance.. Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com ** ** LEGAL DISCLAIMER ** ** This E-mail message and any attachments may contain legally privileged, confidential or proprietary information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution or copying of this E-mail message is strictly prohibited. If you have received this message in error, please
Re: [Veritas-bu] Image Cleanup hangs
Goto /usr/openv/netbackup/db/images/sfd-BV2105-linux-dd/ There you will find a lot of folders with different ctimes goto 1306* first run cp sfd-BV2105-linux-dd_1306900839_INCR /tmp then run cp sfd-BV2105-linux-dd_1306900839_INCR.f.z /tmp delete these two files recycle the services and wait for image cleanup to auto start or run bpimage -cleanup -allclients On Wed, Jun 8, 2011 at 9:01 AM, BeDour, Wayne wbed...@lear.com wrote: I have been looking in the bpdbm log and have found a record that has the following; 10:26:25.419 [23129] 16 delete_expired_backups: Bad image header: sfd-BV2105-linux-dd_1306900839_INCR, error: file read failed (13) 10:26:25.419 [23129] 2 IsCatalogCleanupTerminated: Terminated = 1 Looks to me like that is the problem image. Now what I can’t find is a doc going through how to remove just one image. Anyone have or know of a document that has instructions to remove one image? Thanks Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com *From:* rusty.ma...@sungard.com [mailto:rusty.ma...@sungard.com] *Sent:* Wednesday, June 08, 2011 11:30 AM *To:* BeDour, Wayne *Cc:* veritas-bu@mailman.eng.auburn.edu; veritas-bu-boun...@mailman.eng.auburn.edu *Subject:* Re: [Veritas-bu] Image Cleanup hangs Sounds to me like image cleanup doesn't like something with a particular client's image. As you know, when /usr/openv fills up, nasty things can happen, including corrupt/incomplete catalog entries. I couldn't find a specific technote that addresses this in my quick search, but this one may help you. http://www.symantec.com/business/support/index?page=contentid=TECH55691key=15143actp=LIST It deals with bpdbm core dumping when it encounters the bad image, which would be a good thing to verify (though I think you would already know that). Basically, take a look through BPDBM log to see if you can find the specific image that is causing issues. You can also run image cleanup for a specific client by running bpimage -cleanup -client offending client name, which may help the log searching if nothing else. Once you find the image you will have to clean it up manually. Be careful when cleaning it up, as you don't want to remove any good entries. *Rusty Major, MCSE, BCFP, VCS* ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ P *Think before you print* CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. *BeDour, Wayne wbed...@lear.com* Sent by: veritas-bu-boun...@mailman.eng.auburn.edu 06/08/2011 09:37 AM To veritas-bu@mailman.eng.auburn.edu cc Subject [Veritas-bu] Image Cleanup hangs All: Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u’s thrown in for good measure. Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs. Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn’t notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn’t work Friday so the rest of us didn’t know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn’t submit jobs because the nbperm daemon wouldn’t stay up. We found a document that said “nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active”. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: * 063008.bpjobd.db* ,* bpjobd.act.db , bpjobd.db , jobid , jobid.lock pempersist2. * The bad news was that it cleared out the activity log for the 2 weeks which we can live with. So, that’s where we are, anyone know how to take care of the image cleanup hang? Thanks in advance.. Wayne BeDour Unix System Administrator PH: 248-447-1739 Internet: wbed...@lear.com ** ** LEGAL DISCLAIMER ** ** This E-mail message and any attachments may contain legally privileged, confidential or proprietary information
