Re: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL
As I said, it's not necessarily a disk space issue. But it is harder to find if it isn't :( It can also be a RAM issue, even if it would appear that there's plenty of free memory left. Some kernel and driver parts of windows allocate a fixed pool of memory at boot, and can't grow that later. The high part of the error code won't be of much help here either, 7 is 'facility_win32', which means that a win32 API call failed. Could hardly be more generic. I can only suggest that you google for the error code, see if anything comes up that resembles the symptoms you get, and then check the conditions more deeply and see if they apply in your case... Maybe a good place to start: there is a thread about this error, related to VSS but with EMC instead of Bacula, at backup central. http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/emc-networker-19/64bit-win-client-fails-with-vss-errors-100397/ A couple of answers near the bottom of the page give solutions that worked for them. -Original Message- From: global16 [mailto:bacula-fo...@backupcentral.com] Sent: 21 April 2015 19:19 To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL (From: global16 [mailto:bacula-forum at backupcentral.com] ) On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 Is one of the server's disks full or nearing its capacity? 0x80070008 means Not enough storage is available to process this command. This message does not always indicate a lack of disk space, but that is the first thing to check if you get it. -- Thanks for the tips Luc Van der Veken, unfortunately I don't see disk space as being an issue. The 'E' volume on the client is only 60% used with over 500GB of free space. The destination storage pool also has plenty of space free. It is strange that if I reboot the client server, backups with VSS will succeed for a few days and then start failing again... +-- |This was sent by bumba...@hotmail.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL
Hello, I would be careful about assuming that VSS is reporting errors on the E drive. It could well be some other drive, so it would be a good idea to check them all. Generally, VSS has a number of bugs, and you may be triggering one, which can explain why it works after a reboot and then fails after a few days. In general rather than rebooting, you can correct problems in VSS itself by starting and restarting all the VSS services -- there are a good number of them, which may be a better temporary workaround than rebooting. The best solution would be to find the cause. Note: Windows 2012 has changed some of the VSS API, and this can cause problems too. Hopeful in the next couple of months I will have all the Windows 2012 specific problems fixed in the next Bacula Enterprise Windows binary release. At some point, there will be a community release of the same code, but this is waiting on additional man power as explained in my last Bacula Status Report. Best regards, Kern On 21.04.2015 19:18, global16 wrote: (From: global16 [mailto:bacula-forum at backupcentral.com] ) On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 Is one of the server's disks full or nearing its capacity? 0x80070008 means Not enough storage is available to process this command. This message does not always indicate a lack of disk space, but that is the first thing to check if you get it. -- Thanks for the tips Luc Van der Veken, unfortunately I don't see disk space as being an issue. The 'E' volume on the client is only 60% used with over 500GB of free space. The destination storage pool also has plenty of space free. It is strange that if I reboot the client server, backups with VSS will succeed for a few days and then start failing again... +-- |This was sent by bumba...@hotmail.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL
(From: global16 [mailto:bacula-fo...@backupcentral.com] ) On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 Is one of the server's disks full or nearing its capacity? 0x80070008 means Not enough storage is available to process this command. This message does not always indicate a lack of disk space, but that is the first thing to check if you get it. Tip: to quickly translate a Windows error code like 0x80070008 to a readable message, take the lower half - the last 4 digits, in this case 0008, convert it from hexadecimal to decimal - in this case 8, and then pass that value after 'net helpmsg ' at a command prompt. Works for most error codes, up to a decimal value of a few hundreds. C:\net helpmsg 8 Not enough storage is available to process this command. Some other common ones: C:\net helpmsg 3 The system cannot find the path specified. C:\net helpmsg 5 Access is denied. The upper part - 0x8007 - gives info about where the error originated, and the sort of message (first digit 8 or C = error), but there's no such way to quickly decode it at a command prompt. -Original Message- From: global16 [mailto:bacula-fo...@backupcentral.com] Sent: 21 April 2015 2:03 To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL I have been running the same version of Bacula on a Windows 2k12 server for over a year (5.2.10). A few weeks ago, we started getting all our jobs failing over the weekend due to an apparent issue with VSS. Has anyone encountered the following fatal error: 20-Apr 19:30 bacula-zha JobId 5546: Start Backup JobId 5546, Job=Software_Share_Backup_-_Encrypted.2015-04-20_19.30.00_41 20-Apr 19:30 bacula-zha JobId 5546: Using Device FileStorage-software to write. 20-Apr 19:30 bacula-zha JobId 5546: Sending Accurate information. 20-Apr 19:30 bacula-sd JobId 5546: Volume zha-software-encrypted-vol0360 previously written, moving to end of data. 20-Apr 19:30 bacula-sd JobId 5546: Ready to append to end of Volume zha-software-encrypted-vol0360 size=1775302040 20-Apr 19:30 fs01 JobId 5546: Fatal error: VSS API failure calling CoInitialize. ERR=Unexpected error. The error code is logged in the error log file. 20-Apr 19:30 fs01 JobId 5546: Fatal error: VSS was not initialized properly. ERR=Cannot create a file when that file already exists. 20-Apr 19:30 fs01 JobId 5546: Fatal error: VssOject is NULL. 20-Apr 19:30 bacula-sd JobId 5546: Elapsed time=00:00:06, Transfer rate=0 Bytes/second 20-Apr 19:30 bacula-zha JobId 5546: Error: Bacula bacula-zha 5.2.13 (19Jan13): The only way to fix this is to reboot the client. Items I have tried: * Restarted the Bacula service on the client - backups still failed * Tested connection to client using the 'estimate' command, success. Issue isolated to Windows server. * List current VSS instances via cmd prompt (vssadmin list writers) - All states reported as Stable / No writers reporting errors * Manually create a shadow copy on the client (vssadmin create shadow /for=e: ) and then manually remove all shadow copies (vssadmin delete shadows /all) - backup still fails * Ran System File Checker (sfc /scannow) - No issues found, backups still fail After rebooting the client, backups using VSS work fine for a few days and then start failing with the same FATAL error. It is not always the same job that fails first. On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 If anyone has any ideas on how to further troubleshoot this issue, or have encountered this before, I'd greatly appreciate it.[/img] -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL
On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 Luc Is one of the server's disks full or nearing its capacity? Luc 0x80070008 means Not enough storage is available to process this command. Luc This message does not always indicate a lack of disk space, but that is the first thing to check if you get it. Luc Tip: to quickly translate a Windows error code like 0x80070008 to Luc a readable message, take the lower half - the last 4 digits, in Luc this case 0008, convert it from hexadecimal to decimal - in this Luc case 8, and then pass that value after 'net helpmsg ' at a Luc command prompt. Works for most error codes, up to a decimal Luc value of a few hundreds. God, this is so damn useful to know! Thanks a ton for showing this. Too bad we can't have Bacula do this translation for us and log it, along with the original error code just in case. That would be a huge help for people debugging prblems like this. Luc C:\net helpmsg 8 Luc Not enough storage is available to process this command. Luc Some other common ones: Luc C:\net helpmsg 3 Luc The system cannot find the path specified. Luc C:\net helpmsg 5 Luc Access is denied. Luc The upper part - 0x8007 - gives info about where the error originated, and the sort of message (first digit 8 or C = error), but there's no such way to quickly decode it at a command prompt. Luc -Original Message- Luc From: global16 [mailto:bacula-fo...@backupcentral.com] Luc Sent: 21 April 2015 2:03 Luc To: bacula-users@lists.sourceforge.net Luc Subject: [Bacula-users] Shadow Copy issue - Fatal error: VssOject is NULL Luc I have been running the same version of Bacula on a Windows 2k12 server for over a year (5.2.10). A few weeks ago, we started getting all our jobs failing over the weekend due to an apparent issue with VSS. Has anyone encountered the following fatal error: Luc 20-Apr 19:30 bacula-zha JobId 5546: Start Backup JobId 5546, Job=Software_Share_Backup_-_Encrypted.2015-04-20_19.30.00_41 Luc 20-Apr 19:30 bacula-zha JobId 5546: Using Device FileStorage-software to write. Luc 20-Apr 19:30 bacula-zha JobId 5546: Sending Accurate information. Luc 20-Apr 19:30 bacula-sd JobId 5546: Volume zha-software-encrypted-vol0360 previously written, moving to end of data. Luc 20-Apr 19:30 bacula-sd JobId 5546: Ready to append to end of Volume zha-software-encrypted-vol0360 size=1775302040 Luc 20-Apr 19:30 fs01 JobId 5546: Fatal error: VSS API failure calling CoInitialize. ERR=Unexpected error. The error code is logged in the error log file. Luc 20-Apr 19:30 fs01 JobId 5546: Fatal error: VSS was not initialized properly. ERR=Cannot create a file when that file already exists. Luc 20-Apr 19:30 fs01 JobId 5546: Fatal error: VssOject is NULL. Luc 20-Apr 19:30 bacula-sd JobId 5546: Elapsed time=00:00:06, Transfer rate=0 Bytes/second Luc 20-Apr 19:30 bacula-zha JobId 5546: Error: Bacula bacula-zha 5.2.13 (19Jan13): Luc The only way to fix this is to reboot the client. Items I have tried: Luc * Restarted the Bacula service on the client - backups still failed Luc * Tested connection to client using the 'estimate' command, success. Issue isolated to Windows server. Luc * List current VSS instances via cmd prompt (vssadmin list writers) - All states reported as Stable / No writers reporting errors Luc * Manually create a shadow copy on the client (vssadmin create shadow /for=e: ) and then manually remove all shadow copies (vssadmin delete shadows /all) - backup still fails Luc * Ran System File Checker (sfc /scannow) - No issues found, backups still fail Luc After rebooting the client, backups using VSS work fine for a few days and then start failing with the same FATAL error. It is not always the same job that fails first. Luc On the client, these entries are logged in C:\Program Files\Bacula\working\zhfs01.trace Luc fs01: vss_generic.cpp:366-0 VSSClientGeneric::Initialize: CoInitialize returned 0x80070008 Luc If anyone has any ideas on how to further troubleshoot this issue, or have encountered this before, I'd greatly appreciate it.[/img] Luc -- Luc BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Luc Develop your own process in accordance with the BPMN 2 standard Luc Learn Process modeling best practices with Bonita BPM through live exercises Luc http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ Luc source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF Luc ___ Luc Bacula-users mailing list Luc Bacula-users@lists.sourceforge.net Luc https://lists.sourceforge.net/lists/listinfo/bacula-users -- BPM Camp - Free Virtual Workshop May 6th at
