I really hate it when this happens. I say the problem looks resolved, then 3 days later, it happens again.
Backup was again stalled, this time on an NTBackup log file (which I have now excluded). vssadmin list shadows did not list any VSS shadows. But this time it was a little different. The VSS service was frozen (I couldn't stop it). SQL Server was also frozen. I canceled the bacula job, and had to kill the bacula service on the Windows FD. Since VSS wasn't dieing, nor was SQL server, I had no recourse but to restart the machine. After restarting, the queued backups started just fine and completed with success (an incremental and a differential). I checked the Application log on the system and there were VSS and SQL errors posted, which follow. Hopefully these log entries will help to trace down if it is a Microsoft issue, or a bacula-vss issue. Application Log Errors in order, after reboot, and logged after other services have started: ------------------- Event Type: Error Event Source: VSS Event Category: None Event ID: 6013 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: Sqllib error: OLEDB Error encountered calling IDBInitialize::Initialize. hr = 0x80004005. SQLSTATE: HYT00, Native Error: 0 Source: Microsoft OLE DB Provider for SQL Server Error message: Timeout expired For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0008: 34 37 35 00 00 00 00 00 475..... 0010: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0018: 33 39 35 00 00 00 00 00 395..... -------------- Event Type: Error Event Source: SQLVDI Event Category: None Event ID: 1 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=4600. Thread=5868. Client. Instance=. VD=Global\{D08148B0-DCFF-4B47-B46E-CC335BF0700F}1. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. (two more just like the above error, except with a 2, and 3 after the VD=) ------------------- Event Type: Error Event Source: MSSQLSERVER Event Category: (2) Event ID: 18210 Date: 4/9/2007 Time: 8:35:33 AM User: NT AUTHORITY\SYSTEM Computer: 101-NETWORKTECH Description: BackupMedium::ReportIoError: write failure on backup device '{D08148B0-DCFF-4B47-B46E-CC335BF0700F}2'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.). For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 22 47 00 00 10 00 00 00 "G...... 0008: 10 00 00 00 31 00 30 00 ....1.0. 0010: 31 00 2d 00 4e 00 45 00 1.-.N.E. 0018: 54 00 57 00 4f 00 52 00 T.W.O.R. 0020: 4b 00 54 00 45 00 43 00 K.T.E.C. 0028: 48 00 00 00 00 00 00 00 H....... ------------------- Event Type: Error Event Source: MSSQLSERVER Event Category: (2) Event ID: 18210 Date: 4/9/2007 Time: 8:35:33 AM User: NT AUTHORITY\SYSTEM Computer: 101-NETWORKTECH Description: BackupVirtualDeviceFile::ClearError: failure on backup device '{D08148B0-DCFF-4B47-B46E-CC335BF0700F}3'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.). For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 22 47 00 00 10 00 00 00 "G...... 0008: 10 00 00 00 31 00 30 00 ....1.0. 0010: 31 00 2d 00 4e 00 45 00 1.-.N.E. 0018: 54 00 57 00 4f 00 52 00 T.W.O.R. 0020: 4b 00 54 00 45 00 43 00 K.T.E.C. 0028: 48 00 00 00 00 00 00 00 H....... ------------------- Event Type: Error Event Source: MSSQLSERVER Event Category: (6) Event ID: 3041 Date: 4/9/2007 Time: 8:35:33 AM User: NT AUTHORITY\SYSTEM Computer: 101-NETWORKTECH Description: BACKUP failed to complete the command BACKUP DATABASE master. Check the backup application log for detailed messages. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: e1 0b 00 00 10 00 00 00 รก....... 0008: 10 00 00 00 31 00 30 00 ....1.0. 0010: 31 00 2d 00 4e 00 45 00 1.-.N.E. 0018: 54 00 57 00 4f 00 52 00 T.W.O.R. 0020: 4b 00 54 00 45 00 43 00 K.T.E.C. 0028: 48 00 00 00 00 00 00 00 H....... (Note, I do not have a scheduled backup of any databases on this system!, another error for msdb followed this one) ------------------- Event Type: Error Event Source: SQLVDI Event Category: None Event ID: 1 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: SQLVDI: Loc=SVDS::CloseDevice. Desc=Abort detected. ErrorCode=(0). Process=1524. Thread=5068. Server. Instance=MSSQLSERVER. VD=Global\{D08148B0-DCFF-4B47-B46E-CC335BF0700F}2. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. ------------------- Event Type: Error Event Source: SQLVDI Event Category: None Event ID: 1 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1524. Thread=5068. Server. Instance=MSSQLSERVER. VD=Global\{D08148B0-DCFF-4B47-B46E-CC335BF0700F}2. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. (Another set similar to the above two, except with a 3 after VD=) ------------------- Event Type: Error Event Source: VSS Event Category: None Event ID: 6013 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013 Error state: 1, Severity: 16 Source: Microsoft OLE DB Provider for SQL Server Error message: BACKUP DATABASE is terminating abnormally. SQLSTATE: 42000, Native Error: 3202 Error state: 2, Severity: 16 Source: Microsoft OLE DB Provider for SQL Server Error message: Write on "{D08148B0-DCFF-4B47-B46E-CC335BF0700F}2" failed: 995(The I/O operation has been aborted because of either a thread exit or an application request.) For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0008: 35 39 39 00 00 00 00 00 599..... 0010: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0018: 35 37 33 00 00 00 00 00 573..... ------------------- Event Type: Error Event Source: VSS Event Category: None Event ID: 6013 Date: 4/9/2007 Time: 8:35:33 AM User: N/A Computer: 101-NETWORKTECH Description: Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013 Error state: 1, Severity: 16 Source: Microsoft OLE DB Provider for SQL Server Error message: BACKUP DATABASE is terminating abnormally. SQLSTATE: 42000, Native Error: 3271 Error state: 1, Severity: 16 Source: Microsoft OLE DB Provider for SQL Server Error message: A nonrecoverable I/O error occurred on file "{D08148B0-DCFF-4B47-B46E-CC335BF0700F}3:" 995(The I/O operation has been aborted because of either a thread exit or an application request.). For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0008: 35 39 39 00 00 00 00 00 599..... 0010: 53 51 4c 43 4f 4e 4e 43 SQLCONNC 0018: 35 37 33 00 00 00 00 00 573..... ------------------- It could very well be that my attempt to stop the SQL server caused the above errors on reboot (possibly due to the not-so-clean shutdown of the process). Hopefully these errors will provide some insight into why or how this seems to happen when using the bacula FD client, but not the Microsoft backup client. Thanks Mike Michael Gerszewski wrote: > On a whim, I decided to stop my SQL Server running on this FD. The > backups then ran properly. After I restarted the SQL Server, the FD > continues to backup properly. It has been backing up properly for the > past two days. > > I am also free of the error logs being generated by SQL Server and VSS. > > Maybe there was something strange about my SQL Server instance, and it > causing issues with VSS? Maybe VSS just wanted my SQL Server to be > restarted while the system was running? > > I still don't know why it is working now, when after a reboot it > wouldn't work properly. I will continue to monitor the situation, watch > to see if any more backups hang, and report any log messages on the > Windows box that are possibly relevant. > > Some additional notes for others out there that might experience this > problem...When I initially installed the FD client on this Windows XP > box, there were errors in my Application Log relating to VSS and SQL > Server stating that some of my databases did not have a "simple" > recovery plan. Sure enough, the recovery plan was set to Full on the > databases that errored out. I did change the recovery plan on the > databases and restarted the SQL Server, and that is about when my > hanging backup problems started. Maybe the SQL Server just needed to be > warm restarted again to fully implement the changes and make VSS happy? > Just a guess, and something for others with this issue to try. > > Thanks again for your time and efforts. > > Mike > > Robert Nelson wrote: >> Okay great, I think this should give me something to work with. I'll try >> and repro it here. >> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] [mailto:bacula-users- >>> [EMAIL PROTECTED] On Behalf Of Michael Gerszewski >>> Sent: Friday, March 30, 2007 12:48 PM >>> To: bacula-users@lists.sourceforge.net >>> Cc: Robert Nelson >>> Subject: Re: [Bacula-users] Windows Client, VSS Issues, 2.0.3? >>> >>> Bacula backup sitting at 0 bytes. Floppy drive and hard drive did a >>> seek like a normal VSS startup. >>> >>> vssadmin list shadows: >>> >>> vssadmin 1.0 - Volume Shadow Copy Service administrative command-line tool >>> (C) Copyright 2001 Microsoft Corp. >>> >>> No shadow copies present in the system. >>> >>> ------- >>> >>> When I try to run the vssadmin list writers, it waits for a response. >>> Like VSS is just hung up. I never do get a response from the vssadmin >>> list writers unless I kill the FD and the VSS process. >>> >>> It is difficult for me to reproduce the "hung at end" scenario. But >>> I'll certainly get you the output the next time I see it. >>> >>> With the failed VSS components...When I check and Bacula hasn't been run >>> yet on the system, both are listed as stable. I did see another as >>> failed, and sometimes they return a status: 7, failed. >>> >>> I do run some MS backups on this system as well. Hopefully those aren't >>> causing any conflicts. >>> >>> Thanks again. A great community here. >>> >>> Mike >>> >>> Robert Nelson wrote: >>>>> -----Original Message----- >>>>> From: [EMAIL PROTECTED] [mailto:bacula-users- >>>>> [EMAIL PROTECTED] On Behalf Of Michael Gerszewski >>>>> Sent: Friday, March 30, 2007 12:07 PM >>>>> To: bacula-users@lists.sourceforge.net >>>>> Cc: Robert Nelson >>>>> Subject: Re: [Bacula-users] Windows Client, VSS Issues, 2.0.3? >>>>> >>>>> Thanks for the quick response. >>>>> >>>>> vssadmin list writers: >>>>> vssadmin 1.0 - Volume Shadow Copy Service administrative command-line >>> tool >>>>> (C) Copyright 2001 Microsoft Corp. >>>>> >>>>> Writer name: 'MSDEWriter' >>>>> Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277} >>>>> Writer Instance Id: {b9048c82-c0b6-42e8-b885-3ddc806474bd} >>>>> State: [1] Stable >>>>> >>>>> Writer name: 'WMI Writer' >>>>> Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} >>>>> Writer Instance Id: {a4358c3f-d8b3-4e32-971a-a1aa47f7b0fb} >>>>> State: [9] Failed >>>>> >>>>> Writer name: 'IIS Metabase Writer' >>>>> Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366} >>>>> Writer Instance Id: {38ebb2dd-0122-48e1-915e-23ce2aed2e95} >>>>> State: [9] Failed >>>>> >>>> Interesting the above two seemed to have failed their initialization. >>>> >>>>> Writer name: 'Microsoft Writer (Bootable State)' >>>>> Writer Id: {f2436e37-09f5-41af-9b2a-4ca2435dbfd5} >>>>> Writer Instance Id: {9470d858-515e-47d9-bf71-ba1a26782a69} >>>>> State: [1] Stable >>>>> >>>>> Writer name: 'Microsoft Writer (Service State)' >>>>> Writer Id: {e38c2e3c-d4fb-4f4d-9550-fcafda8aae9a} >>>>> Writer Instance Id: {1a9ca255-4ad6-4172-9d04-d433e9958d33} >>>>> State: [1] Stable >>>>> >>>>> -------------- >>>>> >>>>> vssadmin list shadows (while backup is running): >>>>> >>>>> vssadmin 1.0 - Volume Shadow Copy Service administrative command-line >>> tool >>>>> (C) Copyright 2001 Microsoft Corp. >>>>> >>>>> >>>>> Contents of shadow copy set {26035bf3-a48b-4993-afcb-6f48affefd4c} >>>>> (The shadow copy set contained 1 shadow copies at creation time, 1 >>>>> shadow copie >>>>> s exist now) >>>>> 'MS Software Shadow Copy provider 1.0' shadow copy >>>>> {d6dc446b-5cae-46bb-a555-d344 >>>>> c70d65e5} on volume \\?\Volume{d27e6118-47d1-11d9-aa59-806d6172696f}\ >>>>> >>>>> ------------------ >>>> That looks good. >>>> >>>>> vssadmin list shadows (after backup completes): >>>>> >>>>> vssadmin 1.0 - Volume Shadow Copy Service administrative command-line >>> tool >>>>> (C) Copyright 2001 Microsoft Corp. >>>>> >>>>> No shadow copies present in the system. >>>>> >>>>> ----------------- >>>> That looks very good, doesn't seem we have any leftover crud. >>>> >>>>> A couple more notes that may or may not make a difference. I am >>> running >>>>> SQL Server Express 2005 on this system, and the event logs show the >>>>> databases freezing and unfreezing during the bacula backups. The log >>> is >>>>> telling me the database was successfully backed up, but I am not >>> backing >>>>> up anything from my SQL databases (to my knowledge). >>>>> >>>> That's expected, the system doesn't know what we'll be backing up at the >>>> time we initiate the shadow. >>>> >>>>> Thanks >>>>> >>>>> Mike >>>>> >>>> Can you also post the output from "vssadmin list shadows" when the >>> backup is >>>> hung? Meanwhile, I'll check and make sure we release all our interface >>>> pointers. >>>> >>>>> Robert Nelson wrote: >>>>>> What is the output from the command "vssadmin list writers"? >>>>>> >>>>>> Also what is the output from the command "vssadmin list shadows" after >>>>> the >>>>>> successful backup but prior to the next backup? >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: [EMAIL PROTECTED] [mailto:bacula- >>> users- >>>>>>> [EMAIL PROTECTED] On Behalf Of Michael Gerszewski >>>>>>> Sent: Friday, March 30, 2007 11:27 AM >>>>>>> To: bacula-users >>>>>>> Subject: [Bacula-users] Windows Client, VSS Issues, 2.0.3? >>>>>>> >>>>>>> Greetings, >>>>>>> >>>>>>> I have been evaluating Bacula for the School District I work for as a >>>>>>> backup option for our Windows 2003 Servers. I like many of the >>>>> features >>>>>>> Bacula has compared to other solutions. I have been testing with >>> 2.0.3 >>>>>>> with the director on a linux box, a storage daemon on a Windows 2003 >>>>>>> box, and a file daemon on an XP box. I installed the Linux portion >>>>> from >>>>>>> RPMs and the Windows client with the windows installer. >>>>>>> >>>>>>> I have it configured to do Disk backups. The first couple of days, >>>>>>> everything worked OK with small backups without much change (~100 MB >>>>>>> full, 1-2 MB incremental). Then I switched to a larger amount of >>> files >>>>>>> (~28 GB with ~800 MB of change per day). Now the FD seems to hang >>> up. >>>>>>> Here are the symptoms: >>>>>>> >>>>>>> If I start off a fresh boot, the first backup on this machine will >>> work >>>>>>> fine. If I do not reboot the computer, or kill the bacula-fd and >>>>>>> vssvc.exe processes, the next backup will do one of two things: >>>>>>> >>>>>>> 1) Backup all the way to the end, and just hang >>>>>>> or >>>>>>> 2) Not start the backup at all and sit at 0 Bytes/files >>>>>>> >>>>>>> I can break this cycle by killing the vssvc.exe process (Volume >>> Shadow >>>>>>> Copy). Then the backup finishes successfully, and incidentally >>>>>>> vssvc.exe start again. If it hung at the beginning, it will >>> generally >>>>>>> give a warning or error that VSS could not start and the backup will >>> be >>>>>>> done without VSS. >>>>>>> >>>>>>> One thing that seems odd is that the vssvc.exe process is always >>>>>>> running, even when there isn't a backup running. When I look at our >>>>>>> servers that are running the MS backup utility, vssvc.exe is only >>>>>>> running while a backup is in progress. >>>>>>> >>>>>>> For our implementation, I really need to be able to use VSS, so I >>> have >>>>>>> not tried it with VSS disabled. >>>>>>> >>>>>>> Any assistance would be appreciated. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Mike Gerszewski >>>>>>> >>>>>>> --------------------------------------------------------------------- >>> -- >>>>> -- >>>>>>> Take Surveys. Earn Cash. Influence the Future of IT >>>>>>> Join SourceForge.net's Techsay panel and you'll get the chance to >>> share >>>>>>> your >>>>>>> opinions on IT & business topics through brief surveys-and earn cash >>>>>>> >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >>>>>>> _______________________________________________ >>>>>>> Bacula-users mailing list >>>>>>> Bacula-users@lists.sourceforge.net >>>>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users >>>>> ----------------------------------------------------------------------- >>> -- >>>>> Take Surveys. Earn Cash. Influence the Future of IT >>>>> Join SourceForge.net's Techsay panel and you'll get the chance to share >>>>> your >>>>> opinions on IT & business topics through brief surveys-and earn cash >>>>> >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >>>>> _______________________________________________ >>>>> Bacula-users mailing list >>>>> Bacula-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users >>>> >>> ------------------------------------------------------------------------- >>> Take Surveys. Earn Cash. Influence the Future of IT >>> Join SourceForge.net's Techsay panel and you'll get the chance to share >>> your >>> opinions on IT & business topics through brief surveys-and earn cash >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >>> _______________________________________________ >>> Bacula-users mailing list >>> Bacula-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bacula-users >> >> > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
