Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
amdump now seems to think it should include the old test DLE in new backups. The test DLE is not actually included in the backup as it has no information on where to get it from but it still shows up in the status screen from amdump. When i removed it i ran amadmin delete for the test DLE and then removed it from the disklist. Log has "INFO planner Adding new disk localhost:test" and amdump log has "localhost:test overdue 19105 days for level 0". I removed the tape which the test DLE was on (and some actual backups with it) with 'amrmtape backup --keep-label --cleanup ROT09' and now amadmin find does not see the test DLE anymore and seems to also have resolved amdump. Do i really need to do it this way or should i have done something else to remove the old DLE? Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-20 15:58, Exuvo wrote: Resolved, had an old test DLE on the tape it was trying to use. I don't understand why it still loaded the tape and did not give any useful error message about it. My testing: If i make a new logs directory the estimate actually runs. Tried moving out all the logs from that day but it still fails which i found weird. Tried moving out one log at a time to find the culprit but it seems that as long as any one of the old logs is there my storage DLE fails. I was only running the storage DLE as that was the only one that failed during the normal backup schedule. I also tried clearing the curinfo file for the DLE. Next i tried running a full backup with all DLEs and now they all showed as MISSING immediately. Which was very interesting. The only common thing for when the DLEs started being missing is that they want to use tape ROT09. Which is the first tape i did a test backup on. I think the actual issue is that it is not finding a usable tape even tho it loads the tape first as if it thinks it is an valid tape to use. Removing the test DLE with amadmin delete made it start working again. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-19 03:54, Exuvo wrote: The disk was full of old partial dumps from testing of another backup set so i did not want to flush them to any tapes. The logs "driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896" indicate that it is finding the dump disk with correct size of 2TB free. Even if the dump drive was full i should be getting "FAILED [can't dump required holdingdisk when no holdingdisk space available ]" not MISSING on the DLE. I even tested with the holding disk disabled and it still says it is missing. The path is most certainly not missing from the filesystem. I also tried changing to another file path but still MISSING. It must be something in the database that got stuck, i will have to parse it myself and see if i can figure out what. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-18 02:34, John Lauro wrote: I normally use amflush to clear the holding disk. Not saying rm doesn't work, but that will not update the Amanda info database. It might think that space is still being used. On Sun, Apr 17, 2022 at 2:09 PM Exuvo wrote: I take it you did not read what i wrote. The holding disk is already empty after i 'rm -r':ed it. But even if i tell amanda to never use the holding disk for the failing DLE it just stops immediately when trying to run the planner (it normally takes a few minutes for that DLE). All other DLEs run fine.
Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
Resolved, had an old test DLE on the tape it was trying to use. I don't understand why it still loaded the tape and did not give any useful error message about it. My testing: If i make a new logs directory the estimate actually runs. Tried moving out all the logs from that day but it still fails which i found weird. Tried moving out one log at a time to find the culprit but it seems that as long as any one of the old logs is there my storage DLE fails. I was only running the storage DLE as that was the only one that failed during the normal backup schedule. I also tried clearing the curinfo file for the DLE. Next i tried running a full backup with all DLEs and now they all showed as MISSING immediately. Which was very interesting. The only common thing for when the DLEs started being missing is that they want to use tape ROT09. Which is the first tape i did a test backup on. I think the actual issue is that it is not finding a usable tape even tho it loads the tape first as if it thinks it is an valid tape to use. Removing the test DLE with amadmin delete made it start working again. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-19 03:54, Exuvo wrote: The disk was full of old partial dumps from testing of another backup set so i did not want to flush them to any tapes. The logs "driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896" indicate that it is finding the dump disk with correct size of 2TB free. Even if the dump drive was full i should be getting "FAILED [can't dump required holdingdisk when no holdingdisk space available ]" not MISSING on the DLE. I even tested with the holding disk disabled and it still says it is missing. The path is most certainly not missing from the filesystem. I also tried changing to another file path but still MISSING. It must be something in the database that got stuck, i will have to parse it myself and see if i can figure out what. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-18 02:34, John Lauro wrote: I normally use amflush to clear the holding disk. Not saying rm doesn't work, but that will not update the Amanda info database. It might think that space is still being used. On Sun, Apr 17, 2022 at 2:09 PM Exuvo wrote: I take it you did not read what i wrote. The holding disk is already empty after i 'rm -r':ed it. But even if i tell amanda to never use the holding disk for the failing DLE it just stops immediately when trying to run the planner (it normally takes a few minutes for that DLE). All other DLEs run fine.
Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
The disk was full of old partial dumps from testing of another backup set so i did not want to flush them to any tapes. The logs "driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896" indicate that it is finding the dump disk with correct size of 2TB free. Even if the dump drive was full i should be getting "FAILED [can't dump required holdingdisk when no holdingdisk space available ]" not MISSING on the DLE. I even tested with the holding disk disabled and it still says it is missing. The path is most certainly not missing from the filesystem. I also tried changing to another file path but still MISSING. It must be something in the database that got stuck, i will have to parse it myself and see if i can figure out what. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-18 02:34, John Lauro wrote: I normally use amflush to clear the holding disk. Not saying rm doesn't work, but that will not update the Amanda info database. It might think that space is still being used. On Sun, Apr 17, 2022 at 2:09 PM Exuvo wrote: I take it you did not read what i wrote. The holding disk is already empty after i 'rm -r':ed it. But even if i tell amanda to never use the holding disk for the failing DLE it just stops immediately when trying to run the planner (it normally takes a few minutes for that DLE). All other DLEs run fine.
Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
I take it you did not read what i wrote. The holding disk is already empty after i 'rm -r':ed it. But even if i tell amanda to never use the holding disk for the failing DLE it just stops immediately when trying to run the planner (it normally takes a few minutes for that DLE). All other DLEs run fine. Anton "exuvo" Olsson ex...@exuvo.se On 2022-04-17 01:54, badd...@ingodsfamily.com wrote: I have frequently done an "ls" on my holding disk, and manually deleted files and directories older than 7 days ago, which clearly amanda is not planning to flush to tape. 7 days because I ran amanda nightly. Your time window is up to you. I'm not sure 'amcleanup -rv backup' gets everything. I had an "onboot" job to send me email and remind me of that command, but perhaps I didn't reboot often, or perhaps amanda didn't register those files any more. If amanda hasn't flushed them during normal backups (assuming autoflush = ALL, i think it is), then it isn't planning to. Manual deletions never hurt my setup. If your holding disk isn't empty, but amanda sees nothing she wants to backup, then I suggest a manual cleanup. Deb Baddorf retired from Fermilab On Saturday, April 16, 2022, 01:27:15 PM EDT, Exuvo wrote: My backup failed tonight as i forgot to turn on my tape drive before i went to bed. Nothing was written to my dump drive as it was too full from a previous aborted backup i had done. When i woke up i turned on the tape drive and ran the backup again as i have done before when this has happend, most DLEs ran fine but the one DLE i have with 'holdingdisk required' failed as there was too little space left on the holding disk. I cleared out the holding disk and ran just that one failed DLE but it fails instantly and i get: FAILURE DUMP SUMMARY: planner: FATAL find_est_for_dp return NULL localhost storage RESULTS MISSING I tried running 'amcleanup -rv backup' but it did not do anything: # sudo -u amanda amcleanup -rv backup amcleanup: pid 2972312 is done amcleanup: pid 2972313 is done amcleanup: pid 2972311 is done amcleanup: pid 2972309 is done amcleanup: pid 2972305 is done amcleanup: no unprocessed logfile to clean up I also tested removing 'holdingdisk required' from the affected DLE but same result. Any ideas about what i should try? The logs don't seem to hold any more information (partial log from a RESULTS MISSING run): SENDING FLUSHES... Cleaning up holding disk '/dumps/' driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896 ENDFLUSH SETTING UP FOR ESTIMATES... reserving 1949288448 out of 1949288448 for degraded-mode dumps driver: taper taper0 storage backup-storage tape_size 1499463680 driver: started dumper0 pid 2972312 driver: send-cmd time 0.001 to dumper0: START 20220416185052 planner: time 0.000: setting up estimates for localhost:storage driver: started dumper1 pid 2972313 driver: send-cmd time 0.001 to dumper1: START 20220416185052 setup_estimate: localhost:storage: command 0, options: none last_level 3 next_level0 14 level_days 1 getting estimates 0 (-3) 3 (-3) -1 (-3) planner: time 0.001: setting up estimates took 0.000 secs GETTING ESTIMATES... planner: find_est_for_dp return NULL driver: send-cmd time 0.001 to taper0: START-TAPER taper0 worker0-0 backup-storage 20220416185052 more about starting dumpers which never get used here until quitting. -Mail report 03:14 with drive off- Hostname: minerva Org : exuvo Config : backup Date : april 16, 2022 *** A TAPE ERROR OCCURRED: ['/dev/tape/by-id/scsi-DEC8320699' not found]. Some dumps may have been left in the holding disk. The next 2 tapes Amanda expects to use are: ROT09, ROT07. FAILURE DUMP SUMMARY: localhost efi lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost boot lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost storage lev 3 FAILED [can't dump required holdingdisk when no holdingdisk space available ] localhost backup-minerva lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost exuvo-desktop-root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost backup lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] STATISTICS: Total Full Incr. Level:# Estimate Time (hrs:min) 0:15 Run Time (hrs:min) 0:15 Dump Time (hrs:min) 0:00 0:00 0:00 Output Size (meg) 0.0 0.0 0.0 Original Size (meg) 0.0 0.0 0.0 Avg Compressed Size (%) -- -- -- DLEs Dumped 0 0 0 Avg Dump Rate (k/s) -- -- -- Tape Time (hrs:min) 0:00 0:00 0:00 Tape Size (meg
Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
I have frequently done an "ls" on my holding disk, and manually deleted files and directories older than 7 days ago, which clearly amanda is not planning to flush to tape. 7 days because I ran amanda nightly. Your time window is up to you. I'm not sure 'amcleanup -rv backup' gets everything. I had an "onboot" job to send me email and remind me of that command, but perhaps I didn't reboot often, or perhaps amanda didn't register those files any more. If amanda hasn't flushed them during normal backups (assuming autoflush = ALL, i think it is), then it isn't planning to. Manual deletions never hurt my setup. If your holding disk isn't empty, but amanda sees nothing she wants to backup, then I suggest a manual cleanup. Deb Baddorf retired from Fermilab On Saturday, April 16, 2022, 01:27:15 PM EDT, Exuvo wrote: My backup failed tonight as i forgot to turn on my tape drive before i went to bed. Nothing was written to my dump drive as it was too full from a previous aborted backup i had done. When i woke up i turned on the tape drive and ran the backup again as i have done before when this has happend, most DLEs ran fine but the one DLE i have with 'holdingdisk required' failed as there was too little space left on the holding disk. I cleared out the holding disk and ran just that one failed DLE but it fails instantly and i get: FAILURE DUMP SUMMARY: planner: FATAL find_est_for_dp return NULL localhost storage RESULTS MISSING I tried running 'amcleanup -rv backup' but it did not do anything: # sudo -u amanda amcleanup -rv backup amcleanup: pid 2972312 is done amcleanup: pid 2972313 is done amcleanup: pid 2972311 is done amcleanup: pid 2972309 is done amcleanup: pid 2972305 is done amcleanup: no unprocessed logfile to clean up I also tested removing 'holdingdisk required' from the affected DLE but same result. Any ideas about what i should try? The logs don't seem to hold any more information (partial log from a RESULTS MISSING run): SENDING FLUSHES... Cleaning up holding disk '/dumps/' driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896 ENDFLUSH SETTING UP FOR ESTIMATES... reserving 1949288448 out of 1949288448 for degraded-mode dumps driver: taper taper0 storage backup-storage tape_size 1499463680 driver: started dumper0 pid 2972312 driver: send-cmd time 0.001 to dumper0: START 20220416185052 planner: time 0.000: setting up estimates for localhost:storage driver: started dumper1 pid 2972313 driver: send-cmd time 0.001 to dumper1: START 20220416185052 setup_estimate: localhost:storage: command 0, options: none last_level 3 next_level0 14 level_days 1 getting estimates 0 (-3) 3 (-3) -1 (-3) planner: time 0.001: setting up estimates took 0.000 secs GETTING ESTIMATES... planner: find_est_for_dp return NULL driver: send-cmd time 0.001 to taper0: START-TAPER taper0 worker0-0 backup-storage 20220416185052 more about starting dumpers which never get used here until quitting. -Mail report 03:14 with drive off- Hostname: minerva Org : exuvo Config : backup Date : april 16, 2022 *** A TAPE ERROR OCCURRED: ['/dev/tape/by-id/scsi-DEC8320699' not found]. Some dumps may have been left in the holding disk. The next 2 tapes Amanda expects to use are: ROT09, ROT07. FAILURE DUMP SUMMARY: localhost efi lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost boot lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost storage lev 3 FAILED [can't dump required holdingdisk when no holdingdisk space available ] localhost backup-minerva lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost exuvo-desktop-root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode] localhost backup lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode] STATISTICS: Total Full Incr. Level:# Estimate Time (hrs:min) 0:15 Run Time (hrs:min) 0:15 Dump Time (hrs:min) 0:00 0:00 0:00 Output Size (meg) 0.0 0.0 0.0 Original Size (meg) 0.0 0.0 0.0 Avg Compressed Size (%) -- -- -- DLEs Dumped 0 0 0 Avg Dump Rate (k/s) -- -- -- Tape Time (hrs:min) 0:00 0:00 0:00 Tape Size (meg) 0.0 0.0 0.0 Tape Used (%) 0.0 0.0 0.0 DLEs Taped 0 0 0 Parts Taped 0 0 0 Avg Tp Write Rate (k/s) -- -- -- NOTES: planner: Last full dump of localhost:efi on tape ROT05 overwritten in 4 runs. planner: Last full dump of localhost:boot on tape ROT05 overwr