Re: [Bacula-users] Deleting Obsolete Volumes
Yes that is true. I was concerned that a deleted volume of a job with no files would cause subsequent backups to fail. That doesn't seem to be the case. It is only incr/diffs that would get deleted since fulls are never empty. -Chris- On Fri, 14 Apr 2023, 17:57 Martin Simmons, wrote: > The problem is that the catalog doesn't record which jobs are on each > volume. > It uses the jobmedia table to record which files of a job are on each > volume > (in groups between firstindex and lastindex). As a result, if a job has no > files then it has no rows in the jobmedia table and hence is invisible to > 'query 14'. > > Why are you worried about deleting these empty volumes? There should be no > problem because they will never be used in a restore. > > __Martin > > > > On Thu, 13 Apr 2023 23:17:50 +0100, Chris Wilkinson said: > > > > Yes that is true. In the other cases I found, these were also zero file > > incr/diff jobs. > > > > However it appears that the job wrote a Volume as it appears in the list > > media result and there is such a Volume on disk but only 667 bytes so > only > > the label I presume. > > > > I had expected that 'query 14' would recognise that a Volume has an > > associated job even if that job wrote zero files to the volume but it > > doesn't. > > > > The script needs some additional condition for deletion to avoid deleting > > these zero file volumes. No idea what that might be right now. > > > > Chris- > > > > On Thu, 13 Apr 2023, 22:55 Bill Arlofski, wrote: > > > > > Hello Chris, > > > > > > Your jobid 2,371 wrote 0 files, 0 bytes, and the job's Summary shows > for > > > `Volume name(s):` No volumes. > > > > > > So, unless there are other jobs on this volume, jobid 2,371 did not > > > actually write to it. > > > > > > > > > Hope this helps, > > > Bill > > > > > > -- > > > Bill Arlofski > > > w...@protonmail.com > > > > > > -Chris- > > > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
The problem is that the catalog doesn't record which jobs are on each volume. It uses the jobmedia table to record which files of a job are on each volume (in groups between firstindex and lastindex). As a result, if a job has no files then it has no rows in the jobmedia table and hence is invisible to 'query 14'. Why are you worried about deleting these empty volumes? There should be no problem because they will never be used in a restore. __Martin > On Thu, 13 Apr 2023 23:17:50 +0100, Chris Wilkinson said: > > Yes that is true. In the other cases I found, these were also zero file > incr/diff jobs. > > However it appears that the job wrote a Volume as it appears in the list > media result and there is such a Volume on disk but only 667 bytes so only > the label I presume. > > I had expected that 'query 14' would recognise that a Volume has an > associated job even if that job wrote zero files to the volume but it > doesn't. > > The script needs some additional condition for deletion to avoid deleting > these zero file volumes. No idea what that might be right now. > > Chris- > > On Thu, 13 Apr 2023, 22:55 Bill Arlofski, wrote: > > > Hello Chris, > > > > Your jobid 2,371 wrote 0 files, 0 bytes, and the job's Summary shows for > > `Volume name(s):` No volumes. > > > > So, unless there are other jobs on this volume, jobid 2,371 did not > > actually write to it. > > > > > > Hope this helps, > > Bill > > > > -- > > Bill Arlofski > > w...@protonmail.com > > > > -Chris- > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
Yes that is true. In the other cases I found, these were also zero file incr/diff jobs. However it appears that the job wrote a Volume as it appears in the list media result and there is such a Volume on disk but only 667 bytes so only the label I presume. I had expected that 'query 14' would recognise that a Volume has an associated job even if that job wrote zero files to the volume but it doesn't. The script needs some additional condition for deletion to avoid deleting these zero file volumes. No idea what that might be right now. Chris- On Thu, 13 Apr 2023, 22:55 Bill Arlofski, wrote: > Hello Chris, > > Your jobid 2,371 wrote 0 files, 0 bytes, and the job's Summary shows for > `Volume name(s):` No volumes. > > So, unless there are other jobs on this volume, jobid 2,371 did not > actually write to it. > > > Hope this helps, > Bill > > -- > Bill Arlofski > w...@protonmail.com > > -Chris- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
Hello Chris, Your jobid 2,371 wrote 0 files, 0 bytes, and the job's Summary shows for `Volume name(s):` No volumes. So, unless there are other jobs on this volume, jobid 2,371 did not actually write to it. Hope this helps, Bill -- Bill Arlofski w...@protonmail.com signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
I have come across a problem with this script in that it has deleted some volumes that have an associated job. To test this out I ran the query and list media commands on a volume that the script reports as obsolete. From this I can see that 'query 14' reports the volume has no associated job. However 'list media pool=…' shows the volume is in the pool and the job log shows the volume is on the job. This has me stumped as query reports that there is no associated job, yet the evidence is that there is an associated job. This is the reason for the erroneous deletion of the volume. The implication of all this is that ‘query’ is not producing correct results. Are there some other tests you could suggest to verify ‘query’? pi@raspberrypi:~ $ bconsole Connecting to Director raspberrypi.fritz.box:9101 1000 OK: 10002 raspberrypi-dir Version: 11.0.6 (10 March 2022) Enter a period to cancel a command. *query 14 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" Enter Volume name: archiveusb2-incr1798 No results to list. *list media pool=archiveusb2-incr Using Catalog "MyCatalog" +-+--+---+-+--+--+--+-+--+---+---+-+--+-+---+ | mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | voltype | volparts | lastwritten | expiresin | +-+--+---+-+--+--+--+-+--+---+---+-+--+-+---+ | 1,798 | archiveusb2-incr1798 | Append| 1 | 667 |0 | 604,800 | 1 |0 | 0 | File | 1 |0 | 2023-04-13 17:42:41 | 590,791 | +-+--+---+-+--+--+--+-+--+---+---+-+--+-+---+ * *list joblog jobid=2371 ++ | logtext | ++ | raspberrypi-dir JobId 2371: Start Backup JobId 2371, Job=archive-usb2.2023-04-13_17.42.36_25 | | raspberrypi-dir JobId 2371: Created new Volume="archiveusb2-incr1798", Pool="archiveusb2-incr", MediaType="File" in catalog. | | raspberrypi-dir JobId 2371: Using Device "usb2" to write. | | raspberrypi-sd JobId 2371: Labeled new Volume "archiveusb2-incr1798" on File device "usb2" (/mnt/usb2/bacula). | | raspberrypi-sd JobId 2371: Wrote label to prelabeled Volume "archiveusb2-incr1798" on File device "usb2" (/mnt/usb2/bacula) | | raspberrypi-sd JobId 2371: Elapsed time=00:00:02, Transfer rate=0 Bytes/second | | raspberrypi-sd JobId 2371: Sending spooled attrs to the Director. Despooling 0 bytes ... | | raspberrypi-dir JobId 2371: Bacula raspberrypi-dir 11.0.6 (10Mar22): Build OS: aarch64-unknown-linux-gnu debian 11.3 JobId: 2371 Job:archive-usb2.2023-04-13_17.42.36_25 Backup Level: Incremental, since=2023-04-13 17:40:41 Client: "archive-fd" 11.0.6 (10Mar22) aarch64-unknown-linux-gnu,debian,11.3 FileSet:"archive" 2022-09-10 03:30:00 Pool: "archiveusb2-incr" (From Job IncPool override) Catalog:"MyCatalog" (From Pool resource) Storage:"usb2" (From Pool resource) Scheduled time: 13-Apr-2023 17:42:36 Start time: 13-Apr-2023 17:42:39 End time: 13-Apr-2023 17:42:41 Elapsed time: 2 secs Priority: 10 FD Files Written: 0 SD Files Written: 0 FD Bytes Written: 0 (0 B) SD Bytes Written: 0 (0 B) Rate: 0.0 KB/s Software Compression: None Comm Line Compression: None Snapshot/VSS: no Encryption: no Accurate: no Volume name(s): Volume Session Id: 20 Volume Session Time:1681209840 Last Volume Bytes: 667 (667 B) Non-fatal FD errors:0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK | | raspberrypi-dir JobId 2371: Begin pruning Jobs older than 7 days . | | raspberrypi-dir JobId 2371: Pruned 1 Job for client archive-fd from catalog. | | raspberrypi-dir JobId 2371: Begin pruning Files. | | raspberrypi-dir JobId 2371: No Files found to
Re: [Bacula-users] Deleting Obsolete Volumes
On 4/13/23 02:51, Chris Wilkinson wrote: This is the final mods to Bill's script. It scans the catalog for volumes without a job, deletes them from the catalog and disk. There were a lot of orphaned volumes and this recovered ~1TB from my storage. In my setup, all volumes are in one directory and therefore have unique names e.g. poolname-full1234 [...snip...]> Thank you to Bill You're welcome, glad I could help. :v: Best regards, Bill -- Bill Arlofski w...@protonmail.com signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
This is the final mods to Bill's script. It scans the catalog for volumes
without a job, deletes them from the catalog and disk. There were a lot of
orphaned volumes and this recovered ~1TB from my storage.
In my setup, all volumes are in one directory and therefore have unique
names e.g. poolname-full1234
#!/bin/bash
#list volumes with no associated jobs
#usage ./delete-vols-nojob.sh any-char
store="/mnt/usb2/bacula" #where you store volumes
for vol in $(echo "list media" | bconsole | grep "^| \+[0-9].*" | awk
'{print $4}'); do
echo -e "query 14\n${vol}\n" | bconsole | if grep --quiet "No results to
list"; then
if [ ! -z $1 ];then
echo -e "delete yes volume=${vol}" | bconsole
sudo rm -f ${store}/${vol}
fi
echo ${vol} "deleted"
fi
done
Thank you to Bill
On Wed, 12 Apr 2023, 6:06 pm Bill Arlofski via Bacula-users, <
bacula-users@lists.sourceforge.net> wrote:
> On 4/12/23 11:01, Bill Arlofski via Bacula-users wrote:
>
> Oops... I have a rogue `\` in those last two echo commands:
>
> > for vol in $(echo -e "list media pool=\n\quit\n" | "${bcbin}" -c
> "${bccfg}" \
>
> and
>
> > for vol in $(echo -e "list media\n\quit\n" | "${bcbin}" -c "${bccfg}" \
>
> There should not be the `\` before each `quit`
>
>
>
> --
> Bill Arlofski
> w...@protonmail.com
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
On 4/12/23 11:01, Bill Arlofski via Bacula-users wrote:
Oops... I have a rogue `\` in those last two echo commands:
for vol in $(echo -e "list media pool=\n\quit\n" | "${bcbin}" -c "${bccfg}"
\
and
for vol in $(echo -e "list media\n\quit\n" | "${bcbin}" -c "${bccfg}" \
There should not be the `\` before each `quit`
--
Bill Arlofski
w...@protonmail.com
signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
On 4/12/23 10:44, Chris Wilkinson wrote:
Thanks Bill.
I looked into using the query command but the standard queries prompts the user
to specify the volume name one by one. I
haven't yet been able to find a way to script this command by reading volume
names from storage so that the output can be fed
to the delete and rm commands.
My SQL and catalog structure knowledge is minimal but I'll try to figure out an
additional SQL to add to the query list.
Well, I guess it depends on if the volume names are unique from other ones... I
mean like if the volumes you want to check
are named like `Test_vol-` for example, or if they are all in the same pool
by themselves because then it is easy to do:
#!/bin/bash
for vol in $(echo -e "list media pool=\n\quit\n" | "${bcbin}" -c "${bccfg}"
\
| grep "^| \+[0-9]" | awk '{print $2}'); do
echo "Checking Volume: ${vol}"
echo -e "query\n14\n${vol}\n" | "${bcbin}" -c "${bccfg}" | grep "^+\|^|"
done
If they are not in the same pool, but are names similarly, and different from
other volumes, you can change that first echo
command to something like:
8<
for vol in $(echo -e "list media\n\quit\n" | "${bcbin}" -c "${bccfg}" \
| grep "^| \+[0-9].*| xx"| awk '{print $2}'); do
8<
Where xx is the beginning of these volume names
BTW, the query list in /etc/bacula/scripts is empty so I copied across the
query file from the distribution section.
Yes, this is normal. :)
Hope this helps,
Bill
--
Bill Arlofski
w...@protonmail.com
signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
Thanks Bill.
I looked into using the query command but the standard queries prompts the
user to specify the volume name one by one. I haven't yet been able to find
a way to script this command by reading volume names from storage so that
the output can be fed to the delete and rm commands.
My SQL and catalog structure knowledge is minimal but I'll try to figure
out an additional SQL to add to the query list.
BTW, the query list in /etc/bacula/scripts is empty so I copied across the
query file from the distribution section.
-Chris-
On Wed, 12 Apr 2023, 17:27 Bill Arlofski via Bacula-users, <
bacula-users@lists.sourceforge.net> wrote:
> On 4/11/23 16:35, Chris Wilkinson wrote:
> > I have some disk volumes that are not associated with any job but have
> not yet reached their expiry time so they are not
> > being pruned. These were created by a job run that was deleted. It seems
> deleting a job run doesn't purge the associated volumes.
> >
> > I would like to delete these now obsolete volumes from disk and recover
> the space used.
> >
> > I thought the prune command would do this but it doesn't find the
> obsolete volumes, presumably because they haven't reached
> > their expiry time. I thought perhaps the bls -j utility
> might help but it is extremely slow and produces much
> > unwanted output that is difficult to parse out.
> >
> > How could I list these obsolete volumes so I could script a delete from
> catalog and rm the volume files?
> >
> > Thanks
> > Chris Wilkinson
>
>
> Hello Chris,
>
> Bacula does its best to not delete tings, especially volumes/backup data.
> The rare case being the `ActionOnPurge = truncate`
> f
> eature.
>
> The answer in your case is simple:
>
> In bconsole:
> * delete yes volume=
>
> Then in a shell:
> # rm -f /path/to/filevolume
>
> This can be done with a simple script too (standard warnings apply):
> 8<
> #!/bin/bash
>
> bcbin=/path/to/bconsole
> bccfg=/path/to/bconsole.conf
> path2vols=/path/to/file/vols
>
> For vol in vol_1 vol_2 vol_3; do
>
>echo "delete yes volume="${vol}" | "${bcbin}" -c "${bccfg}"
>rm -f "$path2vols}"/"${vol}
>
> done
> 8<
>
> To be sure there are no jobs associated with a volume, I would recommend
> to use the bconsole `query` command, then choose
> option `14: List Jobs stored for a given Volume name`, and then do the
> deletions after you are sure all is OK.
>
>
> Hope this helps,
> Bill
>
> --
> Bill Arlofski
> w...@protonmail.com
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Obsolete Volumes
On 4/11/23 16:35, Chris Wilkinson wrote:
I have some disk volumes that are not associated with any job but have not yet
reached their expiry time so they are not
being pruned. These were created by a job run that was deleted. It seems
deleting a job run doesn't purge the associated volumes.
I would like to delete these now obsolete volumes from disk and recover the
space used.
I thought the prune command would do this but it doesn't find the obsolete
volumes, presumably because they haven't reached
their expiry time. I thought perhaps the bls -j utility might
help but it is extremely slow and produces much
unwanted output that is difficult to parse out.
How could I list these obsolete volumes so I could script a delete from catalog
and rm the volume files?
Thanks
Chris Wilkinson
Hello Chris,
Bacula does its best to not delete tings, especially volumes/backup data. The
rare case being the `ActionOnPurge = truncate`
f
eature.
The answer in your case is simple:
In bconsole:
* delete yes volume=
Then in a shell:
# rm -f /path/to/filevolume
This can be done with a simple script too (standard warnings apply):
8<
#!/bin/bash
bcbin=/path/to/bconsole
bccfg=/path/to/bconsole.conf
path2vols=/path/to/file/vols
For vol in vol_1 vol_2 vol_3; do
echo "delete yes volume="${vol}" | "${bcbin}" -c "${bccfg}"
rm -f "$path2vols}"/"${vol}
done
8<
To be sure there are no jobs associated with a volume, I would recommend to use
the bconsole `query` command, then choose
option `14: List Jobs stored for a given Volume name`, and then do the
deletions after you are sure all is OK.
Hope this helps,
Bill
--
Bill Arlofski
w...@protonmail.com
signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Deleting Obsolete Volumes
I have some disk volumes that are not associated with any job but have not yet reached their expiry time so they are not being pruned. These were created by a job run that was deleted. It seems deleting a job run doesn't purge the associated volumes. I would like to delete these now obsolete volumes from disk and recover the space used. I thought the prune command would do this but it doesn't find the obsolete volumes, presumably because they haven't reached their expiry time. I thought perhaps the bls -j utility might help but it is extremely slow and produces much unwanted output that is difficult to parse out. How could I list these obsolete volumes so I could script a delete from catalog and rm the volume files? Thanks Chris Wilkinson ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
