[Bacula-users] Script to find/delete disk volumes which are no longer used
Hi,
We are using Bacula to backup a number of VMs. The backup is being done
to disk volumes.
The configuration I've set up is still being tuned. In the meantime, I
found that there are a number of disk volumes which, which the file(s)
themselves are rather large, all the files on the jobs which are in that
volume have been expired, so the physical volume itself can be deleted
both from the database and the disk.
I've written a script which I've included below. I'd appreciate any
comments on it. Hopefully this will help someone else.
JBB
#!/bin/bash
#
# show usage of volumes (ie: how many files are on each volume
# which have not been purged/expired from the database)
#
# Usage:
# purgediskvolumes [-d dir] [volumename(s)]
#
# -d specify a custom directory
# volumename(s) Specific volume names to check
#
set -e
set -u
DBHOST=localhost
DBUSER=bacula
DB=bacula_db
FILEDIR="/mnt/baculaStorage"
# End of changable parameters
if [ "$1" = "-d" ]; then
if [ $# -lt 2 ]; then
echo "Missing directory name, exiting"
exit 1
fi
FILEDIR=$2
shift
shift
fi
if [ ! -d $FILEDIR ]; then
echo "$FILEDIR is not a directory, exiting"
exit 1
fi
START=$(date +%s)
RUN_ON_MYDB="/usr/bin/psql -t --no-align -X -U $DBUSER -h $DBHOST $DB"
files=`ls $FILEDIR`
[ $# -ne 0 ] && files="$*"
emptyvolumes=""
for volume in $files; do
if [ "$volume" != "lost+found" ]; then
#
# Get list of jobs on a volume
#
sql="SELECT DISTINCT Job.JobId
FROM JobMedia,Job,media
WHERE JobMedia.JobId=Job.JobId
AND JobMedia.MediaId=media.mediaid
AND media.volumename='$volume'"
jobids=`$RUN_ON_MYDB -c "$sql"`
#
# now get count of files in Db for each job
#
cnt=0
for jobid in $jobids; do
sql="SELECT count(*) FROM File,Filename,Path WHERE
File.JobId='$jobid' AND
Filename.FilenameId=File.FilenameId AND
Path.PathId=File.PathId"
numfiles=`$RUN_ON_MYDB -c "$sql"`
cnt=$((cnt + numfiles))
done
echo -e "Total files on volume $volume: $cnt\n"
[ $cnt -eq 0 ] && emptyvolumes="$volume $emptyvolumes"
fi
done
echo "Empty Volumes: $emptyvolumes"
cd $FILEDIR
ls -lh $emptyvolumes
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Searchtime: $DIFF seconds"
echo "Do you wish to delete these volumes from the database and the disk
(Y/n): "
read yn
[ "$yn" = "n" -o "$yn" = "N" ] && exit
#
# get the filesystem type, if it is an ext3 filesystem, then
# the default answer to the next question will be yes, otherwise
# it will be no
# This is needed since deleting a very big file on an ext3 filesystem
# can totally lock up the system until the delete is done
#
fstype=`df -TP "${FILEDIR}"`
def="y/N"
[ "$fstype" = "ext3" ] && def="Y/n"
echo "Do you need/want to delete the file by truncating in parts"
echo "to avoid locking up the system. This may be needed on ext3"
echo "filesystems when the files are very big. ($def): "
read trunc
if [ "$trunc" = "" ]; then
[ "$def" = "y/N" ] && trunc="n"
[ "$def" = "Y/n" ] && trunc="y"
fi
for vol in "$emptyvolumes"; do
echo "delete volume=$vol yes"
done | bconsole
if [ "$trunc" = "n" -o "$trunc" = "N" ]; then
for vol in "$emptyvolumes"; do
rm -f ${FILEDIR}/$vol
done
else
#
# use the truncate command to slowly shrink the file by 100 meg
# at a time. RHEL_5 based systems will need to get the
truncate command from
# RHEL6 or CentOS6
#
SLEEP=0.1
TRUNCATEBY=1 # 100 meg
trunc=`which truncate 2>/dev/null`
if [ "$trunc" = "" ]; then
echo "truncate command isn't available"
echo "Will just delete files instead"
fi
for vol in "$emptyvolumes"; do
file=${FILEDIR}/$vol
echo "Deleting $file"
if [ "$trunc" != "" ]; then
size=`ls -l $file |awk '{print $5}'`
while [ $size -gt $TRUNCATEBY ]; do
size=`echo $size-$TRUNCATEBY|bc`;
$trunc -c -s -$TRUNCATEBY $file ;
sleep $SLEEP
done
fi
rm -f $file
done
fi
--
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise servic
Re: [Bacula-users] Script to find/delete disk volumes which are no longer used (slight change)
If you were trying to use this, the "set -u" was misplaced. See below
for the correct location
JBB
On 6/7/2013 11:07 AM, Jonathan Bayer wrote:
Hi,
We are using Bacula to backup a number of VMs. The backup is being
done to disk volumes.
The configuration I've set up is still being tuned. In the meantime,
I found that there are a number of disk volumes which, which the
file(s) themselves are rather large, all the files on the jobs which
are in that volume have been expired, so the physical volume itself
can be deleted both from the database and the disk.
I've written a script which I've included below. I'd appreciate any
comments on it. Hopefully this will help someone else.
JBB
#!/bin/bash
#
# show usage of volumes (ie: how many files are on each volume
# which have not been purged/expired from the database)
#
# Usage:
# purgediskvolumes [-d dir] [volumename(s)]
#
# -d specify a custom directory
# volumename(s) Specific volume names to check
#
set -e
DBHOST=localhost
DBUSER=bacula
DB=bacula_db
FILEDIR="/mnt/baculaStorage"
# End of changable parameters
if [ "$1" = "-d" ]; then
if [ $# -lt 2 ]; then
echo "Missing directory name, exiting"
exit 1
fi
FILEDIR=$2
shift
shift
fi
if [ ! -d $FILEDIR ]; then
echo "$FILEDIR is not a directory, exiting"
exit 1
fi
set -u
START=$(date +%s)
RUN_ON_MYDB="/usr/bin/psql -t --no-align -X -U $DBUSER -h $DBHOST $DB"
files=`ls $FILEDIR`
[ $# -ne 0 ] && files="$*"
emptyvolumes=""
for volume in $files; do
if [ "$volume" != "lost+found" ]; then
#
# Get list of jobs on a volume
#
sql="SELECT DISTINCT Job.JobId
FROM JobMedia,Job,media
WHERE JobMedia.JobId=Job.JobId
AND JobMedia.MediaId=media.mediaid
AND media.volumename='$volume'"
jobids=`$RUN_ON_MYDB -c "$sql"`
#
# now get count of files in Db for each job
#
cnt=0
for jobid in $jobids; do
sql="SELECT count(*) FROM File,Filename,Path WHERE
File.JobId='$jobid' AND
Filename.FilenameId=File.FilenameId AND
Path.PathId=File.PathId"
numfiles=`$RUN_ON_MYDB -c "$sql"`
cnt=$((cnt + numfiles))
done
echo -e "Total files on volume $volume: $cnt\n"
[ $cnt -eq 0 ] && emptyvolumes="$volume $emptyvolumes"
fi
done
echo "Empty Volumes: $emptyvolumes"
cd $FILEDIR
ls -lh $emptyvolumes
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Searchtime: $DIFF seconds"
echo "Do you wish to delete these volumes from the database and the
disk (Y/n): "
read yn
[ "$yn" = "n" -o "$yn" = "N" ] && exit
#
# get the filesystem type, if it is an ext3 filesystem, then
# the default answer to the next question will be yes, otherwise
# it will be no
# This is needed since deleting a very big file on an ext3 filesystem
# can totally lock up the system until the delete is done
#
fstype=`df -TP "${FILEDIR}"`
def="y/N"
[ "$fstype" = "ext3" ] && def="Y/n"
echo "Do you need/want to delete the file by truncating in parts"
echo "to avoid locking up the system. This may be needed on ext3"
echo "filesystems when the files are very big. ($def): "
read trunc
if [ "$trunc" = "" ]; then
[ "$def" = "y/N" ] && trunc="n"
[ "$def" = "Y/n" ] && trunc="y"
fi
for vol in "$emptyvolumes"; do
echo "delete volume=$vol yes"
done | bconsole
if [ "$trunc" = "n" -o "$trunc" = "N" ]; then
for vol in "$emptyvolumes"; do
rm -f ${FILEDIR}/$vol
done
else
#
# use the truncate command to slowly shrink the file by 100 meg
# at a time. RHEL_5 based systems will need to get the
truncate command from
# RHEL6 or CentOS6
#
SLEEP=0.1
TRUNCATEBY=1 # 100 meg
trunc=`which truncate 2>/dev/null`
if [ "$trunc" = "" ]; then
echo "truncate command isn't available"
echo "Will just delete files instead"
fi
for vol in "$emptyvolumes"; do
file=${FILEDIR}/$vol
echo "Deleting $file"
if [ "$trunc" != "" ]; then
size=`ls -l $file |awk '{print $5}'`
while [ $size -gt $TRUNCATEBY ]; do
size=`echo $size-$TRUNCATEBY|bc`;
$trunc -c -s -$TRUNCATEBY $file ;
sleep $SLEEP
done
fi
rm -f $file
done
fi
--
How ServiceNow helps IT people tran
Re: [Bacula-users] Script to find/delete disk volumes which are no longer used (slight change)
Hello, On 6/7/2013 11:07 AM, Jonathan Bayer wrote: > > Hi, > > We are using Bacula to backup a number of VMs. The backup is being done > to disk volumes. > > The configuration I've set up is still being tuned. In the meantime, I > found that there are a number of disk volumes which, which the file(s) > themselves are rather large, all the files on the jobs which are in that > volume have been expired, so the physical volume itself can be deleted both > from the database and the disk. > > I've written a script which I've included below. I'd appreciate any > comments on it. Hopefully this will help someone else. > > Well, you can use Bacula internal truncate volumes functionality for that. You do not need to manually delete volumes. best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
