[Bacula-users] Job failed, how to recycle only volumes touched by this job?
Hi Bacula Users! A backup job I was running failed due to a power error, and I now have several hundred file volumes in a pool of hard drives which have this partial/failed backup job written to them. How can I manually tell bacula that it can discard this partial backup and start overwriting these volumes again without it overwriting other volumes in the same pool (which are storing a successfully completed full backup)? Thanks! Leon -- 陆智诚 | Leon White 绿色和平 | Greenpeace East Asia +86 186 0692 9781 | Skype: strophy 行动，带来改变。 Positive change through action. -- Try New Relic Now & We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, & servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Vchanger attempts to mount 5 volumes before stopping and requesting magazine change
Hi guys, sorry for the delay in replying. To clarify what I mean by 'without typing anything in the terminal', I was hoping to try out the web interface in the future to see if it can be used to start jobs. If the benablemag and update slots commands could be automated in some way, this would make the system sufficiently user friendly that I could just document the mechanism behind it and move on to other projects. Hopefully a few practice runs would be all someone needs to understand the process of switching the drives. Ideally, a message would pop up in the web UI saying 'drive full, insert next drive' or something similarly user friendly and idiot proof. That said, I am sticking to manually issuing the commands for now until I at least have a full backup completed. I decided not to use Bill's new script just yet, mainly because it takes a long time to cycle through all the non-mounted drives and disable them, even if they are already disabled. We have around 18 magazines (2TB drives have 2 magazines), each with 186 volumes, so it just took too long watching text scroll on the screen for what should be a quick drive switch operation. Instead, I issue something like 'benablemag 5 0' followed by 'benablemag 6 1', and set all drives to disabled immediately after formatting and adding the UUID to vchanger config. This has created a new problem: during the full backup which I have been running over the past days, if a drive sits in the changer for a few hours before the next drive in sequence is loaded, the bconsole messages indicate that bacula is automatically "Pruning oldest volume "gpea-backup-0010-0001"" or something similar. This is obviously because I haven't played with the volume recycling settings yet. Does this mean that the backup set is incomplete? Have the volumes been erased in some way, or just marked as next in line for recycling, but not overwritten? Thanks for the help, this is quite an adventure! Cheers, Leon On Wed, Apr 3, 2013 at 11:00 PM, Bill Arlofski wrote: > On 04/03/13 04:59, Leon White wrote: > > Hi Bill and Josh, > > > > I just want to say again how helpful your support and development skills > have > > been, not just for our system here but also helping a relative novice > like me > > get my head around scripting approaches! > > Happy to help out. :) > > > > Bill, your latest script serves to check which magazine is currently > inserted > > and then runs benablemag with the appropriate variable for that > magazine, correct? > > Yes. BUT - more importantly - it also calls Josh's benablemag script to > DISABLE all the volumes on any magazines that are not available at the time > you run my script. That is important to remember. > > > > Our use case here involves very infrequent backups (6 month cycle) of > very > > large amounts of data (30 TB). As such, we don't have much use for timed > jobs > > because we plan to run it manually once the hard drives have been > retrieved > > from safe storage every 6 months. > > Ahhh. I see. > > > > The main issue is the ability for a luddite > > to be able to switch drives in the autochanger "when the light stops > flashing" > > without typing anything into any terminal anywhere. > > Preparing the drives for > > first use and then disabling all volumes works fine with my clunky > scripts, > > but I wonder how to integrate the "benablemag $thismag 1" command with > udev so > > that a newly inserted drive is enabled by the system (rather than a > bacula > > job), followed by "bcommand update slots" or something similar (as this > > command currently also fails). This should only occur for drives passing > > through the /etc/auto.vchanger rule, or whatever the equivalent is in > udev > > (I'm still very fuzzy on this point). Then after the customary 3 minute > wait, > > backup would continue with no user interaction. Rinse and repeat until > the job > > is done. > > > Leon, I am confused as to when or how you intend to start a job. You do > mention "we plan to run it manually" but you also say "without typing > anything > into any terminal anywhere" > > Those two things are somewhat mutually exclusive. > > Even if it has to be "just unplug drive 1, plug in drive 2 and walk away" > simple, you still need a way to start a job. > > > If someone needs to manually start a job, then I would still say use a > RunScript. > > But instead of my previous recommendation of a scheduled "Admin" job to > update > the database of available volumes (using my script and Josh's), and a > bconsole > update slo
Re: [Bacula-users] Vchanger attempts to mount 5 volumes before stopping and requesting magazine change
Hi Bill and Josh,
I just want to say again how helpful your support and development skills
have been, not just for our system here but also helping a relative novice
like me get my head around scripting approaches!
I will be implementing a lot of this for our system over the coming days
(another Chinese holiday) and I want to make sure I have a few things
right. Bill, your latest script serves to check which magazine is currently
inserted and then runs benablemag with the appropriate variable for that
magazine, correct?
Our use case here involves very infrequent backups (6 month cycle) of very
large amounts of data (30 TB). As such, we don't have much use for timed
jobs because we plan to run it manually once the hard drives have been
retrieved from safe storage every 6 months. The main issue is the ability
for a luddite to be able to switch drives in the autochanger "when the
light stops flashing" without typing anything into any terminal anywhere.
Preparing the drives for first use and then disabling all volumes works
fine with my clunky scripts, but I wonder how to integrate the "benablemag
$thismag 1" command with udev so that a newly inserted drive is enabled by
the system (rather than a bacula job), followed by "bcommand update slots"
or something similar (as this command currently also fails). This should
only occur for drives passing through the /etc/auto.vchanger rule, or
whatever the equivalent is in udev (I'm still very fuzzy on this point).
Then after the customary 3 minute wait, backup would continue with no user
interaction. Rinse and repeat until the job is done.
Sorry this is a little long! Takes me a while to think through it all. If
either of you wants to pick this up as a supported job, we do have some
limited funding available :)
Cheers from Beijing,
Leon
On Wed, Apr 3, 2013 at 8:36 AM, Bill Arlofski wrote:
> Leon,
>
> After replying to you earlier today, I spent a little time testing Josh's
> benablemag script, and thought about my "just create a script to run in an
> Admin job" comment.
>
> Thought that it required a little more information.
>
> So, here is my Admin job called "UpdateSlots" which runs before all of my
> other jobs each night.
>
> --[snip]--
> #
> # Update Slots Job
> #
> Job {
> Name = UpdateSlots
> Client = None
> Type = Admin
> FileSet = None
> Storage = c0
> Schedule = WeeklyToOffsiteDisk
> Messages = Standard
> Priority = 8
> SpoolData = no
> Pool = Default
>
> RunScript {
> RunsWhen = Before
> RunsOnClient = no
> Fail Job On Error = no
> Command =
> "/etc/bacula/include/mag_and_vol_scripts/auto_enable_disable_vols.sh"
> Command = "/etc/bacula/include/runbefore-updateslots.sh"
> }
> }
> --[snip]--
>
> Here is the runbefore-updateslots.sh script that is called:
>
> --[snip]--
> #!/bin/bash
> #
> # waa - 20091110
> # --
> BC="/usr/sbin/bconsole"
> CONF="/path/to/bconsole.conf"
> STORAGE="c0"
> DRIVE="0"
>
> $BC -c $CONF << WAAEOF
> update slots storage=$STORAGE drive=$DRIVE
> quit
> WAAEOF
> --[snip]--
>
>
> And here is the auto_enable_disable_vols.sh script I am using starting
> today.
>
> http://www.revpol.com/node/146
>
> Hope this helps - someone. :)
>
> --
> Bill Arlofski
> Reverse Polarity, LLC
>
--
陆智诚 | Leon White
绿色和平 | Greenpeace East Asia
+86 186 0692 9781 | Skype: strophy
行动，带来改变。 Positive change through action.
--
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Vchanger attempts to mount 5 volumes before stopping and requesting magazine change
Hi Josh, thanks for taking the time once again to respond to support my bacula/vchanger project! I was able to read, set up and configure the scripts you supplied, the variables are quite clear and anybody who set up their own bacula system should be able to fill them in. Note that it may be necessary to change the #!/bin/sh at the beginning of the files to #!/bin/bash depending on your system (mine is Ubuntu Server 12.10). I plan to use this script to ensure that all volumes are set to disabled before and after they are used to prevent bacula from trying to use unmounted volumes. Is there any way to trigger this script to run silently as soon as the update slots command is triggered, or is there some other point we could trigger it during the process of swapping drives in the eSATA changer? Perhaps automount could do it and then subsequently use the bcommand script to call update slots in bconsole as well, making the process of changing drives effectively fully automatic and independent from the shell. I won't be operating this system forever, and I want to keep the process as simple as possible for anybody else who has to use it. Thanks mate, Leon On Tue, Mar 26, 2013 at 12:13 AM, Josh Fisher wrote: > > On 3/22/2013 5:59 AM, Leon White wrote: > >> Hi list! >> >> I am using Bacula 5.2.6 with vchanger 0.8.6 to backup a multi-terabyte >> fileset to a bunch of 1 and 2TB drives in an eSATA virtual autochanger. I >> have formatted each drive with one or two 2TB partitions with 186x 5GB >> volume slots, and vchanger switches between 2 magazine bays when necessary >> for the 2TB drives. >> >> My problem is that when bacula hits the last writable volume in the >> virtual magazine, it attempts to load the next appendable volume from the >> next available magazine. Obviously, I haven't changed the drive yet, and >> bacula attempts this in vain 4 more times before pausing the job and >> requesting that I mount the next magazine. I then switch the drives in the >> changer, issue 'update slots', and bacula updates all the slot data. I >> thought it would then continue from the next volume it initially tried to >> mount, but instead it skips the first five volumes and starts writing the >> sixth. These volumes are marked as 'error' and not used. It looks like this: >> http://i.troll.ws/e60e9955.png >> >> This is what hitting the end of the magazine looks like (screenshot shows >> jump to next appendable volume, which happens not to be 0001): >> http://i.troll.ws/250d9942.png >> >> Is there a way to tell bacula to stop and ask for a mount when it uses >> the last volume identified by update slots, and immediately request the >> magazine change? Or to step back to the first volume in a magazine after >> mounting and issuing update slots? >> > > The trick is to disable a drive's volumes when it is not attached and > enable them when attached. This can be done with the update volume=??? > enabled=? command. Bacula will not consider a disabled volume available for > use. I use the attached scripts for vchanger autochangers. They allow > enabling/disabling all volumes on a magazine drive or only volumes on a > magazine that are in a specified pool. > > > > -- > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_mar > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > > -- 陆智诚 | Leon White 绿色和平 | Greenpeace East Asia +86 186 0692 9781 | Skype: strophy 行动，带来改变。 Positive change through action. -- Own the Future-Intel® Level Up Game Demo Contest 2013 Rise to greatness in Intel's independent game demo contest. Compete for recognition, cash, and the chance to get your game on Steam. $5K grand prize plus 10 genre and skill prizes. Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Vchanger attempts to mount 5 volumes before stopping and requesting magazine change
Hi list! I am using Bacula 5.2.6 with vchanger 0.8.6 to backup a multi-terabyte fileset to a bunch of 1 and 2TB drives in an eSATA virtual autochanger. I have formatted each drive with one or two 2TB partitions with 186x 5GB volume slots, and vchanger switches between 2 magazine bays when necessary for the 2TB drives. My problem is that when bacula hits the last writable volume in the virtual magazine, it attempts to load the next appendable volume from the next available magazine. Obviously, I haven't changed the drive yet, and bacula attempts this in vain 4 more times before pausing the job and requesting that I mount the next magazine. I then switch the drives in the changer, issue 'update slots', and bacula updates all the slot data. I thought it would then continue from the next volume it initially tried to mount, but instead it skips the first five volumes and starts writing the sixth. These volumes are marked as 'error' and not used. It looks like this: http://i.troll.ws/e60e9955.png This is what hitting the end of the magazine looks like (screenshot shows jump to next appendable volume, which happens not to be 0001): http://i.troll.ws/250d9942.png Is there a way to tell bacula to stop and ask for a mount when it uses the last volume identified by update slots, and immediately request the magazine change? Or to step back to the first volume in a magazine after mounting and issuing update slots? Thanks! Leon -- 陆智诚 | Leon White 绿色和平 | Greenpeace East Asia +86 186 0692 9781 | Skype: strophy 行动，带来改变。 Positive change through action. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] vchanger device not in SD Device resources
Hi again,
my apologies for posting this with the incorrect subject line. After a lot
of research, I found that the "No volumes found to label" error was caused
by an error in the group configuration for the user which bacula-sd runs
as. I have been following the fantastic instructions at
http://www.revpol.com/node/140?page=show which suggest running vchanger as
user=bacula and group=disk. However, bacula-sd could not access the files
referenced by the barcodes because the default primary group for the user
bacula was set to group=bacula. On some systems this may also be
group=tape. The solution is to ensure that the user/group combination for
the volume owner, bacula-sd and vchanger are all the same.
You can view the user as which daemons are running using "ps aux | grep
bacula".
I'm not particularly adept at Linux, so I found these instructions on
setting managing users and groups particularly useful.
http://www.howtogeek.com/50787/add-a-user-to-a-group-or-second-group-on-linux/
Cheers,
Leon
On Mon, Feb 25, 2013 at 12:53 PM, Leon White wrote:
> Hi,
>
> I am new to bacula and trying to set up a system which will backup our 3
> NAS towers to disks using the vchanger utility. We use an eSata changer
> with 1 drive only - quite a simple configuration. It took a while, but
> vchanger now works as advertised in the testing chapter of the
> documentation, and I am currently testing it with only 3 small slots
> configured to see how the drive swapping works in practice. However, I am
> not able to convince bacula to label the volumes using the label barcodes
> or update slots commands, and backup jobs hang waiting for me to issue a
> label command. The documentation informs me I should not enter volume
> labels for autochangers manually.
>
> The errors:
>
> *label barcodes
> Automatically selected Catalog: MyCatalog
> Using Catalog "MyCatalog"
> Automatically selected Storage: gpea-backup
> Connecting to Storage daemon gpea-backup at 192.168.85.55:9103 ...
> 3306 Issuing autochanger "slots" command.
> Device "vantec-nexstar" has 3 slots.
> Connecting to Storage daemon gpea-backup at 192.168.85.55:9103 ...
> 3306 Issuing autochanger "list" command.
> No Volumes found to label, or no barcodes.
>
> Following a run command:
> 25-Feb 12:47 dell-backup-dir JobId 3: Start Backup JobId 3,
> Job=BackupVideo3.2013-02-25_12.47.37_09
> 25-Feb 12:47 dell-backup-dir JobId 3: Using Device "vantec-drive-0"
> 25-Feb 12:47 dell-backup-sd JobId 3: Job
> BackupVideo3.2013-02-25_12.47.37_09 is waiting. Cannot find any appendable
> volumes.
> Please use the "label" command to create a new Volume for:
> Storage: "vantec-drive-0" (/var/lib/bacula/gpea-backup/0/drive0)
> Pool: File
> Media type: File
>
> However, running the vchanger list command shows that vchanger is in fact
> outputting 3 barcodes:
> strophy@dell-backup:/$ sudo vchanger -u bacula -g disk
> /etc/bacula/gpea-backup.conf list
> 1:gpea-backup_0001_0001
> 2:gpea-backup_0001_0002
> 3:gpea-backup_0001_0003
>
> Some config files:
>
> bacula-dir.conf:
> Storage {
> Name = gpea-backup
> Address = 192.168.85.55
> SDPort= 9103
> Password = "mH2Ps4Jim2UfNMxko_SQqxqTpuGSjJ7Up"
> Device= vantec-nexstar
> Media Type= File
> Autochanger = yes;
> }
>
> bacula-sd.conf
> Autochanger {
> Name = vantec-nexstar
> Device = vantec-drive-0
> Changer Command = "/usr/local/bin/vchanger %c %o %S %a %d"
> Changer Device = "/etc/bacula/gpea-backup.conf"
> }
>
> Device {
> Name = vantec-drive-0
> Drive Index = 0
> Autochanger = yes;
> Device Type = File
> Media Type = File
> Archive Device = /var/lib/bacula/gpea-backup/0/drive0
> RemovableMedia = no;
> RandomAccess = yes;
> }
>
> gpea-backup.conf (the vchanger configuration script):
> changer_name = gpea-backup
> virtual_drives = 1
> slots_per_magazine = 3
> magazine_bays = 1
> automount_dir = /mnt/vchanger
> logfile= /var/lib/bacula/vchanger.log
> log_level = LOG_ERR
> magazine = "UUID:ff4bd90a-b431-431e-b7d6-a9a323d4f5ee"
> magazine = "UUID:aa87944c-3a50-4c1d-a6dd-daca9761603d"
>
>
> Thanks for helping with this, let me know if more details are required!
> Leon
>
> --
> Leon White -- Greenpeace East Asia, Beijing
>
--
Leon White -- Greenpeace East Asia, Beijing
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] vchanger device not in SD Device resources
Hi,
I am new to bacula and trying to set up a system which will backup our 3
NAS towers to disks using the vchanger utility. We use an eSata changer
with 1 drive only - quite a simple configuration. It took a while, but
vchanger now works as advertised in the testing chapter of the
documentation, and I am currently testing it with only 3 small slots
configured to see how the drive swapping works in practice. However, I am
not able to convince bacula to label the volumes using the label barcodes
or update slots commands, and backup jobs hang waiting for me to issue a
label command. The documentation informs me I should not enter volume
labels for autochangers manually.
The errors:
*label barcodes
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
Automatically selected Storage: gpea-backup
Connecting to Storage daemon gpea-backup at 192.168.85.55:9103 ...
3306 Issuing autochanger "slots" command.
Device "vantec-nexstar" has 3 slots.
Connecting to Storage daemon gpea-backup at 192.168.85.55:9103 ...
3306 Issuing autochanger "list" command.
No Volumes found to label, or no barcodes.
Following a run command:
25-Feb 12:47 dell-backup-dir JobId 3: Start Backup JobId 3,
Job=BackupVideo3.2013-02-25_12.47.37_09
25-Feb 12:47 dell-backup-dir JobId 3: Using Device "vantec-drive-0"
25-Feb 12:47 dell-backup-sd JobId 3: Job
BackupVideo3.2013-02-25_12.47.37_09 is waiting. Cannot find any appendable
volumes.
Please use the "label" command to create a new Volume for:
Storage: "vantec-drive-0" (/var/lib/bacula/gpea-backup/0/drive0)
Pool: File
Media type: File
However, running the vchanger list command shows that vchanger is in fact
outputting 3 barcodes:
strophy@dell-backup:/$ sudo vchanger -u bacula -g disk
/etc/bacula/gpea-backup.conf list
1:gpea-backup_0001_0001
2:gpea-backup_0001_0002
3:gpea-backup_0001_0003
Some config files:
bacula-dir.conf:
Storage {
Name = gpea-backup
Address = 192.168.85.55
SDPort= 9103
Password = "mH2Ps4Jim2UfNMxko_SQqxqTpuGSjJ7Up"
Device= vantec-nexstar
Media Type= File
Autochanger = yes;
}
bacula-sd.conf
Autochanger {
Name = vantec-nexstar
Device = vantec-drive-0
Changer Command = "/usr/local/bin/vchanger %c %o %S %a %d"
Changer Device = "/etc/bacula/gpea-backup.conf"
}
Device {
Name = vantec-drive-0
Drive Index = 0
Autochanger = yes;
Device Type = File
Media Type = File
Archive Device = /var/lib/bacula/gpea-backup/0/drive0
RemovableMedia = no;
RandomAccess = yes;
}
gpea-backup.conf (the vchanger configuration script):
changer_name = gpea-backup
virtual_drives = 1
slots_per_magazine = 3
magazine_bays = 1
automount_dir = /mnt/vchanger
logfile= /var/lib/bacula/vchanger.log
log_level = LOG_ERR
magazine = "UUID:ff4bd90a-b431-431e-b7d6-a9a323d4f5ee"
magazine = "UUID:aa87944c-3a50-4c1d-a6dd-daca9761603d"
Thanks for helping with this, let me know if more details are required!
Leon
--
Leon White -- Greenpeace East Asia, Beijing
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
