Add limits for a volume so that it accepts only one job. On Wed, 16 Jul 2025 at 14:53, Leandro Saldivar via Bacula-users < bacula-users@lists.sourceforge.net> wrote:
> Hi Robert! Thanks for your time and input! > > What you suggested was exactly what came to my mind yesterday. I tested it > and saw something very strange: > I was able to start one Backup Job and 5 Copy Jobs in parallel, and > everything looked fine at first. However, on the remote SD, the file sizes > stopped growing independently, and a new file appeared that seems to be > aggregating all 5 Copy Jobs into a single volume: > > -rw-r----- 1 root root 4.6T Jul 16 12:35 mongodb01-2510 ← this should > be 1.5T max, seems to be merging copies > -rw-r----- 1 root root 190K Jul 15 20:05 mongodb01-2499 > -rw-r----- 1 root root 127K Jul 15 20:04 mongodb01-2495 > -rw-r----- 1 root root 64K Jul 15 20:04 mongodb01-2452 > -rw-r----- 1 root root 318 Jul 15 20:04 mongodb01-2441 > -rw-r----- 1 root root 44G Jul 15 20:03 mongodb01-2439 > -rw-r----- 1 root root 1.5T Jun 29 18:43 mongodb01-2273 ← expected > result > -rw-r----- 1 root root 1.5T Jun 29 00:55 mongodb01-2268 ← expected > result > > So it seems the volumes from the parallel Copy Jobs are being merged > unintentionally. > > I added an Autochanger with 5 Devices, declared a Storage using the > Autochanger, and pointed my Copy Jobs to this new Storage resource. > > You can find my config here: > https://gist.github.com/lean2206/54b37b8d599243f4b95e047fa4a8b47a > > Any idea what might be causing this behavior? Meanwhile, until I find a > better approach, I'll run the Restore Job using a RunAfterJob from the > Backup, and trigger the Copy Jobs via cron — both using separate Devices, > in parallel. So far, this setup would require 2 Devices. > > Thanks again! > > On Tue, Jul 15, 2025 at 2:03 PM Rob Gerber <r...@craeon.net> wrote: > >> Leandro, >> >> The following assumes you are using disk based backup, where the "drives" >> and "autochangers" are virtual. >> >> I would suggest creating multiple "drive" devices in your bacula-sd.conf >> for each SD destination, then tie them together with a single autochanger >> resource in those respective bacula-sd.conf files. In your bacula-dir.conf, >> in your storage resources, reference the respective autochanger resources. >> If your jobs are ran with the same priority, and all the various daemons >> have enough "allow simultaneous jobs" specified (not the actual option, I >> can't recall the exact wording used by Bacula), then you should see better >> parallelism. I believe you will then be limited by your disk / netwoek >> bandwidth, and single core performance on various Bacula tasks like hashing >> files. >> >> At least, unless I misunderstand something. >> >> I would provide config examples, but I am afk right now. >> >> Robert Gerber >> 402-237-8692 >> r...@craeon.net >> >> On Tue, Jul 15, 2025, 9:51 AM Leandro Saldivar via Bacula-users < >> bacula-users@lists.sourceforge.net> wrote: >> >>> Hi all, >>> >>> I have a client with ~2TB of data. A full backup job takes about 15 >>> hours. Once the backup finishes, a Copy Job is triggered (Selection Type = >>> PoolUncopiedJobs) to replicate the volume to a remote Storage Daemon — >>> which takes another 15 hours per volume in the best-case scenario. Finally, >>> for each backup, I run a Restore job on a replica server (same client >>> setup), which adds yet another 15 hours. >>> >>> The trivial approach to parallelize these jobs (Backup, Copy, Restore) >>> has been to define a separate Storage + Device pair for each Job. The only >>> difference between them is the Name. All Devices share the same Archive >>> Device path — which I'm unsure is a good practice. >>> >>> Is there a better way to parallelize Backup, Copy, and Restore jobs on >>> the same Pool/Volumes without defining a separate Storage/Device for each >>> Job? What would be considered a best-practice setup here? >>> >>> TL;DR >>> I need to run Backup, Copy, and Restore jobs in parallel for the same >>> client (Restore happens on a replica server). Ideally without creating >>> separate Storage/Device/Pool for each job. >>> >>> Bacula Version: 15.0.2 (21 March 2024) >>> >>> This is the config that currently allows me to parallelize only Backup >>> and Restore. Following this logic, I’d need to create additional Storage + >>> Device just for the Copy jobs. >>> >>> Client { >>> Name = "mongodb01-fd" >>> Address = xx.xx.xx.xx >>> FDPort = 9102 >>> Catalog = MyCatalog >>> Password = "PASSWORD" >>> File Retention = 11 days >>> Job Retention = 2 months >>> AutoPrune = yes >>> Heartbeat Interval = 1m >>> TLS Require = yes >>> TLS PSK Enable = yes >>> } >>> >>> Job { >>> Name = "mongodb01-Backup" >>> JobDefs = "MongoDef" >>> Pool = "mongodb01-baculaserver01-Pool" >>> Level = Full >>> Client = "mongodb01-fd" >>> Storage = "mongodb01_baculaserver01_Storage" >>> Schedule = "DisabledSchedule" # This is how I disable this client >>> so it doesn't affect others using the same JobDef with valid Schedules. I >>> launch the backup manually using a bconsole command. >>> @"/opt/bacula/etc/bacula-dir.conf.d/jobdefs/RunCopyToServer.jd" # >>> This is how I trigger Copy Jobs for clients with small backups. It works >>> well within a 24-hour window. For this client, I should remove this line. >>> } >>> >>> Pool { >>> Name = "mongodb01-baculaserver01-Pool" >>> Use Volume Once = yes >>> Pool Type = Backup >>> LabelFormat = "mongodb01-" >>> AutoPrune = yes >>> Recycle = yes >>> VolumeRetention = 30 days >>> Maximum Volumes = 365 >>> Maximum Volume Jobs = 1 >>> Recycle Oldest Volume = yes >>> Next Pool = "mongodb01-baculaserver02-Pool" # All copies go from >>> server01 to server02 >>> } >>> >>> Pool { >>> Name = "mongodb01-baculaserver02-Pool" >>> Use Volume Once = yes >>> Pool Type = Backup >>> LabelFormat = "mongodb01-" >>> AutoPrune = yes >>> Recycle = yes >>> VolumeRetention = 30 days >>> Maximum Volumes = 365 >>> Maximum Volume Jobs = 1 >>> Recycle Oldest Volume = yes >>> Storage = "mongodb01_baculaserver02_Storage" >>> Next Pool = "mongodb01-baculaserver01-Pool" # If backup runs on >>> server02, the copy goes to server01. Normally, all main backups are done on >>> server01. >>> } >>> >>> Storage { >>> Name = "mongodb01_baculaserver01_Storage" >>> Address = "baculaserver01.domain" >>> SDPort = 9103 >>> Password = "xxxxxxxxxxxxxxxxxxxxxxx" >>> Device = "mongodb01_baculaserver01_Device" >>> Media Type = File1 >>> Maximum Concurrent Jobs = "10" >>> Heartbeat Interval = 10 >>> } >>> >>> Storage { >>> Name = "mongodb01_baculaserver01_Storage_2" >>> Address = "baculaserver01.domain" >>> SDPort = 9103 >>> Password = "xxxxxxxxxxxxxxxxxxxxxxxx" >>> Device = "mongodb01_baculaserver01_Device_2" >>> Media Type = File1 >>> Maximum Concurrent Jobs = "10" >>> Heartbeat Interval = 10 >>> } >>> >>> Storage { >>> Name = "mongodb01_baculaserver02_Storage" >>> Address = "baculaserver02.domain" >>> SDPort = 9103 >>> Password = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" >>> Device = "mongodb01_baculaserver02_Device" >>> Media Type = File1 >>> Maximum Concurrent Jobs = "10" >>> Heartbeat Interval = 10 >>> } >>> >>> Device { >>> Name = "mongodb01_baculaserver01_Device" >>> Media Type = File1 >>> Archive Device = "/backup/bacula-storage/mongodb01/" >>> Maximum Concurrent Jobs = 10 >>> LabelMedia = yes >>> Random Access = Yes >>> AutomaticMount = yes >>> RemovableMedia = no >>> AlwaysOpen = yes >>> } >>> >>> Device { >>> Name = "mongodb01_baculaserver01_Device_2" >>> Media Type = File1 >>> Archive Device = "/backup/bacula-storage/mongodb01/" >>> Maximum Concurrent Jobs = 10 >>> LabelMedia = yes >>> Random Access = Yes >>> AutomaticMount = yes >>> RemovableMedia = no >>> AlwaysOpen = yes >>> } >>> >>> #This Device is stored in baculaserver02 where only a storage daemon is >>> running (not bacula-dir) >>> Device { >>> Name = "mongodb01_baculaserver02_Device" >>> Media Type = File1 >>> Archive Device = "/backup/bacula-storage/mongodb01/" >>> LabelMedia = yes; >>> Random Access = Yes; >>> AutomaticMount = yes; >>> RemovableMedia = no; >>> AlwaysOpen = no; >>> } >>> >>> >>> Backup is launched via cron every 3 days using: >>> echo 'run job=mongodb01-Backup >>> storage="mongodb01_baculaserver01_Storage_2" yes' | docker exec -i >>> bacula-dir /opt/bacula/bin/bconsole >>> Restore is triggered by another cron using the default Storage defined >>> in the Job. >>> >>> Any advice or shared experience would be greatly appreciated. >>> >>> Thanks, >>> _______________________________________________ >>> Bacula-users mailing list >>> Bacula-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bacula-users >>> >> _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users >
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
