Hello, There doesn't seem to be a lot of documentation for copy jobs, although I think they are extremely practical when having 2 storage daemons, and backing up remote clients to only use bandwidth once.
Problem: whenever I (manually, or scheduled) run a copy job, bareos actually runs 3 jobs: 1) one "copy job" named "myserver-sys-copy-file-to-tape" which only selects the job to be copied, and runs for not even a second 2) one "copy job" named "myserver-sys-copy-file-to-tape" as well, which seems to be actual copy job (it has the read, and write storage attributes) 3) one "copy of a job" named "myserver-sys-file" which is initially empty, but then fills with all the log-data of the initial job which was copied (also having the jobid of the file-backup which is copied to the tape) Is this "as intended", or can I improve my config so I declutter my inbox/messages when doing a copy? I plan to have a couple of copy jobs which copies the last Full backup from file to tape. I could try one copy job for all remote clients (doing e.g. a SQL group by/distinct by client-name, and limiting the result to the amount of clients I want to backup), but I guess it would only save me one email/job per client (so for 5 clients, I would still have 11 emails). It would be great if somebody can have a look at my config and point out what is wrong, so I only get (ideally) 1 email per copy job? ==== Here are the logs of the 3 jobs ==== 1) copy job which seems to select the jobid to be copied: *Director Running Jobs* Running Jobs: Console connected at 29-Apr-22 12:38 Console connected at 29-Apr-22 12:38 JobId Level Name Status ====================================================================== 1084 Increme myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 is waiting on max Job jobs 1085 Full myserver-sys-file.2022-04-29_12.38.35_12 is waiting execution ==== *Joblog* Connecting to Director localhost:9101 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021) bareos.org build binary bareos.org binaries are UNSUPPORTED by bareos.com. Get official binaries and vendor support on https://www.bareos.com You are connected using the default console Enter a period (.) to cancel a command. list joblog jobid=1083 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" 2022-04-29 12:38:35 bareos-dir JobId 1083: The following 1 JobId was chosen to be copied: 1069 2022-04-29 12:38:35 bareos-dir JobId 1083: Automatically selected Catalog: MyCatalog 2022-04-29 12:38:35 bareos-dir JobId 1083: Using Catalog "MyCatalog" 2022-04-29 12:38:35 bareos-dir JobId 1083: Job queued. JobId=1084 2022-04-29 12:38:35 bareos-dir JobId 1083: Copying JobId 1084 started. 2022-04-29 12:38:35 bareos-dir JobId 1083: Bareos bareos-dir 21.0.0 (21Dec21): Build OS: Ubuntu 20.04.3 LTS Current JobId: 1083 Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.33_06 Catalog: "MyCatalog" (From Default catalog) Start time: 29-Apr-2022 12:38:35 End time: 29-Apr-2022 12:38:35 Elapsed time: 0 secs Priority: 20 Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com Job triggered by: User Termination: Copying OK You have messages. *Files backed up* You have messages. 2) the actual copy job between the 2 storage daemons *Director Running Jobs* Running Jobs: Console connected at 29-Apr-22 12:38 Console connected at 29-Apr-22 12:39 JobId Level Name Status ====================================================================== 1085 Full myserver-sys-file.2022-04-29_12.38.35_12 has terminated ==== *Joblog* Connecting to Director localhost:9101 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021) bareos.org build binary bareos.org binaries are UNSUPPORTED by bareos.com. Get official binaries and vendor support on https://www.bareos.com You are connected using the default console Enter a period (.) to cancel a command. list joblog jobid=1084 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" 2022-04-29 12:38:35 bareos-dir JobId 1084: Copying using JobId=1069 Job=myserver-sys-file.2022-04-29_12.01.16_47 2022-04-29 12:38:35 bareos-dir JobId 1084: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.9.bsr 2022-04-29 12:38:37 bareos-dir JobId 1084: Start Copying JobId 1084, Job=myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 2022-04-29 12:38:37 bareos-dir JobId 1084: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:38:37 bareos-dir JobId 1084: Using Device "bareos-file" to read. 2022-04-29 12:38:37 bareos-dir JobId 1085: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:38:37 bareos-dir JobId 1085: Using Device "Ultrium1760" to write. 2022-04-29 12:38:37 bareos-sd JobId 1084: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:38:37 bareos-sd JobId 1084: Ready to read from volume "Full-0154" on device "bareos-file" (/mnt/leah/var_lib_bareos_storage). 2022-04-29 12:38:37 eljadid-sd JobId 1085: Spooling data ... 2022-04-29 12:38:37 bareos-sd JobId 1084: Forward spacing Volume "Full-0154" to file:block 3:1118992026. 2022-04-29 12:38:57 bareos-sd JobId 1084: End of Volume at file 3 on device "bareos-file" (/mnt/leah/var_lib_bareos_storage), Volume "Full-0154" 2022-04-29 12:38:57 bareos-sd JobId 1084: End of all volumes. 2022-04-29 12:38:57 eljadid-sd JobId 1085: Committing spooled data to Volume "XM2724L3". Despooling 931,953,968 bytes ... 2022-04-29 12:39:17 eljadid-sd JobId 1085: Despooling elapsed time = 00:00:20, Transfer rate = 46.59 M Bytes/second 2022-04-29 12:39:17 eljadid-sd JobId 1085: Releasing device "Ultrium1760" (/dev/tape/by-id/scsi-HU19326DV6). 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-109-generic] (local build) 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: === START OF READ SMART DATA SECTION === 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: TapeAlert Not Supported 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Error Counter logging not supported 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: 2022-04-29 12:39:19 eljadid-sd JobId 1085: Elapsed time=00:00:42, Transfer rate=22.07 M Bytes/second 2022-04-29 12:39:19 eljadid-sd JobId 1085: Sending spooled attrs to the Director. Despooling 24,595,188 bytes ... 2022-04-29 12:39:21 bareos-sd JobId 1084: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage). 2022-04-29 12:39:21 bareos-dir JobId 1085: Insert of attributes batch table with 74025 entries start 2022-04-29 12:39:23 bareos-dir JobId 1085: Insert of attributes batch table done 2022-04-29 12:39:23 bareos-dir JobId 1084: Bareos bareos-dir 21.0.0 (21Dec21): Build OS: Ubuntu 20.04.3 LTS Prev Backup JobId: 1069 Prev Backup Job: myserver-sys-file.2022-04-29_12.01.16_47 New Backup JobId: 1085 Current JobId: 1084 Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 Backup Level: Incremental Client: myserver-fd FileSet: "myserver-sys-file" Read Pool: "Full" (From Job resource) Read Storage: "bareos-file" (From Pool resource) Write Pool: "tape-automated" (From Job's NextPool resource) Write Storage: "eljadid_HP_1_8_G2" (From Storage from Job's NextPool resource) Next Pool: "tape-automated" (From Job's NextPool resource) Catalog: "MyCatalog" (From Default catalog) Start time: 29-Apr-2022 12:38:37 End time: 29-Apr-2022 12:39:23 Elapsed time: 46 secs Priority: 20 SD Files Written: 74,025 SD Bytes Written: 927,271,942 (927.2 MB) Rate: 20158.1 KB/s Volume name(s): XM2724L3 Volume Session Id: 10 Volume Session Time: 1651214033 Last Volume Bytes: 19,536,942,080 (19.53 GB) SD Errors: 0 SD termination status: OK Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com Job triggered by: User Termination: Copying OK You have messages. *Files backed up* You have messages. 3) the "copy of a job" with a new jobid (1085 in this case), but the logs of the job which is copied *Director Running Jobs* Running Jobs: Console connected at 29-Apr-22 13:01 No Jobs running. ==== *Joblog* Connecting to Director localhost:9101 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021) bareos.org build binary bareos.org binaries are UNSUPPORTED by bareos.com. Get official binaries and vendor support on https://www.bareos.com You are connected using the default console Enter a period (.) to cancel a command. list joblog jobid=1085 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" 2022-04-29 12:01:18 bareos-dir JobId 1069: Start Backup JobId 1069, Job=myserver-sys-file.2022-04-29_12.01.16_47 2022-04-29 12:01:18 bareos-dir JobId 1069: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:01:18 bareos-dir JobId 1069: Using Device "bareos-file" to write. 2022-04-29 12:01:18 bareos-dir JobId 1069: Probing client protocol... (result will be saved until config reload) 2022-04-29 12:01:22 bareos-dir JobId 1069: Connected Client: myserver-fd at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:01:23 bareos-dir JobId 1069: Handshake: Immediate TLS 2022-04-29 12:01:23 bareos-dir JobId 1069: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:01:25 myserver-fd JobId 1069: shell command: run ClientBeforeJob "/usr/local/bin/bareos-runbefore.sh" 2022-04-29 12:01:25 myserver-fd JobId 1069: Extended attribute support is enabled 2022-04-29 12:01:25 myserver-fd JobId 1069: ACL support is enabled 2022-04-29 12:01:23 bareos-sd JobId 1069: Connected File Daemon at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 2022-04-29 12:01:25 bareos-sd JobId 1069: Volume "Full-0154" previously written, moving to end of data. 2022-04-29 12:01:25 bareos-sd JobId 1069: Ready to append to end of Volume "Full-0154" size=14003893914 2022-04-29 12:25:39 bareos-sd JobId 1069: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage). 2022-04-29 12:25:39 bareos-sd JobId 1069: Elapsed time=00:24:14, Transfer rate=637.7 K Bytes/second 2022-04-29 12:25:39 bareos-dir JobId 1069: Insert of attributes batch table with 74025 entries start 2022-04-29 12:25:41 bareos-dir JobId 1069: Insert of attributes batch table done 2022-04-29 12:25:41 bareos-dir JobId 1069: Bareos bareos-dir 21.0.0 (21Dec21): Build OS: Ubuntu 20.04.3 LTS JobId: 1069 Job: myserver-sys-file.2022-04-29_12.01.16_47 Backup Level: Full Client: "myserver-fd" 21.0.0 (21Dec21) Debian GNU/Linux 11 (bullseye),debian FileSet: "myserver-sys-file" 2022-04-28 17:29:32 Pool: "Full" (From command line) Catalog: "MyCatalog" (From Client resource) Storage: "bareos-file" (From Pool resource) Scheduled time: 29-Apr-2022 12:01:16 Start time: 29-Apr-2022 12:01:25 End time: 29-Apr-2022 12:25:41 Elapsed time: 24 mins 16 secs Priority: 10 FD Files Written: 74,025 SD Files Written: 74,025 FD Bytes Written: 881,441,457 (881.4 MB) SD Bytes Written: 927,271,942 (927.2 MB) Rate: 605.4 KB/s Software Compression: 61.5 % (lz4) VSS: no Encryption: yes Accurate: no Volume name(s): Full-0154 Volume Session Id: 4 Volume Session Time: 1651214033 Last Volume Bytes: 14,936,345,934 (14.93 GB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: OK SD termination status: OK Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com Job triggered by: User Termination: Backup OK 2022-04-29 12:25:41 bareos-dir JobId 1069: shell command: run AfterJob "/bin/bash -c '/bin/echo "run myserver-sys-file-v yes" | bconsole >/dev/null'" You have messages. *Files backed up* ==== Here is my config ==== Job { #the job which backs up the remote client to file/disk Name = "myserver-sys-file" Type = Backup Level = Incremental Client = "myserver-fd" FileSet = myserver-sys-file Schedule = myserver-sys-file Storage = bareos-file Messages = Standard Maximum Concurrent Jobs = 7 Pool = Incremental Priority = 10 Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr" Allow Mixed Priority = yes #prevent duplicate jobs Allow Duplicate Jobs = no Cancel Lower Level Duplicates = yes Cancel Queued Duplicates = yes Full Backup Pool = Full # write Full Backups into "Full" Pool (#05) Differential Backup Pool = Differential # write Diff Backups into "Differential" Pool (#08) Incremental Backup Pool = Incremental # write Incr Backups into "Incremental" Pool (#11) RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-file-v yes\" | bconsole >/dev/null'" RunScript { Command = "/usr/local/bin/bareos-runbefore.sh" RunsWhen = before RunsOnClient = yes } } Job { #the job which copies to tape Name = myserver-sys-copy-file-to-tape Type = Copy Level = Full Pool = Full Selection Type = SQLQuery Selection Pattern = "select jobid from job where name = 'myserver-sys-file' and level ='F' order by endtime DESC limit 1;" Next Pool = tape-automated Schedule = myserver-sys-copy-file-to-tape Spool Data = Yes Messages = Standard Priority = 20 Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr" Allow Duplicate Jobs = no Cancel Lower Level Duplicates = yes Cancel Queued Duplicates = yes RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-copy-file-to-tape-v yes\" | bconsole >/dev/null'" } Pool { #file/disk pool Name = Full Pool Type = Backup Recycle = yes # Bareos can automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 365 days # How long should the Full Backups be kept? (#06) Job Retention = 365 days File Retention = 365 days Maximum Volume Bytes = 50G # Limit Volume size to something reasonable Maximum Volumes = 200 # Limit number of Volumes in Pool Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>" Storage = bareos-file Next Pool = tape-automated } Pool { #tape pool Name = tape-automated Pool Type = Backup Recycle = yes # Bareos can automatically recycle Volumes AutoPrune = yes # Prune expired volumes #Recycle Oldest Volume = yes RecyclePool = Scratch Maximum Volume Bytes = 0 Job Retention = 365 days Volume Retention = 4 weeks Volume Use Duration = 12 days Cleaning Prefix = "CLN" Catalog Files = yes #needed, this is default Storage = tapeSD_HP_1_8_G2 } Storage { #this is the file/disk storage daemon Name = bareos-file Address = "bareos.mydomain.org" # N.B. Use a fully qualified name here (do not use "localhost" here). Password = "XXXX" Device = bareos-file Media Type = File Maximum Concurrent Jobs = 7 #dashboard progress / statistics collection Collect Statistics = yes TLS Enable = yes TLS Require = yes TLS DH File = /etc/bareos/ssl/dh4096.pem } Storage { #this is the tape storage daemon on another server Name = "tapeSD_HP_1_8_G2" Address = "tapeSD.mydomain.org" Lan Address = "tapeSD.directlink.mydomain.org" Password = "XXXX" Device = HP_1_8_G2 Autochanger = yes MediaType = "LTO" #dashboard progress / statistics collection Collect Statistics = yes TLS Enable = yes TLS Require = yes TLS DH File = /etc/bareos/ssl/dh4096.pem } -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/508d21ad-1d5e-449f-9753-9735dd97c03cn%40googlegroups.com.
