Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?
On Tue, Jan 23, 2024, at 11:16 AM, Martin Simmons wrote: >> On Wed, 17 Jan 2024 09:54:23 -0500, Dan Langille said: >> >> On Fri, Dec 29, 2023, at 6:26 PM, Martin Simmons wrote: >> >> On Fri, 29 Dec 2023 12:35:59 -0500, Dan Langille said: >> >> >> >> On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote: >> >> > 9.6.6 certainly displayed them for me, so I suspect a config issue. >> >> > >> >> > The messages would be omitted if !notsaved is in the Messages resource >> >> > (but >> >> > they would still be counted as "Non-fatal FD errors" which makes it add >> >> > "with >> >> > warnings" to the status). >> >> > >> >> > Maybe that changed in the client's bacula-fd.conf when you upgraded it? >> >> >> >> That's a good idea. >> >> >> >> [17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf >> >> -rw-r- 1 root bacula 1497 Feb 25 2023 >> >> /usr/local/etc/bacula/bacula-fd.conf >> >> >> >> [17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf >> >> MD5 (/usr/local/etc/bacula/bacula-fd.conf) = >> >> e41a7d835766f563253c0a93418a1c61 >> >> >> >> >> >> No change since February. >> >> >> >> Let's look at snapshots taken before Dec 25, the date of the job in >> >> question. >> >> >> >> [17:32 r730-01 dvl /.zfs/snapshot] % cd autosnap_2023-12-20_00:00:09_daily >> >> [17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % >> >> sudo md5 usr/local/etc/bacula/bacula-fd.conf >> >> MD5 (usr/local/etc/bacula/bacula-fd.conf) = >> >> e41a7d835766f563253c0a93418a1c61 >> >> >> >> >> >> I'm confident this file has not changed. >> > >> > Hmm, looking at src/lib/message.h, I suspect this change broke version >> > compatibility in the message filtering infrastructure: >> > >> > commit fd926fc4671b054234fd3d5957bc05d303d87763 >> > Author: Eric Bollengier >> > Date: Fri Nov 6 21:27:05 2020 +0100 >> > >> > Fix unexpected connection event sent by the FD when the Message >> > resource is not configured >> > >> > The problem is that the message types have been renumbered by moving >> > M_EVENTS >> > higher up, but messages sent to the Director from other daemons use the >> > numeric value of the type so this is an incompatible change in the wire >> > protocol. >> > >> > Dispite the date of this change, it looks like it first appeared in Bacula >> > 13, >> > so will cause problems if a Client < 13 sends a message to a Director >= >> > 13 as >> > in your case. >> >> That is concerning. It means backups may be incomplete and you don't know it. >> >> This has happened at home, and today I noticed it at $WORK. >> >> It is no longer the case that versions can follow the rule: >> >> bacula-dir=bacula-sd>bacula-fd >> >> That rule has been in effect as long as I've been associated with the >> project (about 20 years). Is there any possibility of this being fixed? Or >> is this change irrevocable? > > The change to the numbering is probably irrevocable (changing it back would > create a new set of incompatibilities), but some hack might be possible using > the jcr->FDVersion. > > >> If irrevocable, the users really need to be notified via an announcement. >> Clients must be upgraded or the risk of data loss is present. In my case, >> things I expected to be backed up were not being backed up. I could not tell >> because the warnings were not presented to me. > > I suggest you create a bug report about it at > https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues > so it can tracked. And there we go: https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues/2704 -- Dan Langille d...@langille.org ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Does Bacula allow me to configure different backup retention policies based on job age ?
Thanks, Radoslaw, for confirming this. Best, On 2024-01-23 2:47 a.m., Radosław Korzeniewski wrote: > Hello, > > pon., 22 sty 2024 o 05:56 MylesDearBusiness via Bacula-users > napisał(a): > >> Hello, >> >> How would I configure different backup retension policies (ie. backup >> densities) ? >> >> For example, for backups over a year old, I only want to retain backups >> every three months. >> >> For backups over six months old, I only want to retain backups every two >> months. >> >> For bacups within the last six months, I want to retain monthly backups >> >> Then, I want to do something similar with weekly backups. >> >> Doable? > > Yes, depends on your exact details required. > I'm making this with GFS backup policy, using different backup levels (full, > diff, incr), three pools with different backup retentions and a schedule > showing how often (density) execute a particular backup level makes three > different backup densities. > So, in your example: > - full, retention counted in years (you did not specified it), executed every > three months > - diff, retention 1 year, executed every two months > - incr, retention 6 months, executed every month > If you craft your schedule carefully then my execution times could be even > lower to achieve your requirements. I showed you an exemple. > > best regards -- > > Radosław Korzeniewski > rados...@korzeniewski.net___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?
> On Wed, 17 Jan 2024 09:54:23 -0500, Dan Langille said: > > On Fri, Dec 29, 2023, at 6:26 PM, Martin Simmons wrote: > >> On Fri, 29 Dec 2023 12:35:59 -0500, Dan Langille said: > >> > >> On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote: > >> > 9.6.6 certainly displayed them for me, so I suspect a config issue. > >> > > >> > The messages would be omitted if !notsaved is in the Messages resource > >> > (but > >> > they would still be counted as "Non-fatal FD errors" which makes it add > >> > "with > >> > warnings" to the status). > >> > > >> > Maybe that changed in the client's bacula-fd.conf when you upgraded it? > >> > >> That's a good idea. > >> > >> [17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf > >> -rw-r- 1 root bacula 1497 Feb 25 2023 > >> /usr/local/etc/bacula/bacula-fd.conf > >> > >> [17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf > >> MD5 (/usr/local/etc/bacula/bacula-fd.conf) = > >> e41a7d835766f563253c0a93418a1c61 > >> > >> > >> No change since February. > >> > >> Let's look at snapshots taken before Dec 25, the date of the job in > >> question. > >> > >> [17:32 r730-01 dvl /.zfs/snapshot] % cd autosnap_2023-12-20_00:00:09_daily > >> [17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % > >> sudo md5 usr/local/etc/bacula/bacula-fd.conf > >> MD5 (usr/local/etc/bacula/bacula-fd.conf) = > >> e41a7d835766f563253c0a93418a1c61 > >> > >> > >> I'm confident this file has not changed. > > > > Hmm, looking at src/lib/message.h, I suspect this change broke version > > compatibility in the message filtering infrastructure: > > > > commit fd926fc4671b054234fd3d5957bc05d303d87763 > > Author: Eric Bollengier > > Date: Fri Nov 6 21:27:05 2020 +0100 > > > > Fix unexpected connection event sent by the FD when the Message > > resource is not configured > > > > The problem is that the message types have been renumbered by moving > > M_EVENTS > > higher up, but messages sent to the Director from other daemons use the > > numeric value of the type so this is an incompatible change in the wire > > protocol. > > > > Dispite the date of this change, it looks like it first appeared in Bacula > > 13, > > so will cause problems if a Client < 13 sends a message to a Director >= 13 > > as > > in your case. > > That is concerning. It means backups may be incomplete and you don't know it. > > This has happened at home, and today I noticed it at $WORK. > > It is no longer the case that versions can follow the rule: > > bacula-dir=bacula-sd>bacula-fd > > That rule has been in effect as long as I've been associated with the project > (about 20 years). Is there any possibility of this being fixed? Or is this > change irrevocable? The change to the numbering is probably irrevocable (changing it back would create a new set of incompatibilities), but some hack might be possible using the jcr->FDVersion. > If irrevocable, the users really need to be notified via an announcement. > Clients must be upgraded or the risk of data loss is present. In my case, > things I expected to be backed up were not being backed up. I could not tell > because the warnings were not presented to me. I suggest you create a bug report about it at https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues so it can tracked. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to check a written job by re-read them
Am 23.01.24 um 15:18 schrieb Bill Arlofski via Bacula-users: Yes, a Copy job will need to read the backup data and by doing so, Bacula will verify the signature (checksum) of each file read. You would be notified in the job of a failure to read back a file with the correct checksum. Ok, a signature check additionally is good. By the way I though the data from the job simply will be copy without further checking and usingg smartctl before eject to show possible problems registred. But, as the name implies, you will be copying the data to another storage location, and hence using some additional space - E ven if it is only a temporary scratch space for your copies to be written to. Thats the reason I want in best to write them to /dev/null and hopefully can use some configuration to do them. Alternately, you can run a Verify (level=data) job which read the data from the backup media, also verifying the checksum of every file read - without actually writing the data to a second storage location. Yea, if there is no difference against copy to /dev/null then this is the goal for me. I have written a script which (just for testing purposes), when calls from a Backup Job's RunScript (RunsWhen = After), automatically restores the entire job but also runs all three Verify levels against the job. You can pick the parts of the script you need (maybe just the Verify level=data), and remove/comment out the rest, or just pick and choose what you need. I am attaching the script `AutoRestoreAndVerify.sh` which I use in my environment. Please edit the variables at the top and read through the instructions at the top of the script to understand how to implement this script. Thank you, I will check it carefully what I can use. Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to check a written job by re-read them
On 1/23/24 02:59, Pierre Bernhardt wrote: Hello, after a raid disaster which needs a full restore based on the last full backup on tape which has unreadable blocks and blocks the whole tape drive I want to check the written jobs. A good idea is to create a copy job so I have a copy of the written data which also checks the tape by reading them. By the way I want to test the tape job only so I mean it should be possible to write the copy data simply to /dev/null. Is it possible to use the fifo device? Is there another possibility to read the tape regularly? Pierre Hello Pierre, Yes, a Copy job will need to read the backup data and by doing so, Bacula will verify the signature (checksum) of each file read. You would be notified in the job of a failure to read back a file with the correct checksum. But, as the name implies, you will be copying the data to another storage location, and hence using some additional space - E ven if it is only a temporary scratch space for your copies to be written to. Alternately, you can run a Verify (level=data) job which read the data from the backup media, also verifying the checksum of every file read - without actually writing the data to a second storage location. I have written a script which (just for testing purposes), when calls from a Backup Job's RunScript (RunsWhen = After), automatically restores the entire job but also runs all three Verify levels against the job. You can pick the parts of the script you need (maybe just the Verify level=data), and remove/comment out the rest, or just pick and choose what you need. I am attaching the script `AutoRestoreAndVerify.sh` which I use in my environment. Please edit the variables at the top and read through the instructions at the top of the script to understand how to implement this script. I hope this helps. Best regards, Bill -- Bill Arlofski w...@protonmail.com AutoRestoreAndVerify.sh Description: application/shellscript signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula-Web 9.4.1 release
Hello everyone, I'm pleased to inform you that Bacula-Web 9.4.1 is now available. This is a bug fix and maintenance release. You can find more details in the release notes on the GitHub project ( https://github.com/bacula-web/bacula-web/releases/tag/v9.4.1). Docker images are also available on Docker Hub ( https://hub.docker.com/r/baculaweb/bacula-web) Side note: I’m currently working hard on the next major version 10.x, which will bring much more features and great improvements, I hope you’ll enjoy it once it will be ready. Best regards Davide Website - https://www.bacula-web.org Bacula-Web project on GitHub - https://github.com/bacula-web/bacula-web ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to save /dev tree and base directories only?
> On Mon, 22 Jan 2024 23:08:05 +0100, Pierre Bernhardt said: > > Am 22.01.24 um 16:41 schrieb Martin Simmons: > >> On Mon, 22 Jan 2024 10:44:49 +0100, Pierre Bernhardt said: > > Are you using udev (see the output of "df /dev")? If so, then I would > > expect > > that to recreate the contents of /dev so no backup is wanted. > Debian Buster and Bullseye should have it. By the way the system was not come > up before I recreated the inodes manually. Maybe some of them are essential > for booting the system so not all needs to be recreated but I think it is not > an issue to restore them from a backup. > > Here a list of files found on a fresh with debootstrap installed bullseye > before I firstly boot them up. > > root@backup:/media/file# ls dev > console fd full null ptmx pts random shm stderr stdin stdout tty > urandom zero > root@backup:/media/file# ls -lR dev > dev: > insgesamt 8 > crw-rw-rw- 1 root root 5, 1 Jan 22 22:25 console > lrwxrwxrwx 1 root root 13 Jan 22 22:25 fd -> /proc/self/fd > crw-rw-rw- 1 root root 1, 7 Jan 22 22:25 full > crw-rw-rw- 1 root root 1, 3 Jan 22 22:25 null > crw-rw-rw- 1 root root 5, 2 Jan 22 22:25 ptmx > drwxr-xr-x 2 root root 4096 Jan 22 22:25 pts > crw-rw-rw- 1 root root 1, 8 Jan 22 22:25 random > drwxr-xr-x 2 root root 4096 Jan 22 22:25 shm > lrwxrwxrwx 1 root root 15 Jan 22 22:25 stderr -> /proc/self/fd/2 > lrwxrwxrwx 1 root root 15 Jan 22 22:25 stdin -> /proc/self/fd/0 > lrwxrwxrwx 1 root root 15 Jan 22 22:25 stdout -> /proc/self/fd/1 > crw-rw-rw- 1 root root 5, 0 Jan 22 22:25 tty > crw-rw-rw- 1 root root 1, 9 Jan 22 22:25 urandom > crw-rw-rw- 1 root root 1, 5 Jan 22 22:25 zero > > dev/pts: > insgesamt 0 > > dev/shm: > insgesamt 0 > > I think this is the minimum of dev inodes needed for starting the system. Ah, it looks like there is a real /dev directory containing those minimal files, which is hidden by udev after booting. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to show diff of two backup?
Am 23.01.24 um 10:46 schrieb Pedro Oliveira: Hi Pierre Hello Pedro, I anwered also to the list. It looks like your answer does'nt it. In Bacula, Verify Jobs perform three important tasks to ensure data integrity: Verify Volume Data Integrity Verify Volume Metadata Integrity Verify File System Integrity Verify Volume Data Integrity Verifying the data written to a Volume is done using a level=data for a Verify Job of Bacula. Such a Job will read all data records from a Volume, and for each object encountered, will calculate the checksum and record the size. That information is then compared against the metadata as stored on the Volume. Doing so, the Storage Daemon is able to detect data corruption on the storage media. In this case maybe I can use it instead of my other suggestion to simply use a copy job to /dev/null. Maybe it is a little bit overdrawn to simply make a simple read test the the tape is still readable. But better this than nothing ;-) Verify Volume Metadata Integrity Bacula allows to compare the metadata read from the media against what it stored in the catalog. With Bacula, this comparison is done on a Job-by-Job basis; other backup systems often verify complete volumes. This could be used if it is possible to compare the metadata read from the media against a different job id. But I think this is not possible for the moment. In terms of Job to Job Verification, Bacula does not have that possibility without building some SQL queries. Is there already a way to recreate alike virtual full backup based on given jobids of allready created backups which only created in the database based on already stored data and metadata? Nevertheless, its a nice Feature that can be very useful, let me discuss internally with Bacula Support Team, about this possibility, I will update you soon with more details. Thanks for that. Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] How to check a written job by re-read them
Hello, after a raid disaster which needs a full restore based on the last full backup on tape which has unreadable blocks and blocks the whole tape drive I want to check the written jobs. A good idea is to create a copy job so I have a copy of the written data which also checks the tape by reading them. By the way I want to test the tape job only so I mean it should be possible to write the copy data simply to /dev/null. Is it possible to use the fifo device? Is there another possibility to read the tape regularly? Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] How to show diff of two backup?
Hello, because of an recovery based on the last backup I found that the full backup tape is corrupt. The last backup is based on this full and one diff job (bu1 = f1 + d11) The day before I had created a backup based on a full job on a different backup, a diff and several inc jobs (bu0 = f0 + d03 + i031 + i032 + i033) It was possible to restore the data by using the one day older backup bu0 of the day before the last full backup to the corrupt tape has to been created and additionally the newest difference by restoring from diff job d11. So almost should be recovered. But there is a small gap of one day between bu0 and bu1 which could not recovered because these files are only stored on f1. Before I will waste money to send the corrupt tap to a data recovery company I want to find these files which are only stored on the f1 tape. Is there a way afterwards to use bacula to give me a list of differences based on job ids? I'm able to use psql to do this but maybe it is easier by using it with the internal features like the verify jobs can do it. Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
