Re: [Bacula-users] noticing an occasional incremental backup act like full
Yes, I just set mine to 60 years. I'll let job retention take care of the pruning from now on. On 5/30/21 3:45 PM, Heitor Faria wrote: Hello Joseph, There is no surprise in that finding. If it is not in the catalog, it doesn't exist. IMHO Bacula default File and Job values should be much higher, e.g. 5 years or unlimited. If the user have special machines with millions of files, he should lower the File Retention accordingly. This is the exception use case scenario. Regards, -- MSc Heitor Faria (Miami/USA) CEO Bacula LatAm mobile1: + 1 909 655-8971 mobile2: + 55 61 98268-4220 América Latina [ http://bacula.lat/] Original Message From: Joseph Zatarski Sent: Sunday, May 30, 2021 04:27 PM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] noticing an occasional incremental backup act like full Hello again everyone, I've figured out what is happening to cause this issue. Long story short, bacula pruned the file records from my last full backup job out of the catalog. It didn't prune the job itself, just the file records. This causes any file that hadn't been backed up since my initial full backup job to be considered as a new file to bacula when it does the incremental. Since most of my files don't change very often, that explains the behavior entirely. There's plenty of warnings in the documentation about trying to restore from jobs that have had their file records pruned, but nothing about trying to do an incremental on top of a full that had it's files pruned. Obviously thinking about this a little bit would reveal that it's an issue. Best Regards, Joe Zatarski ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
Hello Joseph, There is no surprise in that finding. If it is not in the catalog, it doesn't exist. IMHO Bacula default File and Job values should be much higher, e.g. 5 years or unlimited. If the user have special machines with millions of files, he should lower the File Retention accordingly. This is the exception use case scenario. Regards, -- MSc Heitor Faria (Miami/USA) CEO Bacula LatAm mobile1: + 1 909 655-8971 mobile2: + 55 61 98268-4220 América Latina [ http://bacula.lat/] Original Message From: Joseph Zatarski Sent: Sunday, May 30, 2021 04:27 PM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] noticing an occasional incremental backup act like full >Hello again everyone, > >I've figured out what is happening to cause this issue. > >Long story short, bacula pruned the file records from my last full >backup job out of the catalog. It didn't prune the job itself, just the >file records. This causes any file that hadn't been backed up since my >initial full backup job to be considered as a new file to bacula when it >does the incremental. Since most of my files don't change very often, >that explains the behavior entirely. > >There's plenty of warnings in the documentation about trying to restore >from jobs that have had their file records pruned, but nothing about >trying to do an incremental on top of a full that had it's files pruned. >Obviously thinking about this a little bit would reveal that it's an issue. > >Best Regards, >Joe Zatarski > > >___ >Bacula-users mailing list >Bacula-users@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
Hello again everyone, I've figured out what is happening to cause this issue. Long story short, bacula pruned the file records from my last full backup job out of the catalog. It didn't prune the job itself, just the file records. This causes any file that hadn't been backed up since my initial full backup job to be considered as a new file to bacula when it does the incremental. Since most of my files don't change very often, that explains the behavior entirely. There's plenty of warnings in the documentation about trying to restore from jobs that have had their file records pruned, but nothing about trying to do an incremental on top of a full that had it's files pruned. Obviously thinking about this a little bit would reveal that it's an issue. Best Regards, Joe Zatarski ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:31:51 Bill Arlofski via Bacula-users wrote: > On Sunday, May 23, 2021 8:25 AM, Bill Arlofski wrote: > > Check this out: > > 8< > > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified > > on 2021-05-23 08:12:32, this is after the last successful backup on > > 2021-05-22 17:35:16. > > > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup > > found in catalog for the current FileSet. Doing FULL backup. > > 8< > > I just checked and I see that this was implemented in June 2019. :) I must have got you wrong. I thought you are referring to "No prior or suitable Full backup". -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:25:37 Bill Arlofski via Bacula-users wrote: > Right, good point! And when a job (typically set to Level = Incremental) > is first run, this will always happen too. :) > > > Oh! And I just remembered that I had made an internal Bacula Enterprise > feature request some time ago. It was accepted and implemented, so > Option 1 is even better than I explained because now it tells you very > clearly why there are "no prior suitable Full backups". > > Check this out: > 8< > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on > 2021-05-23 08:12:32, this is after the last successful backup on > 2021-05-22 17:35:16. > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found > in catalog for the current FileSet. Doing FULL backup. 8< > > > I am looking forward to Joe's feedback in this thread. :) This message is very helpful. I don't even remember when it first appeared. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday, May 23, 2021 8:25 AM, Bill Arlofski wrote: > > Check this out: > 8< > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on > 2021-05-23 08:12:32, this is after the last > successful backup on 2021-05-22 17:35:16. > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found in > catalog for the current FileSet. Doing FULL backup. > 8< I just checked and I see that this was implemented in June 2019. :) Best regards, Bill ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On 5/23/21 3:50 AM, Josip Deanovic wrote: > > There is another option. > > Option 4: > - > If no valid full backup for the job could be found in the catalog, the > backup job will be upgraded from incremental or differential to full > backup. The message would be the same as the one described in option 1. > > This could happen if the job is prematurely pruned (before a new full > backup has been performed). > Right, good point! And when a job (typically set to Level = Incremental) is first run, this will always happen too. :) Oh! And I just remembered that I had made an internal Bacula Enterprise feature request some time ago. It was accepted and implemented, so Option 1 is even better than I explained because now it tells you very clearly why there are "no prior suitable Full backups". Check this out: 8< 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on 2021-05-23 08:12:32, this is after the last successful backup on 2021-05-22 17:35:16. 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found in catalog for the current FileSet. Doing FULL backup. 8< I am looking forward to Joe's feedback in this thread. :) Best regards, Bill -- Bill Arlofski w...@protonmail.com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Saturday 2021-05-22 21:14:20 Bill Arlofski via Bacula-users wrote: > Hello Joe, > > I can think of three things that can cause this to happen, and at least > two of them will show some evidence in the job log > > > Option 1: > - > You have edited the Fileset, and the "IgnoreFilesetChanges = yes" option > was not set at the top of the Fileset. In this case Bacula will log as > one of the first lines in the job's log something like this: 8< > 05-May 11:05 bacula-dir JobId 38357: No prior or suitable Full backup > found in catalog. Doing FULL backup. 8< > > Additionally, in the Summary, where it shows the "Backup Level:" it will > also tell you the job was upgraded from an Inc or Diff 8< > Backup Level: Full (upgraded from Differential) > 8< > > One other thing in the summary to help diagnose if an edit of the > Fileset was the cause is that the "Fileset:" line shows the last time > the Fileset that was used in the job was edited: > 8< > FileSet:"HomeFileSet" 2015-07-03 12:35:32 > 8< > > > Option 2: > - > Something or someone has 'touched' all of the files in the tree. I have > seen this a few times where someone implements a script - perhaps to do > some virus scan or something - and the script or program modifies the > attributes of every single file, causing Bacula to (correctly) noticed > a change and backup each file. > > To mitigate this type of case, you can set specific attributes to check > in the Fileset with the "Accurate = " setting - where is a > list of characters to enable the attributes you want to check. Using > this feature you can tell Bacula to ignore attributes that may get > modified in cases like I described. > > > Option 3: > - > You have set the "MaxFullInterval" in a Job or JobDefs resource. > Personally, I like and use this feature. It is a simple way to help > "spread out" my Full backups from all kicking off at the same time each > month. And, since I don't really care what day or time my Fulls run as > long as they run once a month - this is a prefect feature for me. > > I set it for about 33 days, and then set all of my Jobs to have "Level = > Incremental", and my Schedules are just simple "run every day at x > time" with no mention of running a Full on the First Sunday of a month > and Incrementals every other day. > > Then, when 33 days have passed, instead of doing the specified > Incremental or Differential level configured in the Job/JobDefs, Bacula > will log this in the job log: > 8< > 12-May 23:00 bacula-dir JobId 38614: Max Full Interval exceeded. Doing > FULL backup. 8< > > and then this in the Summary: > 8< > Backup Level: Full (upgraded from Differential) > 8< > > Also, if I kick of a Full for whatever reason, this starts the > MaxFullInterval counter for this job again at this time, adding a > little more entropy into the mix of when Fulls are kicked off. > > Hope this helps, There is another option. Option 4: - If no valid full backup for the job could be found in the catalog, the backup job will be upgraded from incremental or differential to full backup. The message would be the same as the one described in option 1. This could happen if the job is prematurely pruned (before a new full backup has been performed). Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On 5/22/21 4:45 PM, Joseph Zatarski wrote: > I notice that every once in a while, I'll have an incremental run that > seems to back up the entire file set with no regard to what files have > actually changed. > > Firstly, has anybody else noticed this happening? JUST TO BE CERTAIN here: Is this a case where the job RAN as an incremental, but backed up everything anyway, or a case where the job was promoted to Full? I'm pretty sure you're talking about the former, but I want to be sure. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
Hello Joe, I can think of three things that can cause this to happen, and at least two of them will show some evidence in the job log Option 1: - You have edited the Fileset, and the "IgnoreFilesetChanges = yes" option was not set at the top of the Fileset. In this case Bacula will log as one of the first lines in the job's log something like this: 8< 05-May 11:05 bacula-dir JobId 38357: No prior or suitable Full backup found in catalog. Doing FULL backup. 8< Additionally, in the Summary, where it shows the "Backup Level:" it will also tell you the job was upgraded from an Inc or Diff 8< Backup Level: Full (upgraded from Differential) 8< One other thing in the summary to help diagnose if an edit of the Fileset was the cause is that the "Fileset:" line shows the last time the Fileset that was used in the job was edited: 8< FileSet:"HomeFileSet" 2015-07-03 12:35:32 8< Option 2: - Something or someone has 'touched' all of the files in the tree. I have seen this a few times where someone implements a script - perhaps to do some virus scan or something - and the script or program modifies the attributes of every single file, causing Bacula to (correctly) noticed a change and backup each file. To mitigate this type of case, you can set specific attributes to check in the Fileset with the "Accurate = " setting - where is a list of characters to enable the attributes you want to check. Using this feature you can tell Bacula to ignore attributes that may get modified in cases like I described. Option 3: - You have set the "MaxFullInterval" in a Job or JobDefs resource. Personally, I like and use this feature. It is a simple way to help "spread out" my Full backups from all kicking off at the same time each month. And, since I don't really care what day or time my Fulls run as long as they run once a month - this is a prefect feature for me. I set it for about 33 days, and then set all of my Jobs to have "Level = Incremental", and my Schedules are just simple "run every day at x time" with no mention of running a Full on the First Sunday of a month and Incrementals every other day. Then, when 33 days have passed, instead of doing the specified Incremental or Differential level configured in the Job/JobDefs, Bacula will log this in the job log: 8< 12-May 23:00 bacula-dir JobId 38614: Max Full Interval exceeded. Doing FULL backup. 8< and then this in the Summary: 8< Backup Level: Full (upgraded from Differential) 8< Also, if I kick of a Full for whatever reason, this starts the MaxFullInterval counter for this job again at this time, adding a little more entropy into the mix of when Fulls are kicked off. Hope this helps, Best regards, Bill -- Bill Arlofski w...@protonmail.com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
