Re: [Bacula-users] Fileset override in Schedule resource
Hi,
Well, your suggestion of writing automation scripts for configuring hundreds of
Jobs is well taken, we are already doing it.
Also we are using server side dynamic file set creation by executing a script.
Just for information : the idea of using files sets defined for each past
year, against one single job called “archive” is working well.
The restoration is also working well.
So what I am doing is :
1. Backup :
run job=archive FileSet = archive_2018 level=Full
run job=archive FileSet = archive_2019 level=Full
run job=archive FileSet = archive_2020 level=Full
1. Restoration :
restore client=-fd FileSet = archive_2018 select current all yes
done
restore client=-fd FileSet = archive_2019 select current all yes
done
restore client=-fd FileSet = archive_2020 select current all yes
done
I restored the archive backups and compared with the corresponding content on
the clients, both matched 100%
Regards,
-Yateen
From: Radosław Korzeniewski
Sent: Sunday, November 21, 2021 8:28 PM
To: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Fileset override in Schedule resource
Hello,
czw., 18 lis 2021 o 13:40 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)
mailto:yateen.shaligram_bha...@nokia.com>>
napisał(a):
Yateen : We have hundreds of lab Linux servers (let’s say 200), each getting
data generated every day, organized in sub dirs like //
The data generated in the past years is no longer required usually, but may be
needed just in case. We call this data as archive data and want to have job
that will run for each past year data and create an yearly archive (aone time
activity). So instead of creating 200 servers X 5 years = 1000 jobs, I want to
minimize the number of jobs. I intend to do so by creating only 200 archive
jobs, with variable fileset mechanism. I think this is what you have suggested
too.
The "variable" fileset handling is a pain in the ***. Especially during restore.
This will create a job inventory that will run into thousand.
What is a "job inventory"? I never heard of it in Bacula.
Bacula has no problem handling hundreds of thousands of jobs.
Yateen : As explained above, Job inventory means number of Jobs configured. I
know that, Bacula has no limit on handling hundreds of them, but as an admin I
intend to keep the list as minimum as possible by having some creativity.
Why do you need to keep it as minimal as possible? The first step I'll do for
this number of clients is to generate required configuration with automation
scripts, especially that you have an easy file directory pattern to backup. No
manual config file crafting.
I wonder why there is no fileset override for a job in the schedule resource,
when there are so many other overrides provided.
This requirement may be an enhancement candidate ??
Let's assume you have a two totally distinct fileset resources, i.e.
Fileset {
Name = FS1
Include {
File = /home
}
}
Fileset {
Name = FS2
Include {
Plugin = "qemu: vm=vm1"
}
}
and every other day in the Schedule for Incremental level backup you switch
between them back and forth.
What is your expected result in this case, when you select the most recent
backup to restore?
The above question is to understand the requirements.
Yateen : I am planning to do something similar.
So what do you want to achieve during restore?
I think your solution is invalid to your requirements.
To generate a "dynamic" fileset you can use:
* Any name preceded by an at-sign (@) is assumed to be the name of a file,
which contains a list of files each preceded by a “File =”. The named file is
read once when the configuration file is parsed during the Director startup.
* Any name beginning with a vertical bar (|) is assumed to be the name of a
program. This program will be executed on the Director's machine at the time
the Job starts.
* If the vertical bar (|) in front of File= is preceded by a backslash as
in \|, the program will be executed on the Client's machine instead of on the
Director's machine.
* Any file-list item preceded by a less-than sign (<) will be taken to be a
file. This file will be read on the Director's machine (see below for doing it
on the Client machine) at the time the Job starts, and the data will be assumed
to be a list of directories or files, one per line, to be included.
* If you precede the less-than sign (<) with a backslash as in \<, the
file-list will be read on the Client machine instead of on the Director's
machine.
You can find examples at Bacula's manual -
https://www.bacula.org/11.0.x-manuals/en/main/Configuring_Director.html#SECTION002170
This should allow you to achieve what you want exactly with some external
scripting which will dynamically prepare what direc
Re: [Bacula-users] Fileset override in Schedule resource
Hello,
czw., 18 lis 2021 o 13:40 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
yateen.shaligram_bha...@nokia.com> napisał(a):
> Yateen : We have hundreds of lab Linux servers (let’s say 200), each
> getting data generated every day, organized in sub dirs like
> //
>
> The data generated in the past years is no longer required usually, but
> may be needed just in case. We call this data as archive data and want to
> have job that will run for each past year data and create an yearly archive
> (aone time activity). So instead of creating 200 servers X 5 years = 1000
> jobs, I want to minimize the number of jobs. I intend to do so by creating
> only 200 archive jobs, with variable fileset mechanism. I think this is
> what you have suggested too.
>
The "variable" fileset handling is a pain in the ***. Especially during
restore.
>
>
>
>
> This will create a job inventory that will run into thousand.
>
>
>
> What is a "job inventory"? I never heard of it in Bacula.
>
> Bacula has no problem handling hundreds of thousands of jobs.
>
> Yateen : As explained above, Job inventory means number of Jobs
> configured. I know that, Bacula has no limit on handling hundreds of them,
> but as an admin I intend to keep the list as minimum as possible by having
> some creativity.
>
Why do you need to keep it as minimal as possible? The first step I'll do
for this number of clients is to generate required configuration with
automation scripts, especially that you have an easy file directory pattern
to backup. No manual config file crafting.
> I wonder why there is no fileset override for a job in the schedule
> resource, when there are so many other overrides provided.
>
> This requirement may be an enhancement candidate ??
>
>
>
> Let's assume you have a two totally distinct fileset resources, i.e.
>
>
>
> Fileset {
>
> Name = FS1
>
> Include {
>
> File = /home
>
> }
>
> }
>
>
>
> Fileset {
>
> Name = FS2
>
> Include {
>
> Plugin = "qemu: vm=vm1"
>
> }
>
> }
>
>
>
> and every other day in the Schedule for Incremental level backup you
> switch between them back and forth.
>
> What is your expected result in this case, when you select the most recent
> backup to restore?
>
> The above question is to understand the requirements.
>
>
>
> Yateen : I am planning to do something similar.
>
So what do you want to achieve during restore?
I think your solution is invalid to your requirements.
To generate a "dynamic" fileset you can use:
- Any name preceded by an at-sign (@) is assumed to be the name of a
file, which contains a list of files each preceded by a “File =”. The named
file is read once when the configuration file is parsed during the Director
startup.
- Any name beginning with a vertical bar (|) is assumed to be the name
of a program. This program will be executed on the Director's machine at
the time the Job starts.
- If the vertical bar (|) in front of File= is preceded by a backslash
as in \|, the program will be executed on the Client's machine instead of
on the Director's machine.
- Any file-list item preceded by a less-than sign (<) will be taken to
be a file. This file will be read on the Director's machine (see below for
doing it on the Client machine) at the time the Job starts, and the data
will be assumed to be a list of directories or files, one per line, to be
included.
- If you precede the less-than sign (<) with a backslash as in \<, the
file-list will be read on the Client machine instead of on the Director's
machine.
You can find examples at Bacula's manual -
https://www.bacula.org/11.0.x-manuals/en/main/Configuring_Director.html#SECTION002170
This should allow you to achieve what you want exactly with some external
scripting which will dynamically prepare what directories/files to backup
at the selected job.
But, when you change a fileset to switch from one year to another then
Bacula will mark all "missing" files as nonexistent/deleted for the next
job. If this is exactly what you want then great! Or I totally
misunderstood your requirements, then sorry.
best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fileset override in Schedule resource
Hi Radoslaw,
Thanks, please see my comments in-line
Regards,
Yateen
From: Radosław Korzeniewski
Sent: Monday, November 15, 2021 4:36 PM
To: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Fileset override in Schedule resource
Hello,
pt., 12 lis 2021 o 03:39 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)
mailto:yateen.shaligram_bha...@nokia.com>>
napisał(a):
Hi Radoslaw,
Thanks.
Well, that is what I thought.
But my requirement is different.
There are hundreds of clients and I want to create yearly archives (at present
for past years and later for future years too) for each of them.
What do you mean "yearly archives"? Is it a backup job? A cloned/migrated job
to the other storage? Something else?
Yateen : We have hundreds of lab Linux servers (let’s say 200), each getting
data generated every day, organized in sub dirs like //
The data generated in the past years is no longer required usually, but may be
needed just in case. We call this data as archive data and want to have job
that will run for each past year data and create an yearly archive (aone time
activity). So instead of creating 200 servers X 5 years = 1000 jobs, I want to
minimize the number of jobs. I intend to do so by creating only 200 archive
jobs, with variable fileset mechanism. I think this is what you have suggested
too.
This will create a job inventory that will run into thousand.
What is a "job inventory"? I never heard of it in Bacula.
Bacula has no problem handling hundreds of thousands of jobs.
Yateen : As explained above, Job inventory means number of Jobs configured. I
know that, Bacula has no limit on handling hundreds of them, but as an admin I
intend to keep the list as minimum as possible by having some creativity.
The simplest way could be to create one single archive job for each client and
run it with a fileset defined for each year.
How is your filesets for this feature defined?
Why do you need to change filesets? It is so strange.
Yateen : Explained above.
I wonder why there is no fileset override for a job in the schedule resource,
when there are so many other overrides provided.
This requirement may be an enhancement candidate ??
Let's assume you have a two totally distinct fileset resources, i.e.
Fileset {
Name = FS1
Include {
File = /home
}
}
Fileset {
Name = FS2
Include {
Plugin = "qemu: vm=vm1"
}
}
and every other day in the Schedule for Incremental level backup you switch
between them back and forth.
What is your expected result in this case, when you select the most recent
backup to restore?
The above question is to understand the requirements.
Yateen : I am planning to do something similar.
best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net<mailto:rados...@korzeniewski.net>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fileset override in Schedule resource
Hello,
pt., 12 lis 2021 o 03:39 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
yateen.shaligram_bha...@nokia.com> napisał(a):
> Hi Radoslaw,
>
>
>
> Thanks.
>
> Well, that is what I thought.
>
>
>
> But my requirement is different.
>
> There are hundreds of clients and I want to create yearly archives (at
> present for past years and later for future years too) for each of them.
>
What do you mean "yearly archives"? Is it a backup job? A cloned/migrated
job to the other storage? Something else?
> This will create a job inventory that will run into thousand.
>
What is a "job inventory"? I never heard of it in Bacula.
Bacula has no problem handling hundreds of thousands of jobs.
> The simplest way could be to create one single archive job for each client
> and run it with a fileset defined for each year.
>
How is your filesets for this feature defined?
Why do you need to change filesets? It is so strange.
I wonder why there is no fileset override for a job in the schedule
> resource, when there are so many other overrides provided.
>
> This requirement may be an enhancement candidate ??
>
Let's assume you have a two totally distinct fileset resources, i.e.
Fileset {
Name = FS1
Include {
File = /home
}
}
Fileset {
Name = FS2
Include {
Plugin = "qemu: vm=vm1"
}
}
and every other day in the Schedule for Incremental level backup you switch
between them back and forth.
What is your expected result in this case, when you select the most recent
backup to restore?
The above question is to understand the requirements.
best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fileset override in Schedule resource
Hi Radoslaw, Thanks. Well, that is what I thought. But my requirement is different. There are hundreds of clients and I want to create yearly archives (at present for past years and later for future years too) for each of them. This will create a job inventory that will run into thousand. The simplest way could be to create one single archive job for each client and run it with a fileset defined for each year. I wonder why there is no fileset override for a job in the schedule resource, when there are so many other overrides provided. This requirement may be an enhancement candidate ?? -Yateen From: Radosław Korzeniewski Sent: Thursday, November 11, 2021 9:01 PM To: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Fileset override in Schedule resource Hello, czw., 11 lis 2021 o 14:02 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) mailto:yateen.shaligram_bha...@nokia.com>> napisał(a): Hi all, We are using Bacula 9.4.4. on Linux with PostGreSQL. I understand that we can run a job manually through bconsole with a different fileset to override the one defined for the job config. But I find that such an override CAN NOT be specified in the job schedule , if I want to run the job with a different fileset through schedule. Any comments? Define a new job with a new fileset. :) I hope it helps. best regards -- Radosław Korzeniewski rados...@korzeniewski.net<mailto:rados...@korzeniewski.net> ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fileset override in Schedule resource
Hello, czw., 11 lis 2021 o 14:02 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) < yateen.shaligram_bha...@nokia.com> napisał(a): > Hi all, > > > > We are using Bacula 9.4.4. on Linux with PostGreSQL. > > > > I understand that we can run a job manually through bconsole with a > different fileset to override the one defined for the job config. > > > > But I find that such an override CAN NOT be specified in the job schedule > , if I want to run the job with a different fileset through schedule. > > > > Any comments? > Define a new job with a new fileset. :) I hope it helps. best regards -- Radosław Korzeniewski rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Fileset override in Schedule resource
Hi all, We are using Bacula 9.4.4. on Linux with PostGreSQL. I understand that we can run a job manually through bconsole with a different fileset to override the one defined for the job config. But I find that such an override CAN NOT be specified in the job schedule , if I want to run the job with a different fileset through schedule. Any comments? Thanks Yateen S Bhagat ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
