Re: [Bacula-users] How to prevent large files from being backed up?
On Tue, 20 Aug 2013 16:03:51 -0400, Phil Stracchino said:
Now, the above is a bit of a brute-force solution. I have not
personally tried this refinement, but I see no reason it should not
ALSO be possible to create a static fileset with a dynamically
generated exclude list, something like this.
FileSet {
Name = Dynamic Exclude Set
Include {
Options {
signature = SHA1
File = |sh -c 'find /home -size +10G'
Exclude = yes
}
File = /
File = /home
File = /var
}
}
This example should result in automatically excluding any file 10GB or
larger located anywhere under /home.
Unfortunately you can't put File inside the Options clause, so that can't be
used to generate a dynamic exclude list.
You can however add it to an exclude clause like this:
Exclude {
File = |sh -c 'find /home -size +10G'
}
That will work as long as none of the wild or regex patterns in the options
clauses match the excluded files (unless they are also using Exclude=yes).
__Martin
--
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
On 08/19/2013 12:41 PM, Jonathan Bayer wrote: Hi, We have a few users who, for various reasons, constantly create delete huge files (hundreds of gigs). I'd like to exclude these from the backup process. How can I do that, since I don't know where they can appear or what their names are? You can't. I've a similarly unknown set of files to back up and the best I could come up with is run find on the client looking for files not modified in the last 24 hours. And even then bacula isn't saving the metadata (ownership, permissions, timestamps) on parent directories (apparently it's a feature). So if you know these files don't stick around for more than X hours, you can run a script on the client that gives you a list of files older than X hours you back up those. Obviously, that gives you a X-hour window when real files aren't backed up plus a change for false negatives false positives. Plus bacula's dynamic fileset feature. We also have a rule: anything with _unb_ in its name isn't backed up -- but I can educate my users... -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
On 08/20/2013 02:02 PM, Dimitri Maziuk wrote: ... when real files aren't backed up plus a change for false negatives *chance* duh -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 08/20/13 15:02, Dimitri Maziuk wrote:
On 08/19/2013 12:41 PM, Jonathan Bayer wrote:
Hi,
We have a few users who, for various reasons, constantly create
delete huge files (hundreds of gigs). I'd like to exclude these
from the backup process.
How can I do that, since I don't know where they can appear or
what their names are?
You can't.
Actually, ues, you can. Instead of using a static Fileset, you can
configure Bacula to source a script that generates it on the fly. You
could create a dynamic-fileset script that excludes all files over a
specified size.
Here's an example from
http://www.bacula.org/manuals/en/install/install/Configuring_Director.html
of a dynamically-generated Fileset:
Include {
Options {
signature = SHA1
}
File = |sh -c 'df -l | grep \^/dev/hd[ab]\ | grep -v \.*/tmp\ \
| awk \{print \\$6}\'
}
So you could simply create a dynamic fileset script here that
enumerates all directories and all files not exceeding a specified size.
Now, the above is a bit of a brute-force solution. I have not
personally tried this refinement, but I see no reason it should not
ALSO be possible to create a static fileset with a dynamically
generated exclude list, something like this.
FileSet {
Name = Dynamic Exclude Set
Include {
Options {
signature = SHA1
File = |sh -c 'find /home -size +10G'
Exclude = yes
}
File = /
File = /home
File = /var
}
}
This example should result in automatically excluding any file 10GB or
larger located anywhere under /home.
- --
Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater
It's not the years, it's the mileage.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlITy6cACgkQ0DfOju+hMknfvgCaAq9OAedfNULleI25KMyuf/WE
VEgAoKXy0MSGDA24NUWMKzvCLY3TlOdN
=7XM+
-END PGP SIGNATURE-
--
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
On 08/20/2013 03:03 PM, Phil Stracchino wrote: On 08/20/13 15:02, Dimitri Maziuk wrote: On 08/19/2013 12:41 PM, Jonathan Bayer wrote: Hi, We have a few users who, for various reasons, constantly create delete huge files (hundreds of gigs). I'd like to exclude these from the backup process. How can I do that, since I don't know where they can appear or what their names are? You can't. Actually, ues, you can. Did you read the rest of my e-mail? Instead of using a static Fileset, you can configure Bacula to source a script that generates it on the fly. That is exactly what I said. Also that it's error-prone and that bacula won't store attributes on the parent directories unless you explicitly include them. (But then you have to have exclude everything in them and include individual files, which makes the whole mess even uglier and more error-prone.) See also replies to my Missing directory metadata and weird directory timestamps in this month's archive. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/20/13 16:43, Dimitri Maziuk wrote: On 08/20/2013 03:03 PM, Phil Stracchino wrote: On 08/20/13 15:02, Dimitri Maziuk wrote: On 08/19/2013 12:41 PM, Jonathan Bayer wrote: Hi, We have a few users who, for various reasons, constantly create delete huge files (hundreds of gigs). I'd like to exclude these from the backup process. How can I do that, since I don't know where they can appear or what their names are? You can't. Actually, ues, you can. Did you read the rest of my e-mail? Yes, I did. I'm not quite clear here whether you're arguing that I'm contradicting you, or arguing that you contradicted your own statement that You can't. Instead of using a static Fileset, you can configure Bacula to source a script that generates it on the fly. That is exactly what I said. Also that it's error-prone and that bacula won't store attributes on the parent directories unless you explicitly include them. (But then you have to have exclude everything in them and include individual files, which makes the whole mess even uglier and more error-prone.) Dmitri, if you look at the approach I proposed, it would back up the entire tree and only exclude specific files, which entirely sidesteps the parent-directory-metadata problem you mentioned. Excluding only the files you don't want is a simpler solution altogether. Are you arguing that it won't work? Your point is unclear. - -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlIT5FwACgkQ0DfOju+hMknMfACg/C/E+lWgMGX43dpFaXixOG9+ jwcAoM04eFsosd32kUbk7mq9BJB5OF1O =Skuj -END PGP SIGNATURE- -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
On 08/20/2013 04:49 PM, Phil Stracchino wrote:
Dmitri, if you look at the approach I proposed, it would back up the
entire tree and only exclude specific files, which entirely sidesteps
the parent-directory-metadata problem you mentioned. Excluding only
the files you don't want is a simpler solution altogether. Are you
arguing that it won't work? Your point is unclear.
My points are
1. Excluding files based on size alone sounds icky. You need to consider
your false positives negatives carefully.
2. OP didn't say what his fileset normally looks like. Without that
- if he's backing up everything and excluding specific files, then
another exclude should probably work.
- If he's excluding everything and backing up specific files, then
dynamic fileset comes with the parent metadata problem.
3. Most importantly, the only way to do it right is if OP can get his
users to cooperate. Then (e.g.) 'exclude { wild = *__nobak__* }' is
all there's to it.
--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
signature.asc
Description: OpenPGP digital signature
--
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to prevent large files from being backed up?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/20/13 18:17, Dimitri Maziuk wrote: On 08/20/2013 04:49 PM, Phil Stracchino wrote: Dmitri, if you look at the approach I proposed, it would back up the entire tree and only exclude specific files, which entirely sidesteps the parent-directory-metadata problem you mentioned. Excluding only the files you don't want is a simpler solution altogether. Are you arguing that it won't work? Your point is unclear. My points are 1. Excluding files based on size alone sounds icky. You need to consider your false positives negatives carefully. That *was* the original poster's stipulated condition, though. - -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlIUOlUACgkQ0DfOju+hMkmaSwCeKjp22e/qGgH0vpl1zM3QHL3P sYoAn14AyTJBQyYGhUEKtZge8LIyLnsA =cNxq -END PGP SIGNATURE- -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] How to prevent large files from being backed up?
Hi, We have a few users who, for various reasons, constantly create delete huge files (hundreds of gigs). I'd like to exclude these from the backup process. How can I do that, since I don't know where they can appear or what their names are? Thanks in advance. JBB -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
