Too many comments on this thread to read them all (too many other things that
need attention).
However, I just wanted to note that I had been having trouble with include and exclude to break up
DLEs some time ago. When I posted asking for help, JLM replied back and told me that I really needed
to use Application amgtar for my backups. If I were just using the program gnutar, then to implement
my general expressions, the amanda user needed read access all the way down to the level in the
subdirectories where the expressions were being applied. Otherwise, it would not be able to do it,
and would come up with an empty list. I switched all my backups to Application amgtar, and have had
no problems since.
I should note that some of my LVMs are broken up into dozens of DLEs, even though I'm using LTO7.
Back when I was using AIT5, I tried to keep DLEs less than 100GB. Then with LTO6, I loosened it up
to 300GB, with some larger. Now, with LTO7, I let them approach 1T sometimes, but use hardware
compression so that I'm not eating all kinds of CPU with gzip processes. This has worked really well.
Researcher's data is often restricted access, and may be divided up into subdirectories by lab
personnel. Many of those take up many TeraBytes. So, I end up having to go down into their
subdirectories sorting out how to divide them into DLEs. I have about 100TB of storage on each of
two different Departments' servers. We're currently building arrays with 10TB HGST Helium filled
drives, Supermicro servers with Ubuntu 14.04 and 16.04, mdadm and LVM. I've found that I can build a
RAID5 with 2 drives and then grow it by adding drives as needed. When I reach 5 drives, I request
that the next addition be 2 drives so that I can convert it to RAID6 and grow it by 1 drive. All of
that can be done live, although it can get scary sometimes.
My approach to DLEs is as follows. The name after the "/./" in the first line of each DLE is an
arbitrary name for the DLE.
localhost /data/professorA/./catchall /data/professorA {
gnutar-lto7-local
exclude append "./directory1"
exclude append "./directory2"
# etc.
} -1
localhost /data/professorA/./directory1-catchall /data/professorA {
gnutar-lto7-local
include "./directory1"
exclude append "./directory1/EMR/CDH[1-9]*"
# etc.
} -1
localhost /data/professorA/./directory1a /data/professorA {
gnutar-lto7-local
include "./directory1/EMR/CDH[1-4]*"
} -1
localhost /data/professorA/./directory1b /data/professorA {
gnutar-lto7-local
include "./directory1/EMR/CDH[5-9]*"
} -1
localhost /data/professorA/./directory2-catchall /data/professorA {
# and so on, in a hierarchical top down structure.
The above example was constructed for this email message (so as to anonymize) based on real examples
in my disklist. As I said earlier, some of my actual examples end up breaking up an LVM (which would
correspond to, e.g., /data/professorA) into dozens of DLEs.
On 11/8/18 5:30 PM, Nathan Stratton Treadway wrote:
On Thu, Nov 08, 2018 at 15:21:00 -0500, Chris Nighswonger wrote:
On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) <
brian.cutt...@health.ny.gov> wrote:
Your syntax
fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" {
comp-tar
include "./[a-f]*"
estimate server
}
[...]
Well, this fixes my problem, though why I do not know.
fileserver CAMPUS_a-f /netdrives/CAMPUS {
comp-tar
exclude file "./[g-z]*"
estimate server
} 1
It seems a bit of work compared to the include directive. I tried "include
file" to no avail.
I haven't quite followed this whole thread, either, but have you taken a
look at
http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists
? It may help explain some of the nuances in how things work.
As Stefan mentioned, it would be helpeful to know what amanda version
you are using, and which dump program. If your client supports it,
using APPLICATION amgtar (rather than GNUTAR) is definitely a good idea.
But, especially if you are using GNUTAR, an important thing to keep in
mind is that the exclude list is processed by passing the whole list
using an --exclude option to GNU tar, which then processes that list as
it does the dump -- while in contrast the "include" options are
processed ahead of time by Amanda to build a list of directories to pass
on the command line (thus making up tar's list of "what should I be
backing up"?). The tricky thing is that the tar process runs with root
priviledge, so it can see any directory out there... but the "include"
work is done by an unprivileged process and so the directory *above* the
include pattern must be readable by the amanda user...
So, what are the permissions on the /netdrives/CAMPUS/ directory itself?
Nathan
----------------------------------------------------------------------------
Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogen...@bio.umass.edu>
---------------
Erdös 4
