Splitting a DLE with regex
Hello all,
Attempting to follow the examples in the amanda documentation and
results when I search for it in the mailing list Archives, I encounter
problems. I have one DLE (/home) that is approx 200GB and would like to
split this out using regex similar to the examples given in:
http://archives.zmanda.com/amanda-archives/viewtopic.php?t=6935
http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists#Splitting_it_up.2C_the_easy_part
When I create the disklist entry using the hostname, diskname (mnemonic,
so the [hostname diskname] pairs are unique) and devicename (using the
mountpoint):
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
an amcheck returns:
ERROR: localhost: [Nothing found to include for disk /home/a-c]
There are several subdirectories that would fall within the a-c range:
/home/aaronson
/home/aferton
/home/amigos
/home/archery
/home/blank
/home/borders
/home/brynwyn
/home/caronwyn
/home/cirque
/home/common
What am I missing? I'd like to split up this monolithic DLE into more
manageable portions (a-c, d-f, g-i, and so on...) but can't figure out
why amanda is having problems with the regex.
Thanks!
~~Mike
Re: Splitting a DLE with regex
On Tue, Jul 30, 2013 at 08:09:15AM -0400, Mike Neimoyer wrote:
>
> Hello all,
>
> Attempting to follow the examples in the amanda documentation and
> results when I search for it in the mailing list Archives, I
> encounter problems. I have one DLE (/home) that is approx 200GB and
> would like to split this out using regex similar to the examples
> given in:
>
> http://archives.zmanda.com/amanda-archives/viewtopic.php?t=6935
> http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists#Splitting_it_up.2C_the_easy_part
>
> When I create the disklist entry using the hostname, diskname
> (mnemonic, so the [hostname diskname] pairs are unique) and
> devicename (using the mountpoint):
>
> localhost /home/a-c /home {
> vhost2-user-tar
> include "./[a-c]*"
> } 1 local
>
> an amcheck returns:
>
> ERROR: localhost: [Nothing found to include for disk /home/a-c]
>
> There are several subdirectories that would fall within the a-c range:
> /home/aaronson
> /home/aferton
> /home/amigos
> /home/archery
> /home/blank
> /home/borders
> /home/brynwyn
> /home/caronwyn
> /home/cirque
> /home/common
>
> What am I missing? I'd like to split up this monolithic DLE into
> more manageable portions (a-c, d-f, g-i, and so on...) but can't
> figure out why amanda is having problems with the regex.
Nothing jumps out at me. Might /home be a symbolic link
to another location, eg. /export/home? I don't know if
it matters, but I've always used the actual storage dir.
BTW the second parameter (/home/a-c) is just a DLE name,
not a pathname. Omitting the slashes might avoid
future confusion. Ex. HOME:a-c
--
Jon H. LaBadie j...@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (609) 477-8330 (C)
Re: Splitting a DLE with regex
Try:
localhost /home/./a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
That is, put in an intervening "./" in the name on the first line.
I have a couple of examples like that in my disklist, and it all seems to work
just fine.
On 7/30/13 2:00 PM, Jon LaBadie wrote:
On Tue, Jul 30, 2013 at 08:09:15AM -0400, Mike Neimoyer wrote:
Hello all,
Attempting to follow the examples in the amanda documentation and
results when I search for it in the mailing list Archives, I
encounter problems. I have one DLE (/home) that is approx 200GB and
would like to split this out using regex similar to the examples
given in:
http://archives.zmanda.com/amanda-archives/viewtopic.php?t=6935
http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists#Splitting_it_up.2C_the_easy_part
When I create the disklist entry using the hostname, diskname
(mnemonic, so the [hostname diskname] pairs are unique) and
devicename (using the mountpoint):
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
an amcheck returns:
ERROR: localhost: [Nothing found to include for disk /home/a-c]
There are several subdirectories that would fall within the a-c range:
/home/aaronson
/home/aferton
/home/amigos
/home/archery
/home/blank
/home/borders
/home/brynwyn
/home/caronwyn
/home/cirque
/home/common
What am I missing? I'd like to split up this monolithic DLE into
more manageable portions (a-c, d-f, g-i, and so on...) but can't
figure out why amanda is having problems with the regex.
Nothing jumps out at me. Might /home be a symbolic link
to another location, eg. /export/home? I don't know if
it matters, but I've always used the actual storage dir.
BTW the second parameter (/home/a-c) is just a DLE name,
not a pathname. Omitting the slashes might avoid
future confusion. Ex. HOME:a-c
--
---
Chris Hoogendyk
-
O__ Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 347 Morrill Science Center
~~ - University of Massachusetts, Amherst
---
Erdös 4
Re: Splitting a DLE with regex
Replying to multiple folks with this message Thanks for your kind
responses! :)
Starting with this disklist entry which returned an error:
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
Jon was good enough to respond:
> Might /home be a symbolic link to another
> location, eg. /export/home? I don't know
> if it matters, but I've always used the
> actual storage dir.
>
> BTW the second parameter (/home/a-c) is
> just a DLE name, not a pathname. Omitting
> the slashes might avoid future confusion.
> Ex. HOME:a-c
I double-checked that /home was not symbolically linked to anywhere
else. Then I edited the diskname parameter in the DLE:
localhost HOME:a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
and an amcheck Daily returned, after the usualy info dir and index dir
not existing and being created on the next run:
ERROR: localhost: [Nothing found to include for disk "HOME:a-c"]
Client check: 1 host checked in 1.303 seconds. 1 problem found.
No love with that, I reverted the DLE to it's original (above) because
Chris' suggestion had rolled in:
> Try:
>
> localhost /home/./a-c /home {
> vhost2-user-tar
> include "./[a-c]*"
> } 1 local
>
> That is, put in an intervening "./" in the
> name on the first line.
>
> I have a couple of examples like that in my
> disklist, and it all seems to work just fine.
Sounds interesting! So I tried that.
localhost /home/./a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
Unfortunately, an amcheck Daily returned a by-now-familiar message:
ERROR: localhost: [Nothing found to include for disk /home/./a-c]
Client check: 1 host checked in 1.223 seconds. 1 problem found.
Reverting the DLE to original, I then took a crack at it again, based on
Christ's suggestion that the regex was invalid:
> Your regex is invalid. It says "the letters
> a, b, or c repeated any number of times" and
> would match a, aa, cab, but not bad or aaron.
> Try "./[a-c].*"
Ahh, okay, this makes sense (sorta, since I thought the regex would
catch ANY directories starting with a*, b* c*) I'll give this new
method a go, however! I fix up the original DLE with the new regex:
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c].*"
} 1 local
AND THE SURVEY SAAAYS:
ERROR: localhost: [Nothing found to include for disk /home/a-c]
Client check: 1 host checked in 1.241 seconds. 1 problem found.
I then tried mixing John's diskname label with Christ's regex: same
results. I tried Chris' suggestion for the diskname with Christ's
regex: no love.
By all that I've read and all that I've searched, this SHOULD be
working. I'm at wit's end, and I'd really like to get this working
before I'm laid-off on Friday. :)
Anyone have ANY other ideas? Can it be a problem with Amanda 3.3.1
(under Centos 6, with tar 1.26) passing include directives to tar?
Could it be a permission issue with tar? Beuller? Beuller? Anyone?
:)
~~Mike
Re: Splitting a DLE with regex
Mike,
include must be a glob expression, not a regex, so "./[a-c]*" is the
correct syntax,
Do /home and /home/aaronson are on the same filesystem?
df /home
df /home/aaronson
Are you using the application 'amgtar' or the program 'GNUTAR'?
Jean-Louis
On 07/31/2013 12:56 PM, Mike Neimoyer wrote:
Replying to multiple folks with this message Thanks for your kind
responses! :)
Starting with this disklist entry which returned an error:
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
Jon was good enough to respond:
> Might /home be a symbolic link to another
> location, eg. /export/home? I don't know
> if it matters, but I've always used the
> actual storage dir.
>
> BTW the second parameter (/home/a-c) is
> just a DLE name, not a pathname. Omitting
> the slashes might avoid future confusion.
> Ex. HOME:a-c
I double-checked that /home was not symbolically linked to anywhere
else. Then I edited the diskname parameter in the DLE:
localhost HOME:a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
and an amcheck Daily returned, after the usualy info dir and index dir
not existing and being created on the next run:
ERROR: localhost: [Nothing found to include for disk "HOME:a-c"]
Client check: 1 host checked in 1.303 seconds. 1 problem found.
No love with that, I reverted the DLE to it's original (above) because
Chris' suggestion had rolled in:
> Try:
>
> localhost /home/./a-c /home {
> vhost2-user-tar
> include "./[a-c]*"
> } 1 local
>
> That is, put in an intervening "./" in the
> name on the first line.
>
> I have a couple of examples like that in my
> disklist, and it all seems to work just fine.
Sounds interesting! So I tried that.
localhost /home/./a-c /home {
vhost2-user-tar
include "./[a-c]*"
} 1 local
Unfortunately, an amcheck Daily returned a by-now-familiar message:
ERROR: localhost: [Nothing found to include for disk /home/./a-c]
Client check: 1 host checked in 1.223 seconds. 1 problem found.
Reverting the DLE to original, I then took a crack at it again, based
on Christ's suggestion that the regex was invalid:
> Your regex is invalid. It says "the letters
> a, b, or c repeated any number of times" and
> would match a, aa, cab, but not bad or aaron.
> Try "./[a-c].*"
Ahh, okay, this makes sense (sorta, since I thought the regex would
catch ANY directories starting with a*, b* c*) I'll give this new
method a go, however! I fix up the original DLE with the new regex:
localhost /home/a-c /home {
vhost2-user-tar
include "./[a-c].*"
} 1 local
AND THE SURVEY SAAAYS:
ERROR: localhost: [Nothing found to include for disk /home/a-c]
Client check: 1 host checked in 1.241 seconds. 1 problem found.
I then tried mixing John's diskname label with Christ's regex: same
results. I tried Chris' suggestion for the diskname with Christ's
regex: no love.
By all that I've read and all that I've searched, this SHOULD be
working. I'm at wit's end, and I'd really like to get this working
before I'm laid-off on Friday. :)
Anyone have ANY other ideas? Can it be a problem with Amanda 3.3.1
(under Centos 6, with tar 1.26) passing include directives to tar?
Could it be a permission issue with tar? Beuller? Beuller? Anyone?
:)
~~Mike
Re: Splitting a DLE with regex
Op 31-07-13 18:56, Mike Neimoyer schreef: > > include "./[a-c].*" Maybe you should first try an existing pathname (so no wildcard), just to be sure the "./" is correct here. -- Met vriendelijke groeten, AT COMPUTING Gerrit A. Smit Beheer Technische Infrastructuur AT Computing Telefoon: +31 24 352 72 22 Dé one-stop-Linux-shop Telefoon cursussecretariaat: +31 24 352 72 72 Fax: +31 24 352 72 92 Kerkenbos 12-38t...@atcomputing.nl 6546 BE Nijmegen www.atcomputing.nl Nieuw bij AT Computing: onze Linux Reference Card nu ook als gratis app!
Re: Splitting a DLE with regex
I do that sort of thing a lot.
finsen /export/home-AZ /export/home {
user-tar2
include "./[A-Z]*"
}
trel /trelRZ /trel {
comp-server-user-tar
include "./[R-Z]*"
}
you have teh case correct? The dot between the first letter
and the wild card is intentional (between the right closing
bracket and the astrix)?
Remember, to check where you are anchored. I've lost your
earlier emails, but you do have to be careful to know what
your starting point is.
In the above example its /export/home, or the /trel directory,
respectively. And you are using TAR (gtar, star, as long as its
an amanda compatible version) not DUMP.
On Wed, Jul 31, 2013 at 07:19:31PM +0200, Gerrit A. Smit TI wrote:
> Op 31-07-13 18:56, Mike Neimoyer schreef:
> >
> > include "./[a-c].*"
> Maybe you should first try an existing pathname (so no wildcard), just
> to be sure the "./" is correct here.
>
>
>
>
> --
> Met vriendelijke groeten,
> AT COMPUTING
>
> Gerrit A. Smit
> Beheer Technische Infrastructuur
>
> AT Computing Telefoon: +31 24 352 72 22
> Dé one-stop-Linux-shop Telefoon cursussecretariaat: +31 24 352 72 72
>Fax: +31 24 352 72 92
> Kerkenbos 12-38t...@atcomputing.nl
> 6546 BE Nijmegen www.atcomputing.nl
>
> Nieuw bij AT Computing: onze Linux Reference Card nu ook als gratis app!
>
---
Brian R Cuttler brian.cutt...@wadsworth.org
Computer Systems Support(v) 518 486-1697
Wadsworth Center(f) 518 473-6384
NYS Department of HealthHelp Desk 518 473-0773
Re: Splitting a DLE with regex
Thanks to Gerrit, Brian and Jean-Louis. I'll be responding to them in
this message
Hello Gerrit, thanks for chiming in!!>> include "./[a-c].*"
> Maybe you should first try an existing pathname (so no wildcard), just
> to be sure the "./" is correct here.
Good point, thanks!
localhost /home/a1 /home {
vhost2-user-tar
include "./aaronson"
} 1 local
And an amcheck seemed to work:
Client check: 1 host checked in 1.220 seconds. 0 problems found.
So, it appears that the ./ is correct, which follows what Jean-louis
follows with:
> include must be a glob expression, not a regex, so "./[a-c]*" is the
> correct syntax,
>
> Do /home and /home/aaronson are on the same filesystem?
>df /home
>df /home/aaronson
Yes, they are on the same filesystem:
bash-4.1$ df /home /home/aaronson
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1935102536 420335640 514756896 45% /
/dev/vda1935102536 420335640 514756896 45% /
> Are you using the application 'amgtar' or the program 'GNUTAR'?
I *think* that amanda was configured --with-program GNUTAR, but it was
compiled before I took over. Is there a way to check, via a debug file
or log?
Brian added:
> Remember, to check where you are anchored. I've lost your
> earlier emails, but you do have to be careful to know what
> your starting point is.
Starting point is /home and then the include directive is for "./[a-c]*"
And all subdirectories are lower-case, except the numbered which will
get "./[0-9]*" as their include directive.
Thanks so much,
~~Mike
Re: Splitting a DLE with regex
For compile time options: $ /usr/sbin/amadmin '' version Look for the values of DUMP en RESTORE Gerrit
Re: Splitting a DLE with regex
Op 31-07-13 19:58, Gerrit A. Smit TI schreef: > For compile time options: > > $ /usr/sbin/amadmin '' version > > Look for the values of DUMP en RESTORE Sorry, look for GNUTAR. I think ... Gerrit
Re: Splitting a DLE with regex
On Wed, Jul 31, 2013 at 01:43:50PM -0400, Mike Neimoyer wrote:
> Thanks to Gerrit, Brian and Jean-Louis. I'll be responding to them in
> this message
>
>
> Hello Gerrit, thanks for chiming in!!>> include "./[a-c].*"
> > Maybe you should first try an existing pathname (so no wildcard), just
> > to be sure the "./" is correct here.
>
> Good point, thanks!
>
> localhost /home/a1/home {
> vhost2-user-tar
> include "./aaronson"
> } 1 local
>
> And an amcheck seemed to work:
> Client check: 1 host checked in 1.220 seconds. 0 problems found.
>
> So, it appears that the ./ is correct, which follows what Jean-louis
> follows with:
>
> > include must be a glob expression, not a regex, so "./[a-c]*" is the
> > correct syntax,
> >
> > Do /home and /home/aaronson are on the same filesystem?
> >df /home
> >df /home/aaronson
>
> Yes, they are on the same filesystem:
> bash-4.1$ df /home /home/aaronson
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/vda1935102536 420335640 514756896 45% /
> /dev/vda1935102536 420335640 514756896 45% /
>
>
> > Are you using the application 'amgtar' or the program 'GNUTAR'?
>
> I *think* that amanda was configured --with-program GNUTAR, but it was
> compiled before I took over. Is there a way to check, via a debug file
> or log?
>
>
> Brian added:
>
> > Remember, to check where you are anchored. I've lost your
> > earlier emails, but you do have to be careful to know what
> > your starting point is.
>
> Starting point is /home and then the include directive is for "./[a-c]*"
> And all subdirectories are lower-case, except the numbered which will
> get "./[0-9]*" as their include directive.
Oh - interesting to note.
I'm using globbing on a system with ZFS file systems, each
user directory has its own zfs file system carved out of a
large zpool. I have no problems with GNUTAR backups globbing
together the a* directories and then the b*, etc. Each file
system below my anchor point is a unique file system, but I
can glob them together in a single DLE. [My manager has suggested
separate DLE for use userid, but that would have been unworkable
for us], this scheme however does prevent me from being able to
backup the globs using snapshots.
> Thanks so much,
> ~~Mike
>
>
>
---
Brian R Cuttler brian.cutt...@wadsworth.org
Computer Systems Support(v) 518 486-1697
Wadsworth Center(f) 518 473-6384
NYS Department of HealthHelp Desk 518 473-0773
Re: Splitting a DLE with regex
On 07/31/2013 01:43 PM, Mike Neimoyer wrote: > Are you using the application 'amgtar' or the program 'GNUTAR'? I *think* that amanda was configured --with-program GNUTAR, but it was compiled before I took over. Is there a way to check, via a debug file or log? In the amandad and selfcheck debug files, do you have amgtar debug files? amadmin CONFIG disklist Do the amanda user can read /home? Jean-Louis
Re: Splitting a DLE with regex
Jean-Louis:
> In the amandad and selfcheck debug files, do you have amgtar debug files?
No amgtar debug files.
> amadmin CONFIG disklist
Performing an amadmin Daily disklist shows that each uses program
"GNUTAR" in it. Checking the amadmin '' version that Gerrit recommended
shows:
bash-4.1$ amadmin '' version
build: VERSION="Amanda-3.3.1"
BUILT_DATE="Fri Nov 16 09:09:53 EST 2012" BUILT_MACH=""
BUILT_REV="4553" BUILT_BRANCH="community_3_3_1" CC="gcc"
paths: bindir="/usr/local/bin" sbindir="/usr/local/sbin"
libexecdir="/usr/local/libexec"
amlibexecdir="/usr/local/libexec/amanda"
mandir="/usr/local/share/man" AMANDA_TMPDIR="/tmp/amanda"
AMANDA_DBGDIR="/tmp/amanda"
CONFIG_DIR="/usr/local/etc/amanda" DEV_PREFIX="/dev/"
RDEV_PREFIX="/dev/" DUMP=UNDEF RESTORE=UNDEF VDUMP=UNDEF
VRESTORE=UNDEF XFSDUMP=UNDEF XFSRESTORE=UNDEF VXDUMP=UNDEF
VXRESTORE=UNDEF SAMBA_CLIENT=UNDEF GNUTAR="/bin/gtar"
COMPRESS_PATH="/bin/gzip" UNCOMPRESS_PATH="/bin/gzip"
LPRCMD=UNDEF MAILER=UNDEF
listed_incr_dir="/usr/local/var/amanda/gnutar-lists"
defs: DEFAULT_SERVER="vhost2.summersault.com"
DEFAULT_CONFIG="DailySet1"
DEFAULT_TAPE_SERVER="vhost2.summersault.com"
DEFAULT_TAPE_DEVICE="" NEED_STRSTR AMFLOCK_POSIX
AMFLOCK_FLOCK AMFLOCK_LOCKF AMFLOCK_LNLOCK SETPGRP_VOID
AMANDA_DEBUG_DAYS=4 BSD_SECURITY USE_AMANDAHOSTS
CLIENT_LOGIN="amandabackup" CHECK_USERID HAVE_GZIP
COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
The GNUTAR path is in the paths section.
> Do the amanda user can read /home?
If I put the directories into the disklist, as individual DLEs, like this:
/home/aaronson
/home/aferton
/home/amigos
/home/archery
/home/blank
/home/borders
/home/brynwyn
/home/caronwyn
/home/cirque
/home/common
Then they are backed up without any issues, but they cause problems with
the glob {include "./[a-c]*"}
However, as the amandabackup user, when I cd /home and then ls -lah I
get a permission denied. This could be the reason the glob directive is
failing, since it cannot scan the contents of that directory to put
together an accurate include list for tar.
Unfortunately, each /home/username is owned by username:username as the
UID:GID. It now looks like I need to figure out a way to get the
amandabackup user access via a groupID.
Thanks!
~~Mike
Re: Splitting a DLE with regex
On 07/31/2013 04:00 PM, Mike Neimoyer wrote:
However, as the amandabackup user, when I cd /home and then ls -lah I
get a permission denied. This could be the reason the glob directive
is failing, since it cannot scan the contents of that directory to put
together an accurate include list for tar.
Unfortunately, each /home/username is owned by username:username as
the UID:GID. It now looks like I need to figure out a way to get the
amandabackup user access via a groupID.
Try the amgtar application, it should do the glob expansion as root.
Put the following in the dumptype:
program "APPLICATION"
application {
plugin "amgtar"
}
Jean-Louis
Re: Splitting a DLE with regex
Jean-Louis Martineau schreef op 31-07-13 23:02: On 07/31/2013 04:00 PM, Mike Neimoyer wrote: However, as the amandabackup user, when I cd /home and then ls -lah I get a permission denied. This could be the reason the glob directive is failing, since it cannot scan the contents of that directory to put together an accurate include list for tar. Unfortunately, each /home/username is owned by username:username as the UID:GID. It now looks like I need to figure out a way to get the amandabackup user access via a groupID. Try the amgtar application, it should do the glob expansion as root. With Amanda, plugins are rather new for me but AFIK root-permission is only needed for reading DLE's and is done by means of calling setuid-root executables. There are two of them involved in all Amanda-work: calcsize and dumper, both called by the daemon amandad which is called by xinetd. So in this case, root-permission should not be the problem. Unless of course callcsize and dumper to NOT have a s (from setuid) in front of them or are not owned by root. $ cd /usr/lib/amanda $ ls -l -rwsr-x--- 1 root amanda 19056 28 jan 2013 calcsize -rwsr-x--- 1 root amanda 52592 28 jan 2013 dumper $ All other executables including scripts inherit root permission form these two. Surely, amgtar adds some nice features on top of GNUTAR. Met vriendelijke groeten, AT COMPUTING Gerrit A. Smit Beheer Technische Infrastructuur -- AT Computing Telefoon: +31 24 352 72 22 Linux/UNIXperts, Telefoon cursussecretariaat: +31 24 352 72 72 opleiders & oplossers Fax: +31 24 352 72 92 Kerkenbos 12-38 t...@atcomputing.nl 6546 BE Nijmegen www.atcomputing.nl Nieuw bij AT Computing: onze Linux Reference Card nu ook als gratis app!
