Re: GNUTAR exclude lists not working in Windows or Linux
I've had no time to work on this lately. I will get back to it soon. On Wed, Jun 15, 2005 at 05:39:54PM +0200, Paul Bijnens wrote: Paul Bijnens wrote on 2 June 2005: OK, I'll jump on this. Any progress on this problem? Can you reproduce the problem with a small setup, that I could duplicate here? -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * *** -- Joe Rhett senior geek meer.net
Re: GNUTAR exclude lists not working in Windows or Linux
Paul Bijnens wrote on 2 June 2005: OK, I'll jump on this. Any progress on this problem? Can you reproduce the problem with a small setup, that I could duplicate here? -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: GNUTAR exclude lists not working in Windows or Linux
On Thursday 02 June 2005 23:47, Joe Rhett wrote: Okay, so if it isn't a documentation problem then what do we test now? A test lab just demonstrated what I already knew. On Thu, Jun 02, 2005 at 11:16:18PM +0200, Paul Bijnens wrote: Joe Rhett wrote: In the meantime, can you confirm exclude file versus exclude list ? Someone else reported a different syntax that conflicts with the man page, but actually makes more sense to the naked eye. This may be a documentation problem. The documentation is correct: exclude file ./some*thing this excludes all the files matching name some*thing exclude list /some/file /some/file on the client contains a list of patterns to be excluded I'm curious to see if Joe Rhett made his work last night. With my corrected syntax, my file by file include, overriding the exclude list, appeared to have worked as intended last night. The only puzzlement was that while the amverify output was, as usual, included in the email I got from my wrapper script, backup.sh, I normally get a somewhat nicer formatted email from the amverify run. I did not get that email. So I first installed snapshot 2.4.5-20050603 and re-ran that amverify step for exercise, and did get the email this time. The strange case of the disappearing email strikes again. Odd... -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: GNUTAR exclude lists not working in Windows or Linux
Joe Rhett wrote: Okay, so if it isn't a documentation problem then what do we test now? A test lab just demonstrated what I already knew. You know it, but we don't. And we both would like to know why. Then I would like to have the config of that test: amanda.conf, disklist, the contents of the exclude file, etc. The output of ls -lR of the directory that you tested with. and the corresponding debug files on the client in /tmp/amanda/*.debug The output of ls -lR of the directory that you tested with. Or maybe even a tar.gz of ~amanda/Test completely, including all log files amdump* log.*, info-dir and index-dir? And maybe even the resulting VTape file if not too large? Don't send the large files to the list, send them to me privately. I try to duplicate the problem here. There must be some difference. And until we can understand it, and point it to some specific bug (in the code or in the config), most computer problems look like witchcraft. (That's also why the users see systemadminstrators as wizards.) -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: GNUTAR exclude lists not working in Windows or Linux
On Thursday 02 June 2005 15:50, Joe Rhett wrote:
On Thu, Jun 02, 2005 at 08:02:21AM -0400, Gene Heskett wrote:
And as I noted before, and someone tried to explain away, it
appears that the command line invocation for the exclude list
is wrong (missing an equals) runtar.20050601020202.debug:
running: /bin/tar: gtar --create --file - --directory /
--one-file-system --listed-incremental
/var/lib/amanda/gnutar-lists/client-host3__0.new --sparse
--ignore-failed-read --totals --exclude-from
/etc/exclude.gtar .
I don't know what equals sign you feel is missing.
The man page I have for gnutar does not show an equal sign
needed with the --exclude-from option:
My amanda.conf shows that syntax, however my dumptypes that use
this option use it like this:
exclude file /amanda/excludes
Hm. Mine says exclude list, and according to the man page
exclude [ list|file ][[optional][ append ][ string ]+]
Default: file. There is two exclude list exclude file
and exclude list. With exclude file , the string is a
gnutar exclude expression. With exclude list , the
string is a file name on the client containing gnutar
exclude expression.
Does yours work? If so, perhaps it's a documentation problem?
Yes, mine works as intended, very nicely ignoreing about 2.4 GB worth
of FC3 iso's, which are then covered by an individual disklist entry
per iso involved. This gives amanda a chance to spread the load and
balance the amount backed up per nightly run.
The contents of the above file:
./*.iso
./FC3
./FC3-SRPMS
And it works, albeit in an all linux environment with tar-1.15 on
the server, and tar-1.13-25 on my one client.
Does the server version of tar matter for excludes?
In terms of excludes, I do not know. What I do know is that tar-1.13
is totally busted for other file format reasons, 1.13-19 and 1.13-25
work fine, as does 1.15. 1.14 was fairly shortlived as it apparently
broke a lot of stuff.
Can we have the output of a 'tar --version' on each of these
boxes?
tar (GNU tar) 1.13.25
This is fine AFAIK.
[...]
Also, that line of the revelant dumptype that specifys the exclude
file, starting the lookup by consulting the disklist to get the
actual dumptype being used, and showing us a snip of that dumptype
please.
I had supplied these in my original report, and in the latest report
as well.
In my defense, I did want to see them as they existed most recently.
They were removed from the quoted text. Here it is
again:
disklist:
client-host1 /user-tar
I don't think I've ever done this, I've always declared many more as
this gives amanda room to play, adjusting schedules to equalize the
amount of data backed up each night. My disklist defines just short
of 50 entries, 20 some of which are subdirs of /.
I'm not saying thats the problem, but it does tend to make amanda's
schedueling into a nightmare.
You also have skipped over defineing whether or not its a local drive,
and the network interface. Here is two of my disklist entries:
coyote /usr/games coyote-tar 1 local
gene /bin gene-tar2 le0
wheer the 4th argument is the spindle number, set so that amanda won't
thrash the drives seek mechanism by atempting to access two different
file systems on that one drive. The second line shows a different
spindle number, which amanda will then run in parallel. The
local/le0 switch in the last column tells amanda which interface to
use when talking to a client.
I don't believe amanda is married to this hard specification and won't
work without it, but its still good practice.
amanda.conf:
define dumptype global {
index yes
}
define dumptype root-tar {
global
program GNUTAR
comment root partitions dumped with tar
compress none
exclude list /etc/exclude.gtar
priority low
}
define dumptype user-tar {
root-tar
comment user partitions dumped with tar
priority medium
}
Humm, user-tar includes root-tar, which includes global. And they seem
to be in the proper order (amanda cannot use an as yet unread
dumptype, eg anything included must have been read previously in
order to stack specs as you have done here (and so do I) So this
looks good AFAICT.
I'd try changing the exclude 'list' above to 'file' just for grins.
--
Cheers, Gene
There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: GNUTAR exclude lists not working in Windows or Linux
Okay, so if it isn't a documentation problem then what do we test now? A test lab just demonstrated what I already knew. On Thu, Jun 02, 2005 at 11:16:18PM +0200, Paul Bijnens wrote: Joe Rhett wrote: In the meantime, can you confirm exclude file versus exclude list ? Someone else reported a different syntax that conflicts with the man page, but actually makes more sense to the naked eye. This may be a documentation problem. The documentation is correct: exclude file ./some*thing this excludes all the files matching name some*thing exclude list /some/file /some/file on the client contains a list of patterns to be excluded -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * *** -- Joe Rhett senior geek meer.net
Re: GNUTAR exclude lists not working in Windows or Linux
Gene Heskett wrote: On Thursday 02 June 2005 17:16, Paul Bijnens wrote: The documentation is correct: exclude file ./some*thing this excludes all the files matching name some*thing exclude list /some/file /some/file on the client contains a list of patterns to be excluded I'd argue that point, I'm using 'file' to specify a file that contains a list, and its working just fine. To settle the argument, we need some proof, and correct the bug in the documentation or correct the bug in your configuration. I would like to see the entries as you have them defined in the disklist, and the output of amadmin TheConfig disklist (the ultimate interpretation by amanda, after resolving all included types) e.g.: My disklist entry in the test configuration: amatest/space/scratch/topdir user-tar And then this command would give the ultimate configuration: $ amadmin test disklist amatest '^/space/scratch/topdir$' line 26: host amatest: interface default disk /space/scratch/topdir: program GNUTAR exclude list /var/opt/amanda/exclude.gtar priority 1 dumpcycle 0 maxdumps 1 maxpromoteday 1 strategy STANDARD compress NONE auth BSD kencrypt NO holdingdisk YES record NO index YES skip-incr NO skip-full NO Where we can see that the dumptype user-tar somehow has an exclude defined. Moreover, when I specify exclude file /non/existing/file, then amcheck does not complain. However with exclude list /non/existing/file, amcheck complains about the non-existing file, just as expected. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: GNUTAR exclude lists not working in Windows or Linux
On Friday 03 June 2005 06:23, Paul Bijnens wrote: Gene Heskett wrote: On Thursday 02 June 2005 17:16, Paul Bijnens wrote: The documentation is correct: exclude file ./some*thing this excludes all the files matching name some*thing exclude list /some/file /some/file on the client contains a list of patterns to be excluded I'd argue that point, I'm using 'file' to specify a file that contains a list, and its working just fine. See below, I'm full of it. I'd given up making it work and found a workaround of sorts. And had forgotten about it. Memory, second thing to go you know. :) To settle the argument, we need some proof, and correct the bug in the documentation or correct the bug in your configuration. I would like to see the entries as you have them defined in the disklist, and the output of amadmin TheConfig disklist (the ultimate interpretation by amanda, after resolving all included types) e.g.: My disklist entry in the test configuration: amatest/space/scratch/topdir user-tar And then this command would give the ultimate configuration: $ amadmin test disklist amatest '^/space/scratch/topdir$' line 26: host amatest: interface default disk /space/scratch/topdir: program GNUTAR exclude list /var/opt/amanda/exclude.gtar priority 1 dumpcycle 0 maxdumps 1 maxpromoteday 1 strategy STANDARD compress NONE auth BSD kencrypt NO holdingdisk YES record NO index YES skip-incr NO skip-full NO Where we can see that the dumptype user-tar somehow has an exclude defined. Moreover, when I specify exclude file /non/existing/file, then amcheck does not complain. However with exclude list /non/existing/file, amcheck complains about the non-existing file, just as expected. Here, the file exists, and if I ran a diff against the two outputs of the above command, one when its says file, and one when it says list, the only diff would be to remove this line: -exclude file /amanda/excludes with: +exclude list /amanda/excludes Here is the output of that command for 'file' against coyote /bin: line 109: host coyote: interface LOCAL disk /bin: program GNUTAR exclude file /amanda/excludes priority 0 dumpcycle 5 maxdumps 4 maxpromoteday 1 bumpsize 10240 bumpdays 1 bumpmult 2.00 strategy STANDARD estimate CLIENT compress NONE auth BSD kencrypt NO holdingdisk YES record YES index YES skip-incr NO skip-full NO and here is that same line after changing it to 'list': line 109: host coyote: interface LOCAL disk /bin: program GNUTAR exclude list /amanda/excludes priority 0 dumpcycle 5 maxdumps 4 maxpromoteday 1 bumpsize 10240 bumpdays 1 bumpmult 2.00 strategy STANDARD estimate CLIENT compress NONE auth BSD kencrypt NO holdingdisk YES record YES index YES skip-incr NO skip-full NO Now, lets see if the files I wanted excluded are in the tarballs. I am indeed wrong! They are as the /usr/FC3 backup at the last level 0 is indeed 2.3 GB. So I am mistaken. But, if indeed it works while using the word 'list', then I should remove those 3 dirs from the disklist, and re-enable the entries for the individual files, one a day until all are enabled again. An interesting observation here, since I was wrong. The 'file' usage in that event would never have had a 'hit' because while the dumptype is specified, thats not the dumptype used to backup the directory that file lives in. And when I started this little exercise this morning, that disklist entry didn't use a dumptype that had the exclude. As a test for tonight, I've used a dumptype that specifies the excludes list, which will exclude the contents of that directory, but then specified just one of the files in that directory with an include directive. So that one then looks like this: line 195: host coyote: interface LOCAL disk /usr/dlds-misc/FC3/FC3-i386-rescuecd.iso: device /usr/dlds-misc/FC3 program GNUTAR exclude list /amanda/excludes include file ./FC3-i386-rescuecd.iso priority 0 dumpcycle 5 maxdumps 4 maxpromoteday 1 bumpsize 10240 bumpdays 1 bumpmult 2.00 strategy STANDARD estimate CLIENT compress NONE auth BSD kencrypt NO holdingdisk YES record YES index YES skip-incr NO skip-full NO Does this look kosher? Actually, I should quit backing this FC3 stuff up, I'll probably never install it, I'll be jumping to FC4 when the final is out, its in
Re: GNUTAR exclude lists not working in Windows or Linux
Joe Rhett wrote: These systems I can test with to my heart's content, unlike those Windows boxes, so tell me what you need to know. OK, I'll jump on this. I can assure you there is at least one configuration in the world where the excludes do work. First some general info needed: Amanda version on server and client (is client the server itself too?) Gnutar version of server and client? And OS version of server and client? To test and experiment with amanda, it's nice to set up a special config for this, and the file-driver is perfect. (we're testing excludes, not tape drives.). Create a new config for amanda, e.g. Test, which has it's own amanda.conf and disklist. Configure the chg-disk driver for this configuration, see: http://www.amanda.org/docs/howto-filedriver.html Now set up a little disklist too, backing up only a small directory where you have complete control (create files or directories with names you need to test. And make sure the amcheck Test works and a basic amdump Test works too. Do not make the disklist too large, so that amadmin Test force followed by amdump Test runs a few minutes at most. Do not add filesystems or directories to the disklist that also are added in the production disklists. And if you really need to, add record no to its disktype, to avoid interference with your production environment. Creating such a setup takes about an hour, but it's very handy to experiment with amanda. I mean, it's not a wasted resource for the future either. Once you have set it up, try to duplicate the problem. I have already done that all here, and I cannot reproduce the problem. That means there must be some difference between your config and mine. We just need to get that nailed down. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: GNUTAR exclude lists not working in Windows or Linux
On Wed, Jun 01, 2005 at 10:11:07PM -0700, Joe Rhett wrote: On Wed, Jun 01, 2005 at 11:27:11PM -0400, Jon LaBadie wrote: On Wed, Jun 01, 2005 at 07:25:44PM -0700, Joe Rhett wrote: Okay, last year I had observed that perfectly valid gnutar exclude lists were being ignored by amanda on Windows machines. The best answer anyone could give me was to build my own tar program that does the excludes, and replace runtar. I forget the discussion, but there must have been more than that, A bit more, mostly telling me to use SMB instead, which is nonsense. backing up of windows boxes does not use gnutar but smbclient. Different cmd line syntax for excludes. Every time I talk about Windows, you keep talking about smbclient. Nobody, flat nobody that I know of is stupid enough to run SMB on a public webserver. The windows machines in question are using amanda under cygwin. Gee Joe, I apologize. Somehow I lost track of the point that you use cygwin. It should have been obvious to me since in the past year there was only one posting about cygwin besides yours. I can't imagine why I might have assumed that like 95% of the other posters you were using samba to backup a windows box. And as I noted before, and someone tried to explain away, it appears that the command line invocation for the exclude list is wrong (missing an equals) runtar.20050601020202.debug: running: /bin/tar: gtar --create --file - --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/client-host3__0.new --sparse --ignore-failed-read --totals --exclude-from /etc/exclude.gtar . I don't know what equals sign you feel is missing. The man page I have for gnutar does not show an equal sign needed with the --exclude-from option: Some versions of tar out there apparently don't work properly without the equals sign. Search for it, or trust me. In any case, I agree that it isn't affecting these linux boxes -- either syntax appears to work. /etc/exclude.gtar $ cat /etc/exclude.gtar ./* ,/ * If I recall the syntax correctly, each entry must begin ./ The first is valid and would exclude everything with the possible exception of dot files (eg .profile). Second and third are invalid. That's fine, I was trying everything possible. Right now with those regexs I'm backing up 60gb a night from that system. Wouldn't that suggest something is wrong? I started with just the first regex and added others when it didn't work, so it isn't due to the bad regexes... Actually I think it somehow is. I ran a few tests with my gtar. The command line was a pipe from one gtar creating an archive to another generating a listing. I wanted to see the effect of various exclude patterns in the file. My commandline was like this: amgtar --create --exclude-from /var/tmp/exclude \ --directory /tmp --file - . | amgtar --list --file - I ran it first without the --exclude-from option, then with the option and an empty file. Same large list of files. When I ran it with the option, but with the exclude file missing I got an error and no archive was created. Next I put a simple pair of patterns in the file: ./s2 ./s4 and the file s2 was excluded, there was no s4 to begin with. The next test was to put in the exclude file a single entry, ./* This resulted in a single line of output, ./ i.e. no files were archived but the directory was noted. Lastly I put in your three patterns, ./* ,/* * The result I got was that nothing was archived, not even . I looked at the last two a little further, collected the result of the gtar creation into a file instead of a pipe. Each command created a file exactly 10K. The one from a single exclude entry (./*) had a little data in the first few hundred bytes, then nulls. The file command recognized this as a tar file. The corresponding 10K file created with your three patterns was simply a file of null bytes. No data and of course it was not recognized by the file command as a tar file. Perhaps running the command by hand like this could help sort out the exclude patterns for you. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: GNUTAR exclude lists not working in Windows or Linux
On Thursday 02 June 2005 01:11, Joe Rhett wrote: On Wed, Jun 01, 2005 at 11:27:11PM -0400, Jon LaBadie wrote: On Wed, Jun 01, 2005 at 07:25:44PM -0700, Joe Rhett wrote: Okay, last year I had observed that perfectly valid gnutar exclude lists were being ignored by amanda on Windows machines. The best answer anyone could give me was to build my own tar program that does the excludes, and replace runtar. I forget the discussion, but there must have been more than that, A bit more, mostly telling me to use SMB instead, which is nonsense. backing up of windows boxes does not use gnutar but smbclient. Different cmd line syntax for excludes. Every time I talk about Windows, you keep talking about smbclient. Nobody, flat nobody that I know of is stupid enough to run SMB on a public webserver. The windows machines in question are using amanda under cygwin. However, having an identical problem on Linux proves that this isn't related to Windows or Cygwin. I know you've got your head wrapped around smbclient, so let's drop the windows and focus on Linux. Tar isn't honoring the exclude files on Linux. Well now I've enabled my first gnutar linux clients, and they are seeing the exact same problem. /tmp/amanda/sendsize.debug and runtar.debug both show that the exclude list is being passed to tar, but they are ignored. Just to clarify, the entire system (60gb+) is backed up every night. And as I noted before, and someone tried to explain away, it appears that the command line invocation for the exclude list is wrong (missing an equals) runtar.20050601020202.debug: running: /bin/tar: gtar --create --file - --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/client-host3__0.new --sparse --ignore-failed-read --totals --exclude-from /etc/exclude.gtar . I don't know what equals sign you feel is missing. The man page I have for gnutar does not show an equal sign needed with the --exclude-from option: My amanda.conf shows that syntax, however my dumptypes that use this option use it like this: exclude file /amanda/excludes The contents of the above file: ./*.iso ./FC3 ./FC3-SRPMS And it works, albeit in an all linux environment with tar-1.15 on the server, and tar-1.13-25 on my one client. Some versions of tar out there apparently don't work properly without the equals sign. Search for it, or trust me. In any case, I agree that it isn't affecting these linux boxes -- either syntax appears to work. /etc/exclude.gtar $ cat /etc/exclude.gtar ./* ,/ * If I recall the syntax correctly, each entry must begin ./ The first is valid and would exclude everything with the possible exception of dot files (eg .profile). Second and third are invalid. That's fine, I was trying everything possible. Right now with those regexs I'm backing up 60gb a night from that system. Wouldn't that suggest something is wrong? I started with just the first regex and added others when it didn't work, so it isn't due to the bad regexes... Can we have the output of a 'tar --version' on each of these boxes? Also, that line of the revelant dumptype that specifys the exclude file, starting the lookup by consulting the disklist to get the actual dumptype being used, and showing us a snip of that dumptype please. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: GNUTAR exclude lists not working in Windows or Linux
On Wed, Jun 01, 2005 at 10:11:07PM -0700, Joe Rhett wrote: That's fine, I was trying everything possible. Right now with those regexs I'm backing up 60gb a night from that system. Wouldn't that suggest something is wrong? I started with just the first regex and added others when it didn't work, so it isn't due to the bad regexes... On Thu, Jun 02, 2005 at 02:22:27AM -0400, Jon LaBadie wrote: Actually I think it somehow is. I ran a few tests with my gtar. The command line was a pipe from one gtar creating an archive to another generating a listing. I wanted to see the effect of various exclude patterns in the file. My commandline was like this: amgtar --create --exclude-from /var/tmp/exclude \ --directory /tmp --file - . | amgtar --list --file - I ran it first without the --exclude-from option, then with the option and an empty file. Same large list of files. When I ran it with the option, but with the exclude file missing I got an error and no archive was created. Next I put a simple pair of patterns in the file: ./s2 ./s4 and the file s2 was excluded, there was no s4 to begin with. The next test was to put in the exclude file a single entry, ./* This resulted in a single line of output, ./ i.e. no files were archived but the directory was noted. Lastly I put in your three patterns, ./* ,/* * The result I got was that nothing was archived, not even . I looked at the last two a little further, collected the result of the gtar creation into a file instead of a pipe. Each command created a file exactly 10K. The one from a single exclude entry (./*) had a little data in the first few hundred bytes, then nulls. The file command recognized this as a tar file. The corresponding 10K file created with your three patterns was simply a file of null bytes. No data and of course it was not recognized by the file command as a tar file. Perhaps running the command by hand like this could help sort out the exclude patterns for you. That's nice. Ignore a consistent and repeatable bug report for over a year, then be sarcastic to the reporter. I HAVE run these commands by hand. I sent directly to you the results of running these commands by hand last year, versus the results displayed in the amanda report. And besides, you are making my arguement for me. Your results show that no files should have been backed up with the file as shown, however 62gb was backed up the previous night. Now that your own tests have demonstrated that running the command by hand works, yet amanda is getting something different, would you possibly take an interest in finding out why? -- Joe Rhett senior geek meer.net
Re: GNUTAR exclude lists not working in Windows or Linux
On Thu, Jun 02, 2005 at 08:02:21AM -0400, Gene Heskett wrote:
And as I noted before, and someone tried to explain away, it
appears that the command line invocation for the exclude list is
wrong (missing an equals) runtar.20050601020202.debug:
running: /bin/tar: gtar --create --file - --directory /
--one-file-system --listed-incremental
/var/lib/amanda/gnutar-lists/client-host3__0.new --sparse
--ignore-failed-read --totals --exclude-from /etc/exclude.gtar .
I don't know what equals sign you feel is missing.
The man page I have for gnutar does not show an equal sign
needed with the --exclude-from option:
My amanda.conf shows that syntax, however my dumptypes that use this
option use it like this:
exclude file /amanda/excludes
Hm. Mine says exclude list, and according to the man page
exclude [ list|file ][[optional][ append ][ string ]+]
Default: file. There is two exclude list exclude file
and exclude list. With exclude file , the string is a
gnutar exclude expression. With exclude list , the
string is a file name on the client containing gnutar
exclude expression.
Does yours work? If so, perhaps it's a documentation problem?
The contents of the above file:
./*.iso
./FC3
./FC3-SRPMS
And it works, albeit in an all linux environment with tar-1.15 on the
server, and tar-1.13-25 on my one client.
Does the server version of tar matter for excludes?
Can we have the output of a 'tar --version' on each of these boxes?
tar (GNU tar) 1.13.25
Copyright (C) 2001 Free Software Foundation, Inc.
This program comes with NO WARRANTY, to the extent permitted by law.
You may redistribute it under the terms of the GNU General Public License;
see the file named COPYING for details.
Written by John Gilmore and Jay Fenlason.
Also, that line of the revelant dumptype that specifys the exclude
file, starting the lookup by consulting the disklist to get the
actual dumptype being used, and showing us a snip of that dumptype
please.
I had supplied these in my original report, and in the latest report as
well. They were removed from the quoted text. Here it is again:
disklist:
client-host1 /user-tar
amanda.conf:
define dumptype global {
index yes
}
define dumptype root-tar {
global
program GNUTAR
comment root partitions dumped with tar
compress none
exclude list /etc/exclude.gtar
priority low
}
define dumptype user-tar {
root-tar
comment user partitions dumped with tar
priority medium
}
--
Joe Rhett
senior geek
meer.net
Re: GNUTAR exclude lists not working in Windows or Linux
Thanks for the advice on doing that. I had such an environment set up last year, and was able to replicate it at will on the Windows boxes. I'll set it up again for the linux systems. Working this out on Windows stalled because I was reluctant to toy with these production systems. Windows doesn't need any help to be shaky ;-) In the meantime, can you confirm exclude file versus exclude list ? Someone else reported a different syntax that conflicts with the man page, but actually makes more sense to the naked eye. This may be a documentation problem. On Thu, Jun 02, 2005 at 10:33:23AM +0200, Paul Bijnens wrote: Joe Rhett wrote: These systems I can test with to my heart's content, unlike those Windows boxes, so tell me what you need to know. OK, I'll jump on this. I can assure you there is at least one configuration in the world where the excludes do work. First some general info needed: Amanda version on server and client (is client the server itself too?) Gnutar version of server and client? And OS version of server and client? To test and experiment with amanda, it's nice to set up a special config for this, and the file-driver is perfect. (we're testing excludes, not tape drives.). Create a new config for amanda, e.g. Test, which has it's own amanda.conf and disklist. Configure the chg-disk driver for this configuration, see: http://www.amanda.org/docs/howto-filedriver.html Now set up a little disklist too, backing up only a small directory where you have complete control (create files or directories with names you need to test. And make sure the amcheck Test works and a basic amdump Test works too. Do not make the disklist too large, so that amadmin Test force followed by amdump Test runs a few minutes at most. Do not add filesystems or directories to the disklist that also are added in the production disklists. And if you really need to, add record no to its disktype, to avoid interference with your production environment. Creating such a setup takes about an hour, but it's very handy to experiment with amanda. I mean, it's not a wasted resource for the future either. Once you have set it up, try to duplicate the problem. I have already done that all here, and I cannot reproduce the problem. That means there must be some difference between your config and mine. We just need to get that nailed down. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * *** -- Joe Rhett senior geek meer.net
Re: GNUTAR exclude lists not working in Windows or Linux
Joe Rhett wrote: In the meantime, can you confirm exclude file versus exclude list ? Someone else reported a different syntax that conflicts with the man page, but actually makes more sense to the naked eye. This may be a documentation problem. The documentation is correct: exclude file ./some*thing this excludes all the files matching name some*thing exclude list /some/file /some/file on the client contains a list of patterns to be excluded -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: GNUTAR exclude lists not working in Windows or Linux
On Wed, Jun 01, 2005 at 07:25:44PM -0700, Joe Rhett wrote: Okay, last year I had observed that perfectly valid gnutar exclude lists were being ignored by amanda on Windows machines. The best answer anyone could give me was to build my own tar program that does the excludes, and replace runtar. I forget the discussion, but there must have been more than that, backing up of windows boxes does not use gnutar but smbclient. Different cmd line syntax for excludes. Well now I've enabled my first gnutar linux clients, and they are seeing the exact same problem. /tmp/amanda/sendsize.debug and runtar.debug both show that the exclude list is being passed to tar, but they are ignored. These systems I can test with to my heart's content, unlike those Windows boxes, so tell me what you need to know. And as I noted before, and someone tried to explain away, it appears that the command line invocation for the exclude list is wrong (missing an equals) runtar.20050601020202.debug: running: /bin/tar: gtar --create --file - --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/client-host3__0.new --sparse --ignore-failed-read --totals --exclude-from /etc/exclude.gtar . I don't know what equals sign you feel is missing. The man page I have for gnutar does not show an equal sign needed with the --exclude-from option: SYNOPSIS tar [ - ] A --catenate --concatenate | c --create [ -- atime-preserve ] [ -b, --block-size N ] [ -B, --read-full- ... --confirmation ] [ -W, --verify] [ --exclude FILE ] [ - X, --exclude-from FILE ] [ -Z, --compress, --uncompress ] [ ... -X, --exclude-from FILE exclude files listed in FILE /etc/exclude.gtar $ cat /etc/exclude.gtar ./* ,/ * If I recall the syntax correctly, each entry must begin ./ The first is valid and would exclude everything with the possible exception of dot files (eg .profile). Second and third are invalid. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: GNUTAR exclude lists not working in Windows or Linux
On Wed, Jun 01, 2005 at 11:27:11PM -0400, Jon LaBadie wrote: On Wed, Jun 01, 2005 at 07:25:44PM -0700, Joe Rhett wrote: Okay, last year I had observed that perfectly valid gnutar exclude lists were being ignored by amanda on Windows machines. The best answer anyone could give me was to build my own tar program that does the excludes, and replace runtar. I forget the discussion, but there must have been more than that, A bit more, mostly telling me to use SMB instead, which is nonsense. backing up of windows boxes does not use gnutar but smbclient. Different cmd line syntax for excludes. Every time I talk about Windows, you keep talking about smbclient. Nobody, flat nobody that I know of is stupid enough to run SMB on a public webserver. The windows machines in question are using amanda under cygwin. However, having an identical problem on Linux proves that this isn't related to Windows or Cygwin. I know you've got your head wrapped around smbclient, so let's drop the windows and focus on Linux. Tar isn't honoring the exclude files on Linux. Well now I've enabled my first gnutar linux clients, and they are seeing the exact same problem. /tmp/amanda/sendsize.debug and runtar.debug both show that the exclude list is being passed to tar, but they are ignored. Just to clarify, the entire system (60gb+) is backed up every night. And as I noted before, and someone tried to explain away, it appears that the command line invocation for the exclude list is wrong (missing an equals) runtar.20050601020202.debug: running: /bin/tar: gtar --create --file - --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/client-host3__0.new --sparse --ignore-failed-read --totals --exclude-from /etc/exclude.gtar . I don't know what equals sign you feel is missing. The man page I have for gnutar does not show an equal sign needed with the --exclude-from option: Some versions of tar out there apparently don't work properly without the equals sign. Search for it, or trust me. In any case, I agree that it isn't affecting these linux boxes -- either syntax appears to work. /etc/exclude.gtar $ cat /etc/exclude.gtar ./* ,/ * If I recall the syntax correctly, each entry must begin ./ The first is valid and would exclude everything with the possible exception of dot files (eg .profile). Second and third are invalid. That's fine, I was trying everything possible. Right now with those regexs I'm backing up 60gb a night from that system. Wouldn't that suggest something is wrong? I started with just the first regex and added others when it didn't work, so it isn't due to the bad regexes... -- Joe Rhett senior geek meer.net
