RE: Breaking DLEs up
I have been using a very similar setup for years, though I did not have any quotes in the first line of each DLE. I do NOT believe the quotes are an issue. What do the /tmp/amanda files show for these attempted dumps? From: owner-amanda-us...@amanda.org On Behalf Of Chris Nighswonger Sent: Thursday, November 8, 2018 11:12 AM To: amanda-users@amanda.org Subject: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. I attempted this and it appears to not have worked. I'm not sure why. Here is the relevant portion of my DLEs: fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } fileserver "/netdrives/CAMPUS/gl" "/netdrives/CAMPUS" { comp-tar include "./[g-l]*" estimate server } fileserver "/netdrives/CAMPUS/mr" "/netdrives/CAMPUS" { comp-tar include "./[m-r]*" estimate server } fileserver "/netdrives/CAMPUS/sz" "/netdrives/CAMPUS" { comp-tar include "./[s-z]*" estimate server } Here are the corresponding lines from amreport for the last backup run: fileserver:/netdrives/CAMPUS/af 0 1k dump done (5:28:16), waiting for writing to tape fileserver:/netdrives/CAMPUS/gl 0 1k dump done (5:28:11), waiting for writing to tape fileserver:/netdrives/CAMPUS/mr 0 1k dump done (5:28:06), waiting for writing to tape fileserver:/netdrives/CAMPUS/sz 0 1k dump done (5:27:11), waiting for writing to tape Kind regards, Chris
Re: Breaking DLEs up
Oddly enough, /tmp/amanda is empty. On Thu, Nov 8, 2018 at 11:33 AM Cuttler, Brian R (HEALTH) < brian.cutt...@health.ny.gov> wrote: > I have been using a very similar setup for years, though I did not have > any quotes in the first line of each DLE. I do NOT believe the quotes are > an issue. > > > > What do the /tmp/amanda files show for these attempted dumps? > > > > *From:* owner-amanda-us...@amanda.org *On > Behalf Of *Chris Nighswonger > *Sent:* Thursday, November 8, 2018 11:12 AM > *To:* amanda-users@amanda.org > *Subject:* Breaking DLEs up > > > > *ATTENTION: This email came from an external source. Do not open > attachments or click on links from unknown senders or unexpected emails.* > > I attempted this and it appears to not have worked. I'm not sure why. > > > > Here is the relevant portion of my DLEs: > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > comp-tar > include "./[a-f]*" > estimate server > } > fileserver "/netdrives/CAMPUS/gl" "/netdrives/CAMPUS" { > comp-tar > include "./[g-l]*" > estimate server > } > fileserver "/netdrives/CAMPUS/mr" "/netdrives/CAMPUS" { > comp-tar > include "./[m-r]*" > estimate server > } > fileserver "/netdrives/CAMPUS/sz" "/netdrives/CAMPUS" { > comp-tar > include "./[s-z]*" > estimate server > } > > > > Here are the corresponding lines from amreport for the last backup run: > > > > fileserver:/netdrives/CAMPUS/af > 0 1k dump done (5:28:16), waiting for writing to tape > fileserver:/netdrives/CAMPUS/gl > 0 1k dump done (5:28:11), waiting for writing to tape > fileserver:/netdrives/CAMPUS/mr > 0 1k dump done (5:28:06), waiting for writing to tape > fileserver:/netdrives/CAMPUS/sz > 0 1k dump done (5:27:11), waiting for writing to tape > > > > Kind regards, > > Chris >
RE: Breaking DLEs up
Client and server side? /var/log/amanda/ ? From: Chris Nighswonger Sent: Thursday, November 8, 2018 11:43 AM To: Cuttler, Brian R (HEALTH) Cc: amanda-users@amanda.org Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. Oddly enough, /tmp/amanda is empty. On Thu, Nov 8, 2018 at 11:33 AM Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> wrote: I have been using a very similar setup for years, though I did not have any quotes in the first line of each DLE. I do NOT believe the quotes are an issue. What do the /tmp/amanda files show for these attempted dumps? From: owner-amanda-us...@amanda.org<mailto:owner-amanda-us...@amanda.org> mailto:owner-amanda-us...@amanda.org>> On Behalf Of Chris Nighswonger Sent: Thursday, November 8, 2018 11:12 AM To: amanda-users@amanda.org<mailto:amanda-users@amanda.org> Subject: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. I attempted this and it appears to not have worked. I'm not sure why. Here is the relevant portion of my DLEs: fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } fileserver "/netdrives/CAMPUS/gl" "/netdrives/CAMPUS" { comp-tar include "./[g-l]*" estimate server } fileserver "/netdrives/CAMPUS/mr" "/netdrives/CAMPUS" { comp-tar include "./[m-r]*" estimate server } fileserver "/netdrives/CAMPUS/sz" "/netdrives/CAMPUS" { comp-tar include "./[s-z]*" estimate server } Here are the corresponding lines from amreport for the last backup run: fileserver:/netdrives/CAMPUS/af 0 1k dump done (5:28:16), waiting for writing to tape fileserver:/netdrives/CAMPUS/gl 0 1k dump done (5:28:11), waiting for writing to tape fileserver:/netdrives/CAMPUS/mr 0 1k dump done (5:28:06), waiting for writing to tape fileserver:/netdrives/CAMPUS/sz 0 1k dump done (5:27:11), waiting for writing to tape Kind regards, Chris
RE: Breaking DLEs up
What does amcheck say, no logs may mean authentication failure between client and server. Try running the client on the client side, as the amanda user, from the command line. You should get a log, on occasion I’ve seen failures that proved to be a path issue to the binary in the inetd.conf (or equiv). From: Cuttler, Brian R (HEALTH) Sent: Thursday, November 8, 2018 12:00 PM To: 'Chris Nighswonger' Cc: amanda-users@amanda.org Subject: RE: Breaking DLEs up Client and server side? /var/log/amanda/ ? From: Chris Nighswonger mailto:cnighswon...@foundations.edu>> Sent: Thursday, November 8, 2018 11:43 AM To: Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> Cc: amanda-users@amanda.org<mailto:amanda-users@amanda.org> Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. Oddly enough, /tmp/amanda is empty. On Thu, Nov 8, 2018 at 11:33 AM Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> wrote: I have been using a very similar setup for years, though I did not have any quotes in the first line of each DLE. I do NOT believe the quotes are an issue. What do the /tmp/amanda files show for these attempted dumps? From: owner-amanda-us...@amanda.org<mailto:owner-amanda-us...@amanda.org> mailto:owner-amanda-us...@amanda.org>> On Behalf Of Chris Nighswonger Sent: Thursday, November 8, 2018 11:12 AM To: amanda-users@amanda.org<mailto:amanda-users@amanda.org> Subject: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. I attempted this and it appears to not have worked. I'm not sure why. Here is the relevant portion of my DLEs: fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } fileserver "/netdrives/CAMPUS/gl" "/netdrives/CAMPUS" { comp-tar include "./[g-l]*" estimate server } fileserver "/netdrives/CAMPUS/mr" "/netdrives/CAMPUS" { comp-tar include "./[m-r]*" estimate server } fileserver "/netdrives/CAMPUS/sz" "/netdrives/CAMPUS" { comp-tar include "./[s-z]*" estimate server } Here are the corresponding lines from amreport for the last backup run: fileserver:/netdrives/CAMPUS/af 0 1k dump done (5:28:16), waiting for writing to tape fileserver:/netdrives/CAMPUS/gl 0 1k dump done (5:28:11), waiting for writing to tape fileserver:/netdrives/CAMPUS/mr 0 1k dump done (5:28:06), waiting for writing to tape fileserver:/netdrives/CAMPUS/sz 0 1k dump done (5:27:11), waiting for writing to tape Kind regards, Chris
Re: Breaking DLEs up
>From the client: Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 euid 10195 version 3.3.1: start at Thu Nov 8 05:27:11 2018 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Version 3.3.1 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 euid 10195 version 3.3.1: rename at Thu Nov 8 05:27:11 2018 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed request as: program `GNUTAR' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: disk `/netdrives/CAMPUS/sz' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: device `/netdrives/CAMPUS' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: level 0 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: since NODATE Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: options `' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: datapath `AMANDA' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: start: host:/netdrives/CAMPUS/sz lev 0 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/bin/gzip /bin/gzip --fast" in pipeline Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: pid 23694: /bin/gzipThu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23694: /bin/gzip --fast Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: doing level 0 dump as listed-incremental to '/var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Nothing found to include for disk /netdrives/CAMPUS/sz Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/usr/libexec/amanda/runtar runtar campus /bin/tar --create --file - --directory /netdrives/CAMPUS --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new --sparse --ignore-failed-read --totals --files-from /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include" in pipeline Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: /usr/libexec/amanda/runtar: pid 23696 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started backup Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started index creator: "/bin/tar -tf - 2>/dev/null | sed -e 's/^\.//'" Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: 46:size(|): Total bytes written: 10240 (10KiB, 78MiB/s) Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Index created successfully Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed backup messages Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 finish time Thu Nov 8 05:27:11 2018 >From the server: /var/log/amanda/server/campus/dumper.20181108020002007.debug:165691:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: getcmd: PORT-DUMP 03-00023 50013 1 host 9efefbff1f /netdrives/CAMPUS/sz /netdrives/CAMPUS 0 1970:1:1:0:0:0 GNUTAR "" "" "" "" bsdtcp AMANDA 127.0.0.1:50014 20 |" bsdtcp\n FAST\n YES\n YES\n AMANDA\n \n./[s-z]*\n \n" /var/log/amanda/server/campus/dumper.20181108020002007.debug:165705: /netdrives/CAMPUS/sz /var/log/amanda/server/campus/dumper.20181108020002007.debug:165706: /netdrives/CAMPUS /var/log/amanda/server/campus/dumper.20181108020002007.debug:165740: /netdrives/CAMPUS/sz /var/log/amanda/server/campus/dumper.20181108020002007.debug:165741: /netdrives/CAMPUS /var/log/amanda/server/campus/dumper.20181108020002007.debug:165857:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: Building type FILE header of 32768-32768 bytes with name='host' disk='/netdrives/CAMPUS/sz' dumplevel=0 and blocksize=32768 /var/log/amanda/server/campus/dumper.20181108020002007.debug:165944:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: Building type FILE header of 32768-32768 bytes with name='host' disk='/netdrives/CAMPUS/sz' dumplevel=0 and blocksize=32768 On Thu, Nov 8, 2018 at 12:00 PM Cuttler, Brian R (HEALTH) < brian.cutt...@health.ny.gov> wrote: > Client and server side? > > /var/log/amanda/ ? > > > > > > *From:* Chris Nighswonger > *Sent:* Thursday, November 8, 2018 11:43 AM > *To:* Cuttler, Brian R (HEALTH) > *Cc:* amanda-users@amanda.org > *Subject:* Re: Breaking DLEs up > > > > *ATTENTION: This email came from an external source. Do not open > attachments or click on links from unknown senders or unexpected emails.* > > Oddly enough, /tmp/amanda is empty. > > > > On Thu, Nov 8, 2018 at 11:33 AM Cuttler, Brian R (HEALTH) < > brian.cutt...@health.ny.gov> wrote: > > I have been using a very similar setup for years, though I did not have > any quotes in the first line of each DLE. I do NOT believe the quotes are > an issue. > > > > What do the /tmp/amanda files show for these attempted dumps? > > > > *From:* owner-amanda-us...@amanda.org *On > Behalf
RE: Breaking DLEs up
Stupid question, host fileserver, directories /netdrives/CAMPUS/s* to /netdrives/CAMPUS/z* exist and have some files in them? From: Chris Nighswonger Sent: Thursday, November 8, 2018 12:11 PM To: Cuttler, Brian R (HEALTH) Cc: amanda-users@amanda.org Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. From the client: Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 euid 10195 version 3.3.1: start at Thu Nov 8 05:27:11 2018 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Version 3.3.1 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 euid 10195 version 3.3.1: rename at Thu Nov 8 05:27:11 2018 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed request as: program `GNUTAR' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: disk `/netdrives/CAMPUS/sz' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: device `/netdrives/CAMPUS' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: level 0 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: since NODATE Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: options `' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: datapath `AMANDA' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: start: host:/netdrives/CAMPUS/sz lev 0 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/bin/gzip /bin/gzip --fast" in pipeline Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: pid 23694: /bin/gzipThu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23694: /bin/gzip --fast Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: doing level 0 dump as listed-incremental to '/var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new' Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Nothing found to include for disk /netdrives/CAMPUS/sz Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/usr/libexec/amanda/runtar runtar campus /bin/tar --create --file - --directory /netdrives/CAMPUS --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new --sparse --ignore-failed-read --totals --files-from /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include" in pipeline Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: /usr/libexec/amanda/runtar: pid 23696 Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started backup Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started index creator: "/bin/tar -tf - 2>/dev/null | sed -e 's/^\.//'" Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: 46:size(|): Total bytes written: 10240 (10KiB, 78MiB/s) Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Index created successfully Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed backup messages Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 finish time Thu Nov 8 05:27:11 2018 From the server: /var/log/amanda/server/campus/dumper.20181108020002007.debug:165691:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: getcmd: PORT-DUMP 03-00023 50013 1 host 9efefbff1f /netdrives/CAMPUS/sz /netdrives/CAMPUS 0 1970:1:1:0:0:0 GNUTAR "" "" "" "" bsdtcp AMANDA 127.0.0.1:50014<http://127.0.0.1:50014> 20 |" bsdtcp\n FAST\n YES\n YES\n AMANDA\n \n./[s-z]*\n \n" /var/log/amanda/server/campus/dumper.20181108020002007.debug:165705: /netdrives/CAMPUS/sz /var/log/amanda/server/campus/dumper.20181108020002007.debug:165706: /netdrives/CAMPUS /var/log/amanda/server/campus/dumper.20181108020002007.debug:165740: /netdrives/CAMPUS/sz /var/log/amanda/server/campus/dumper.20181108020002007.debug:165741: /netdrives/CAMPUS /var/log/amanda/server/campus/dumper.20181108020002007.debug:165857:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: Building type FILE header of 32768-32768 bytes with name='host' disk='/netdrives/CAMPUS/sz' dumplevel=0 and blocksize=32768 /var/log/amanda/server/campus/dumper.20181108020002007.debug:165944:Thu Nov 8 05:27:11 2018: thd-0x5579048e2400: dumper: Building type FILE header of 32768-32768 bytes with name='host' disk='/netdrives/CAMPUS/sz' dumplevel=0 and blocksize=32768 On Thu, Nov 8, 2018 at 12:00 PM Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> wrote: Client and server side? /var/log/amanda/ ? From: Chris Nighswonger mailto:cnighswon...@foundations.edu>> Sent: Thursday, November 8, 2018 11:43 AM To: Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> Cc: amanda-users@amanda.org<mailto:amanda-users@amanda.org> Subject: Re: Bre
Re: Breaking DLEs up
Yeah, I have found that with TAR, there needs to be SOME content in the backup. I would also suggest adding another set: (add the words around this; this is only the idea) CAMPUS-Other exclude “./a-z]*” # and by default, INCLUDE everything besides a-z This will catch anything that starts with a capital letter, or a number. Even if somebody adds them later, and you weren’t planning on them. But again, make sure there is initially something *in* the set. Perhaps a text file at the top level “00info” which describes the way you are splitting the backup “a-f”, …. Mostly just to have something there, but it might also be useful to know. Deb Baddorf Fermilab > On Nov 8, 2018, at 11:31 AM, Cuttler, Brian R (HEALTH) > wrote: > > > > Stupid question, host fileserver, directories /netdrives/CAMPUS/s* to > /netdrives/CAMPUS/z* exist and have some files in them? > > From: Chris Nighswonger > Sent: Thursday, November 8, 2018 12:11 PM > To: Cuttler, Brian R (HEALTH) > Cc: amanda-users@amanda.org > Subject: Re: Breaking DLEs up > > ATTENTION: This email came from an external source. Do not open attachments > or click on links from unknown senders or unexpected emails. > > From the client: > > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: start at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Version 3.3.1 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: rename at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed request as: > program `GNUTAR' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > disk `/netdrives/CAMPUS/sz' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > device `/netdrives/CAMPUS' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > level 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > since NODATE > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > options `' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > datapath `AMANDA' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: start: > host:/netdrives/CAMPUS/sz lev 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/bin/gzip > /bin/gzip --fast" in pipeline > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: pid 23694: > /bin/gzipThu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23694: > /bin/gzip --fast > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: doing level 0 dump as > listed-incremental to > '/var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Nothing found to include > for disk /netdrives/CAMPUS/sz > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning > "/usr/libexec/amanda/runtar runtar campus /bin/tar --create --file - > --directory /netdrives/CAMPUS --one-file-system --listed-incremental > /var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new --sparse > --ignore-failed-read --totals --files-from > /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include" in > pipeline > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: > /usr/libexec/amanda/runtar: pid 23696 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started backup > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started index creator: > "/bin/tar -tf - 2>/dev/null | sed -e 's/^\.//'" > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: 46:size(|): Total > bytes written: 10240 (10KiB, 78MiB/s) > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Index created > successfully > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed backup messages > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 finish time > Thu Nov 8 05:27:11 2018 > > From the server: > > /var/log/amanda/server/campus/dumper.20181108020002007.debug:165691:Thu Nov > 8 05:27:11 2018: thd-0x5579048e2400: dumper: getcmd: PORT-DUMP 03-00023 50013 > 1 host 9efefbff1f /netdrives/CAMPUS/sz /netdrives/CAMPUS 0 > 1970:1:1:0:0:0 GNUTAR "" "" "" "" bsdtcp AMANDA 127.0.0.1:50014 20 |" > bsdtcp\n FAST\n YES\n > YES\n AMANDA\n \n > ./[s-z]*\n \n" > /var/log/amanda/server/campus/dumper.20181108020002007.debug:165705: > /netdrives/CAMPUS/sz > /var/log/amanda/server/campus/dumper.20181108020002007.debug:165706:
Re: Breaking DLEs up
No question is stupid. I learned that beating my head against the wall for long hours. :-) /netdrives/CAMPUS/ is a path which contains users' network drives. The level below CAMPUS contains folders which follow the naming convention of the username of each account. ie. Chris Nighswonger would be cnighswonger and thus /netdrives/CAMPUS/cnighswonger/ would be my related network drive. There are somewhat less than 100 user directories. Prior to this I have been backing them all up with a DLE which looks like this: host "/netdrives/CAMPUS" { comp-tar estimate server } This works fine with the caveat that it results in a huge level 0 backup. In the supplied disklist file example in Amanda's documentation (/usr/share/doc/amanda-server/examples/disklist), I discovered the DLE form I am currently attempting. According to the example this should limit each DLE to backing up subdirectories of /netdrives/CAMPUS/ based on the regexp supplied in the "include" directive. It appears to me that something may have changed with the way Amanda handles this since that document was written. As Stefan points out, Amanda seems to thing that there is "nothing" to be backed up. Furthermore, it appears that the log excerpts I posted also show that the regexp is not being applied but that Amanda is actually looking for specific subdirectories like /netdrives/CAMPUS/af and the like. Maybe the DLE syntax is incorrect? On Thu, Nov 8, 2018 at 12:31 PM Cuttler, Brian R (HEALTH) < brian.cutt...@health.ny.gov> wrote: > > > > > Stupid question, host fileserver, directories /netdrives/CAMPUS/s* to > /netdrives/CAMPUS/z* exist and have some files in them? > > > > *From:* Chris Nighswonger > *Sent:* Thursday, November 8, 2018 12:11 PM > *To:* Cuttler, Brian R (HEALTH) > *Cc:* amanda-users@amanda.org > *Subject:* Re: Breaking DLEs up > > > > *ATTENTION: This email came from an external source. Do not open > attachments or click on links from unknown senders or unexpected emails.* > > From the client: > > > > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: start at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Version 3.3.1 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: rename at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed request as: > program `GNUTAR' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > disk `/netdrives/CAMPUS/sz' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > device `/netdrives/CAMPUS' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > level 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > since NODATE > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > options `' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > datapath `AMANDA' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: start: > host:/netdrives/CAMPUS/sz lev 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/bin/gzip > /bin/gzip --fast" in pipeline > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: pid 23694: > /bin/gzipThu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23694: > /bin/gzip --fast > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: doing level 0 dump as > listed-incremental to > '/var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Nothing found to > include for disk /netdrives/CAMPUS/sz > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning > "/usr/libexec/amanda/runtar runtar campus /bin/tar --create --file - > --directory /netdrives/CAMPUS --one-file-system --listed-incremental > /var/lib/amanda/gnutar-lists/host_netdrives_CAMPUS_sz_0.new --sparse > --ignore-failed-read --totals --files-from > /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include" in > pipeline > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: > /usr/libexec/amanda/runtar: pid 23696 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started backup > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Started index > creator: "/bin/tar -tf - 2>/dev/null | sed -e 's/^\.//'" > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: 46:size(|): > Total bytes written: 10240 (10KiB, 78MiB/s) > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Index created > successfully > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed backup messages > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 finish time > Thu Nov 8 05:27:11 2018 >
RE: Breaking DLEs up
Chris, We seem to be doing the same thing. We are doing it for the same reason. I went a little more finely grained as I had about 1200 user accounts and some stored quite a bit of data in their home directories, vs the group samba share or data directories for projects (usually on the compute engine or cluster, rather than the home directory server). [Latest storage server came in the door with 250 Tbytes of space] Note that I have ‘a’ to ‘z’ and then the catch-all that never catches anything, but that is expected and fits in with Debra’s comments. I always worry about the leading dot to anchor the path, but we did it the same. That is why I copied the two near each other, for easy comparison. You can see the tar commands in the log files, what do you catch if you run the tar command manually? Could it be a permissions problem for Amanda to read the user files? Your syntax fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } my syntax finsen /export/home-A /export/home { user-tar2 include "./[a]*" } finsen /export/home-AZ /export/home { user-tar2 include "./[A-Z]*" } I’ve removed all but the last email you wrote from this email, it was getting long for little or no gain. From: Chris Nighswonger Sent: Thursday, November 8, 2018 1:25 PM To: Cuttler, Brian R (HEALTH) Cc: amanda-users@amanda.org Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. No question is stupid. I learned that beating my head against the wall for long hours. :-) /netdrives/CAMPUS/ is a path which contains users' network drives. The level below CAMPUS contains folders which follow the naming convention of the username of each account. ie. Chris Nighswonger would be cnighswonger and thus /netdrives/CAMPUS/cnighswonger/ would be my related network drive. There are somewhat less than 100 user directories. Prior to this I have been backing them all up with a DLE which looks like this: host "/netdrives/CAMPUS" { comp-tar estimate server } This works fine with the caveat that it results in a huge level 0 backup. In the supplied disklist file example in Amanda's documentation (/usr/share/doc/amanda-server/examples/disklist), I discovered the DLE form I am currently attempting. According to the example this should limit each DLE to backing up subdirectories of /netdrives/CAMPUS/ based on the regexp supplied in the "include" directive. It appears to me that something may have changed with the way Amanda handles this since that document was written. As Stefan points out, Amanda seems to thing that there is "nothing" to be backed up. Furthermore, it appears that the log excerpts I posted also show that the regexp is not being applied but that Amanda is actually looking for specific subdirectories like /netdrives/CAMPUS/af and the like. Maybe the DLE syntax is incorrect?
Re: Breaking DLEs up
re: what works I use the following, and it works well. I’m still on 3.3.9 and on RedHat variants, fwiw. define dumptype alphaTara { dailyTarKRB5 include "./a*” } and IN my disk list, these variants to use the above defined dumptype: heimdall.fnal.gov homea /home { alphaTara estimate server } here’s one that is defined IN the disklist mtamaster1.fnal.gov 17-Mod-rawTOP /data/mta/Run2017/ModularCavity/rawdata { MTA-TarKRB5 exclude "./Agi80304B" exclude append "./Lec625Zi" exclude append "./Tek7104" # still includes Agi7104A and Tek7254 # since it must have SOME content } and one with RANGES, so it more close matches your case: heimdall.fnal.gov homeOther /home { dailyTarKRB5 exclude "./[a-pr-tv-z]*" exclude append "./ROOT-5.34*" estimate server } Just some examples of working set lists. Deb Baddorf Fermilab > On Nov 8, 2018, at 12:24 PM, Chris Nighswonger > wrote: > > No question is stupid. I learned that beating my head against the wall for > long hours. :-) > > /netdrives/CAMPUS/ is a path which contains users' network drives. The level > below CAMPUS contains folders which follow the naming convention of the > username of each account. ie. Chris Nighswonger would be cnighswonger and > thus /netdrives/CAMPUS/cnighswonger/ would be my related network drive. > > There are somewhat less than 100 user directories. Prior to this I have been > backing them all up with a DLE which looks like this: > > host "/netdrives/CAMPUS" { > comp-tar > estimate server > } > > This works fine with the caveat that it results in a huge level 0 backup. > > In the supplied disklist file example in Amanda's documentation > (/usr/share/doc/amanda-server/examples/disklist), I discovered the DLE form I > am currently attempting. According to the example this should limit each DLE > to backing up subdirectories of /netdrives/CAMPUS/ based on the regexp > supplied in the "include" directive. > > It appears to me that something may have changed with the way Amanda handles > this since that document was written. > > As Stefan points out, Amanda seems to thing that there is "nothing" to be > backed up. Furthermore, it appears that the log excerpts I posted also show > that the regexp is not being applied but that Amanda is actually looking for > specific subdirectories like /netdrives/CAMPUS/af and the like. > > Maybe the DLE syntax is incorrect? > > On Thu, Nov 8, 2018 at 12:31 PM Cuttler, Brian R (HEALTH) > wrote: > > > > > Stupid question, host fileserver, directories /netdrives/CAMPUS/s* to > /netdrives/CAMPUS/z* exist and have some files in them? > > > > From: Chris Nighswonger > Sent: Thursday, November 8, 2018 12:11 PM > To: Cuttler, Brian R (HEALTH) > Cc: amanda-users@amanda.org > Subject: Re: Breaking DLEs up > > > > ATTENTION: This email came from an external source. Do not open attachments > or click on links from unknown senders or unexpected emails. > > From the client: > > > > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: start at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Version 3.3.1 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23692 ruid 10195 > euid 10195 version 3.3.1: rename at Thu Nov 8 05:27:11 2018 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Parsed request as: > program `GNUTAR' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > disk `/netdrives/CAMPUS/sz' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > device `/netdrives/CAMPUS' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > level 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > since NODATE > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > options `' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: > datapath `AMANDA' > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: start: > host:/netdrives/CAMPUS/sz lev 0 > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: Spawning "/bin/gzip > /bin/gzip --fast" in pipeline > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: gnutar: pid 23694: > /bin/gzipThu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: pid 23694: > /bin/gzip --fast > Thu Nov 8 05:27:11 2018: thd-0x1c8f200: sendbackup: doing level 0 dump as > listed-incremental to > '/v
Re: Breaking DLEs up
On Thu, Nov 08, 2018 at 01:24:46PM -0500, Chris Nighswonger wrote: > No question is stupid. I learned that beating my head against the wall for > long hours. :-) > [snip] > > > > > > Here is the relevant portion of my DLEs: > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > > comp-tar > > include "./[a-f]*" > > estimate server > > } One difference I noticed. Deb, Brian, and myself all use "disknames" that are not pathnames. I've not heard of pathnanmes causing a problem, but you might consider changine /netdrives/CAMPUS/af to something like CAMPUSa-f. -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: Breaking DLEs up
On Thu, Nov 8, 2018 at 2:34 PM Jon LaBadie wrote: > On Thu, Nov 08, 2018 at 01:24:46PM -0500, Chris Nighswonger wrote: > > > Here is the relevant portion of my DLEs: > > > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > > > comp-tar > > > include "./[a-f]*" > > > estimate server > > > } > > One difference I noticed. Deb, Brian, and myself all use > "disknames" that are not pathnames. I've not heard of pathnanmes > causing a problem, but you might consider changine /netdrives/CAMPUS/af > to something like CAMPUSa-f. > > You may be on to something there. I can easily conceive of Amanda not discerning the difference given that the default behavior is to use the diskname as the diskdevice. [1] Kind regards, Chris [1] https://wiki.zmanda.com/man/disklist.5.html
Re: Breaking DLEs up
On Thu, 2018-11-08 at 13:24 -0500, Chris Nighswonger wrote: > No question is stupid. I learned that beating my head against the > wall for long hours. :-) > > /netdrives/CAMPUS/ is a path which contains users' network drives. > The level below CAMPUS contains folders which follow the naming > convention of the username of each account. ie. Chris Nighswonger > would be cnighswonger and thus /netdrives/CAMPUS/cnighswonger/ would > be my related network drive. > > There are somewhat less than 100 user directories. Prior to this I > have been backing them all up with a DLE which looks like this: > > host "/netdrives/CAMPUS" { > comp-tar > estimate server > } > > This works fine with the caveat that it results in a huge level 0 > backup. > > In the supplied disklist file example in Amanda's documentation > (/usr/share/doc/amanda-server/examples/disklist), I discovered the > DLE form I am currently attempting. According to the example this > should limit each DLE to backing up subdirectories of > /netdrives/CAMPUS/ based on the regexp supplied in the "include" > directive. > > It appears to me that something may have changed with the way Amanda > handles this since that document was written. > > As Stefan points out, Amanda seems to thing that there is "nothing" > to be backed up. Furthermore, it appears that the log excerpts I > posted also show that the regexp is not being applied but that Amanda > is actually looking for specific subdirectories like > /netdrives/CAMPUS/af and the like. > > Maybe the DLE syntax is incorrect? I looked at your original post and it looked like it should work, and the logs show it mostly did work, it just didn't see anything to back up. I'd suspect permission problems or a bug in your version of tar maybe. The only thing the logs don't show are the contents of /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include, which presumable has the regex in it. You should be able to recreate that and test the tar command manually. For comparison, I use the following for one host and it works fine: hyperion.example.com /home-A /home { gentoo-ssh-encrypt-user-comp include "./[aA0-9]*" include append "./.amandastub" } 1 hyperion.example.com /home-B /home { gentoo-ssh-encrypt-user-comp include "./[bB]*" include append "./.amandastub" } 1 hyperion.example.com /home-C /home { gentoo-ssh-encrypt-user-comp include "./[cC]*" include append "./.amandastub" } 1 ... etc. The 1's shouldn't matter, although you might want them to control parallel backups. And my appends are just so the DLE always finds something to backup even if I don't happen to have any directories matching those names. I'm using Amanda 3.5.1-r1, but those DLEs have been used for many previous versions too.
Re: Breaking DLEs up
On Thu, Nov 8, 2018 at 2:57 PM Alan Hodgson wrote: > > I looked at your original post and it looked like it should work, and > the logs show it mostly did work, it just didn't see anything to back > up. I'd suspect permission problems or a bug in your version of tar > maybe. The only thing the logs don't show are the contents of > /tmp/amanda/sendbackup._netdrives_CAMPUS_sz.20181108052711.include, > which presumable has the regex in it. You should be able to recreate > that and test the tar command manually. > > I finally looked in the right place and located the sendbackup...include files: sendbackup._netdrives_CAMPUS_af.20181108052816.include, etc. They are empty. sendbackup...exclude files related to a completely different DLE on the same server DO contain the correct exclude lists for that DLE. That doesn't sound right.
Re: Breaking DLEs up
On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) < brian.cutt...@health.ny.gov> wrote: > > > Your syntax > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > comp-tar > include "./[a-f]*" > estimate server > } > > > > my syntax > > > > finsen /export/home-A /export/home { > > user-tar2 > > include "./[a]*" > > } > > > > finsen /export/home-AZ /export/home { > > user-tar2 > > include "./[A-Z]*" > > } > > > Well, this fixes my problem, though why I do not know. fileserver CAMPUS_a-f /netdrives/CAMPUS { comp-tar exclude file "./[g-z]*" estimate server } 1 It seems a bit of work compared to the include directive. I tried "include file" to no avail. I'll see how the backup runs tonight, but amcheck likes it. Kind regards, Chris
RE: Breaking DLEs up
Interesting, not sure. For part 2, I will say that it is far easier to exclude files from backup than include them. You had done an excellent job of exclusion, you’ll pardon the poor attempt at humor, it is getting late in the day. From: Chris Nighswonger Sent: Thursday, November 8, 2018 3:21 PM To: Cuttler, Brian R (HEALTH) Cc: amanda-users@amanda.org Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) mailto:brian.cutt...@health.ny.gov>> wrote: Your syntax fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } my syntax finsen /export/home-A /export/home { user-tar2 include "./[a]*" } finsen /export/home-AZ /export/home { user-tar2 include "./[A-Z]*" } Well, this fixes my problem, though why I do not know. fileserver CAMPUS_a-f /netdrives/CAMPUS { comp-tar exclude file "./[g-z]*" estimate server } 1 It seems a bit of work compared to the include directive. I tried "include file" to no avail. I'll see how the backup runs tonight, but amcheck likes it. Kind regards, Chris
Re: Breaking DLEs up
Yeah, I do use includes, but I only do a single letter at a time include "./a*” Perhaps the problem is with the syntax of doing more than one letter. I only do [a-f] on my excludes. Weird! Deb Baddorf > On Nov 8, 2018, at 2:33 PM, Cuttler, Brian R (HEALTH) > wrote: > > > Interesting, not sure. > > For part 2, I will say that it is far easier to exclude files from backup > than include them. You had done an excellent job of exclusion, you’ll pardon > the poor attempt at humor, it is getting late in the day. > > > From: Chris Nighswonger > Sent: Thursday, November 8, 2018 3:21 PM > To: Cuttler, Brian R (HEALTH) > Cc: amanda-users@amanda.org > Subject: Re: Breaking DLEs up > > ATTENTION: This email came from an external source. Do not open attachments > or click on links from unknown senders or unexpected emails. > > On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) > wrote: > > Your syntax > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > comp-tar > include "./[a-f]*" > estimate server > } > > my syntax > > finsen /export/home-A /export/home { > user-tar2 > include "./[a]*" > } > > finsen /export/home-AZ /export/home { > user-tar2 > include "./[A-Z]*" > } > > > Well, this fixes my problem, though why I do not know. > > fileserver CAMPUS_a-f /netdrives/CAMPUS { > comp-tar > exclude file "./[g-z]*" > estimate server > } 1 > > It seems a bit of work compared to the include directive. I tried "include > file" to no avail. > > I'll see how the backup runs tonight, but amcheck likes it. > > Kind regards, > Chris
RE: Breaking DLEs up
My login directories are all lower case, but I did have one include "./[A-Z]*", it never found anything, which didn't bother me because there wasn't supposed to be anything to be found, now you make me wonder if it would have failed if I had in fact needed it. Good the situation never come up. -Original Message- From: Debra S Baddorf Sent: Thursday, November 8, 2018 3:44 PM To: Cuttler, Brian R (HEALTH) Cc: Debra S Baddorf ; Chris Nighswonger ; amanda-users@amanda.org Subject: Re: Breaking DLEs up ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. Yeah, I do use includes, but I only do a single letter at a time include "./a*” Perhaps the problem is with the syntax of doing more than one letter. I only do [a-f] on my excludes. Weird! Deb Baddorf > On Nov 8, 2018, at 2:33 PM, Cuttler, Brian R (HEALTH) > wrote: > > > Interesting, not sure. > > For part 2, I will say that it is far easier to exclude files from backup > than include them. You had done an excellent job of exclusion, you’ll pardon > the poor attempt at humor, it is getting late in the day. > > > From: Chris Nighswonger > Sent: Thursday, November 8, 2018 3:21 PM > To: Cuttler, Brian R (HEALTH) > Cc: amanda-users@amanda.org > Subject: Re: Breaking DLEs up > > ATTENTION: This email came from an external source. Do not open attachments > or click on links from unknown senders or unexpected emails. > > On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) > wrote: > > Your syntax > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > comp-tar > include "./[a-f]*" > estimate server > } > > my syntax > > finsen /export/home-A /export/home { > user-tar2 > include "./[a]*" > } > > finsen /export/home-AZ /export/home { > user-tar2 > include "./[A-Z]*" > } > > > Well, this fixes my problem, though why I do not know. > > fileserver CAMPUS_a-f /netdrives/CAMPUS { > comp-tar > exclude file "./[g-z]*" > estimate server > } 1 > > It seems a bit of work compared to the include directive. I tried "include > file" to no avail. > > I'll see how the backup runs tonight, but amcheck likes it. > > Kind regards, > Chris
Re: Breaking DLEs up
On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote: > Yeah, I do use includes, but I only do a single letter at a time >include "./a*” > > Perhaps the problem is with the syntax of doing more than one letter. > I only do [a-f] on my excludes. Weird! > > Deb Baddorf I have a working entry that matches the OP. include file "./201[7-9]*" Jon > > > On Nov 8, 2018, at 2:33 PM, Cuttler, Brian R (HEALTH) > > wrote: > > > > > > Interesting, not sure. > > > > For part 2, I will say that it is far easier to exclude files from backup > > than include them. You had done an excellent job of exclusion, you’ll > > pardon the poor attempt at humor, it is getting late in the day. > > > > > > From: Chris Nighswonger > > Sent: Thursday, November 8, 2018 3:21 PM > > To: Cuttler, Brian R (HEALTH) > > Cc: amanda-users@amanda.org > > Subject: Re: Breaking DLEs up > > > > ATTENTION: This email came from an external source. Do not open attachments > > or click on links from unknown senders or unexpected emails. > > > > On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) > > wrote: > > > > Your syntax > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > > comp-tar > > include "./[a-f]*" > > estimate server > > } > > > > my syntax > > > > finsen /export/home-A /export/home { > > user-tar2 > > include "./[a]*" > > } > > > > finsen /export/home-AZ /export/home { > > user-tar2 > > include "./[A-Z]*" > > } > > > > > > Well, this fixes my problem, though why I do not know. > > > > fileserver CAMPUS_a-f /netdrives/CAMPUS { > > comp-tar > > exclude file "./[g-z]*" > > estimate server > > } 1 > > > > It seems a bit of work compared to the include directive. I tried "include > > file" to no avail. > > > > I'll see how the backup runs tonight, but amcheck likes it. > > > > Kind regards, > > Chris >>> End of included message <<< -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: Breaking DLEs up
Ah, good! What does “file” do in your include line? Deb > On Nov 8, 2018, at 2:54 PM, Jon LaBadie wrote: > > On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote: >> Yeah, I do use includes, but I only do a single letter at a time >> include "./a*” >> >> Perhaps the problem is with the syntax of doing more than one letter. >> I only do [a-f] on my excludes. Weird! >> >> Deb Baddorf > > I have a working entry that matches the OP. > >include file "./201[7-9]*" > > Jon >> >>> On Nov 8, 2018, at 2:33 PM, Cuttler, Brian R (HEALTH) >>> wrote: >>> >>> >>> Interesting, not sure. >>> >>> For part 2, I will say that it is far easier to exclude files from backup >>> than include them. You had done an excellent job of exclusion, you’ll >>> pardon the poor attempt at humor, it is getting late in the day. >>> >>> >>> From: Chris Nighswonger >>> Sent: Thursday, November 8, 2018 3:21 PM >>> To: Cuttler, Brian R (HEALTH) >>> Cc: amanda-users@amanda.org >>> Subject: Re: Breaking DLEs up >>> >>> ATTENTION: This email came from an external source. Do not open attachments >>> or click on links from unknown senders or unexpected emails. >>> >>> On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) >>> wrote: >>> >>> Your syntax >>> >>> fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { >>> comp-tar >>> include "./[a-f]*" >>> estimate server >>> } >>> >>> my syntax >>> >>> finsen /export/home-A /export/home { >>>user-tar2 >>>include "./[a]*" >>>} >>> >>> finsen /export/home-AZ /export/home { >>>user-tar2 >>>include "./[A-Z]*" >>>} >>> >>> >>> Well, this fixes my problem, though why I do not know. >>> >>> fileserver CAMPUS_a-f /netdrives/CAMPUS { >>> comp-tar >>> exclude file "./[g-z]*" >>> estimate server >>> } 1 >>> >>> It seems a bit of work compared to the include directive. I tried "include >>> file" to no avail. >>> >>> I'll see how the backup runs tonight, but amcheck likes it. >>> >>> Kind regards, >>> Chris >>>> End of included message <<< > > -- > Jon H. LaBadie j...@jgcomp.com > 11226 South Shore Rd. (703) 787-0688 (H) > Reston, VA 20190 (703) 935-6720 (C)
Re: Breaking DLEs up
Am 08.11.18 um 22:06 schrieb Debra S Baddorf: Ah, good! What does “file” do in your include line? exactly include file ... means "read include regex(es) from that file you want include "./[a-g]*" and not "include file" see manpage
Re: Breaking DLEs up
Am 08.11.18 um 22:49 schrieb Stefan G. Weichinger: Am 08.11.18 um 22:06 schrieb Debra S Baddorf: Ah, good! What does “file” do in your include line? exactly include file ... means "read include regex(es) from that file you want include "./[a-g]*" and not "include file" see manpage I might have been to quick to reply, sorry. manpage says: exclude [ list | file ][[optional][append][ string ]+] Default: file. Exclude is the opposite of include and specifies files that will be excluded from the backup. The format of the exclude expressions depends on the application, and some applications do not support excluding files at all. ... although I always skip that default "file" option (and recommend to test that)
Re: Breaking DLEs up
Am 08.11.18 um 22:52 schrieb Stefan G. Weichinger: manpage says: exclude [ list | file ][[optional][append][ string ]+] Default: file. Exclude is the opposite of include and specifies files that will be excluded from the backup. The format of the exclude expressions depends on the application, and some applications do not support excluding files at all. ... although I always skip that default "file" option (and recommend to test that) another thought (maybe you mentioned it, it's late here and I haven't scanned the whole thread): do you use GNUTAR as dump program (parameter "program" in global dumptype or so)? If you didn't specify the default is DUMP and not GNUTAR (and the include parameter might do completely other things)
Re: Breaking DLEs up
Am 08.11.18 um 22:55 schrieb Stefan G. Weichinger: Am 08.11.18 um 22:52 schrieb Stefan G. Weichinger: manpage says: exclude [ list | file ][[optional][append][ string ]+] Default: file. Exclude is the opposite of include and specifies files that will be excluded from the backup. The format of the exclude expressions depends on the application, and some applications do not support excluding files at all. ... although I always skip that default "file" option (and recommend to test that) another thought (maybe you mentioned it, it's late here and I haven't scanned the whole thread): do you use GNUTAR as dump program (parameter "program" in global dumptype or so)? If you didn't specify the default is DUMP and not GNUTAR (and the include parameter might do completely other things) Some more examples for you ;-) these work for years afaik: garten projekte_1 /home/Samba/Daten/Projekte { comp-cl-highprio include "./[a-gA-G]*" } garten projekte_2 /home/Samba/Daten/Projekte { comp-cl-highprio include "./[h-nH-N]*" } garten projekte_3 /home/Samba/Daten/Projekte { comp-cl-highprio include "./[o-pO-P]*" }
Re: Breaking DLEs up
On Thu, Nov 08, 2018 at 15:21:00 -0500, Chris Nighswonger wrote: > On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) < > brian.cutt...@health.ny.gov> wrote: > > > > > > > Your syntax > > > > > > > > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { > > comp-tar > > include "./[a-f]*" > > estimate server > > } > > [...] > > Well, this fixes my problem, though why I do not know. > > fileserver CAMPUS_a-f /netdrives/CAMPUS { > comp-tar > exclude file "./[g-z]*" > estimate server > } 1 > > It seems a bit of work compared to the include directive. I tried "include > file" to no avail. I haven't quite followed this whole thread, either, but have you taken a look at http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists ? It may help explain some of the nuances in how things work. As Stefan mentioned, it would be helpeful to know what amanda version you are using, and which dump program. If your client supports it, using APPLICATION amgtar (rather than GNUTAR) is definitely a good idea. But, especially if you are using GNUTAR, an important thing to keep in mind is that the exclude list is processed by passing the whole list using an --exclude option to GNU tar, which then processes that list as it does the dump -- while in contrast the "include" options are processed ahead of time by Amanda to build a list of directories to pass on the command line (thus making up tar's list of "what should I be backing up"?). The tricky thing is that the tar process runs with root priviledge, so it can see any directory out there... but the "include" work is done by an unprivileged process and so the directory *above* the include pattern must be readable by the amanda user... So, what are the permissions on the /netdrives/CAMPUS/ directory itself? Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Breaking DLEs up
On Thu, Nov 8, 2018 at 5:35 PM Nathan Stratton Treadway wrote: > > > I haven't quite followed this whole thread, either, but have you taken a > look at > http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists > ? It may help explain some of the nuances in how things work. > That information is very similar to the information in the example disklist file included with Amanda. > As Stefan mentioned, it would be helpeful to know what amanda version > you are using, and which dump program. If your client supports it, > using APPLICATION amgtar (rather than GNUTAR) is definitely a good idea. This client is running 3.3.1. The server is running 3.3.6. Dump program is GNUTAR. I can switch to amgtar. > > But, especially if you are using GNUTAR, an important thing to keep in > mind is that the exclude list is processed by passing the whole list > using an --exclude option to GNU tar, which then processes that list as > it does the dump -- while in contrast the "include" options are > processed ahead of time by Amanda to build a list of directories to pass > on the command line (thus making up tar's list of "what should I be > backing up"?). The tricky thing is that the tar process runs with root > priviledge, so it can see any directory out there... but the "include" > work is done by an unprivileged process and so the directory *above* the > include pattern must be readable by the amanda user... > That explains a lot. > So, what are the permissions on the /netdrives/CAMPUS/ directory itself? root@fileserver:~# stat -c "%a %n" /netdrives 755 /netdrives root@fileserver:~# stat -c "%a %n" /netdrives/CAMPUS 2771 /netdrives/CAMPUS Which explains the problem if I understand correctly. 771 would prohibit the amanda user from reading everything below CAMPUS. Chris
Re: Breaking DLEs up
On Thu, Nov 08, 2018 at 21:24:04 -0500, Chris Nighswonger wrote: > Dump program is GNUTAR. I can switch to amgtar. > Amgtar is more flexible to use (and easier to maintain, should development on Amanda ever resume), so while you're rolling this out it's probably worth switching. (And it should let you work around this permission problem, too.) > root@fileserver:~# stat -c "%a %n" /netdrives > 755 /netdrives > root@fileserver:~# stat -c "%a %n" /netdrives/CAMPUS > 2771 /netdrives/CAMPUS > > Which explains the problem if I understand correctly. 771 would > prohibit the amanda user from reading everything below CAMPUS. Right (assuming that the Amanda user is not in the group which owns the CAMPUS directory). Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Breaking DLEs up
On Thu, Nov 08, 2018 at 09:06:47PM +, Debra S Baddorf wrote: > Ah, good! What does “file” do in your include line? > Deb > Include (and exclude) can take a first argument of "file" or "list". With "file" the following string is a "glob" expression. "file" is the default but I like to specify it anyway. With "include list" the string that follows is the name of a file containing "globs", one per line. You can have multiple "include file" globs if all but the first are "include file append". So include file "./[a-gA-G]* could also be specified as include file "./[a-g]* include file append "./[A-G]* Jon > > > > On Nov 8, 2018, at 2:54 PM, Jon LaBadie wrote: > > > > On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote: > >> Yeah, I do use includes, but I only do a single letter at a time > >> include "./a*” > >> > >> Perhaps the problem is with the syntax of doing more than one letter. > >> I only do [a-f] on my excludes. Weird! > >> > >> Deb Baddorf > > > > I have a working entry that matches the OP. > > > >include file "./201[7-9]*" > > > > Jon > >> -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: Breaking DLEs up
Jon LaBadie writes: > On Thu, Nov 08, 2018 at 09:06:47PM +, Debra S Baddorf wrote: >> Ah, good! What does “file” do in your include line? >> Deb >> > Include (and exclude) can take a first argument of "file" or "list". > With "file" the following string is a "glob" expression. > "file" is the default but I like to specify it anyway. > > With "include list" the string that follows is the name of a file > containing "globs", one per line. > > You can have multiple "include file" globs if all but the first > are "include file append". So > > include file "./[a-gA-G]* > > could also be specified as > > include file "./[a-g]* > include file append "./[A-G]* > > Jon A long, long time ago, I wrote a script to test the exclde files. I believe it could be adapted to test the include files too. I noticed the script was not online anymore, so I put it back: http://www.cs.ait.ac.th/~on/testgtar Best regards, Olivier >> >> >> > On Nov 8, 2018, at 2:54 PM, Jon LaBadie wrote: >> > >> > On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote: >> >> Yeah, I do use includes, but I only do a single letter at a time >> >> include "./a*” >> >> >> >> Perhaps the problem is with the syntax of doing more than one letter. >> >> I only do [a-f] on my excludes. Weird! >> >> >> >> Deb Baddorf >> > >> > I have a working entry that matches the OP. >> > >> >include file "./201[7-9]*" >> > >> > Jon >> >> --
Re: Breaking DLEs up
On Thu, Nov 8, 2018 at 10:57 PM Nathan Stratton Treadway wrote: > On Thu, Nov 08, 2018 at 21:24:04 -0500, Chris Nighswonger wrote: > > Dump program is GNUTAR. I can switch to amgtar. > > > > Amgtar is more flexible to use (and easier to maintain, should > development on Amanda ever resume), so while you're rolling this out > it's probably worth switching. (And it should let you work around this > permission problem, too.) > This makes the most sense as it will hopefully require less maintenance. Thanks to everyone for the help. It has been quite educational in many ways. Kind regards, Chris
Re: Breaking DLEs up
Too many comments on this thread to read them all (too many other things that need attention). However, I just wanted to note that I had been having trouble with include and exclude to break up DLEs some time ago. When I posted asking for help, JLM replied back and told me that I really needed to use Application amgtar for my backups. If I were just using the program gnutar, then to implement my general expressions, the amanda user needed read access all the way down to the level in the subdirectories where the expressions were being applied. Otherwise, it would not be able to do it, and would come up with an empty list. I switched all my backups to Application amgtar, and have had no problems since. I should note that some of my LVMs are broken up into dozens of DLEs, even though I'm using LTO7. Back when I was using AIT5, I tried to keep DLEs less than 100GB. Then with LTO6, I loosened it up to 300GB, with some larger. Now, with LTO7, I let them approach 1T sometimes, but use hardware compression so that I'm not eating all kinds of CPU with gzip processes. This has worked really well. Researcher's data is often restricted access, and may be divided up into subdirectories by lab personnel. Many of those take up many TeraBytes. So, I end up having to go down into their subdirectories sorting out how to divide them into DLEs. I have about 100TB of storage on each of two different Departments' servers. We're currently building arrays with 10TB HGST Helium filled drives, Supermicro servers with Ubuntu 14.04 and 16.04, mdadm and LVM. I've found that I can build a RAID5 with 2 drives and then grow it by adding drives as needed. When I reach 5 drives, I request that the next addition be 2 drives so that I can convert it to RAID6 and grow it by 1 drive. All of that can be done live, although it can get scary sometimes. My approach to DLEs is as follows. The name after the "/./" in the first line of each DLE is an arbitrary name for the DLE. localhost /data/professorA/./catchall /data/professorA { gnutar-lto7-local exclude append "./directory1" exclude append "./directory2" # etc. } -1 localhost /data/professorA/./directory1-catchall /data/professorA { gnutar-lto7-local include "./directory1" exclude append "./directory1/EMR/CDH[1-9]*" # etc. } -1 localhost /data/professorA/./directory1a /data/professorA { gnutar-lto7-local include "./directory1/EMR/CDH[1-4]*" } -1 localhost /data/professorA/./directory1b /data/professorA { gnutar-lto7-local include "./directory1/EMR/CDH[5-9]*" } -1 localhost /data/professorA/./directory2-catchall /data/professorA { # and so on, in a hierarchical top down structure. The above example was constructed for this email message (so as to anonymize) based on real examples in my disklist. As I said earlier, some of my actual examples end up breaking up an LVM (which would correspond to, e.g., /data/professorA) into dozens of DLEs. On 11/8/18 5:30 PM, Nathan Stratton Treadway wrote: On Thu, Nov 08, 2018 at 15:21:00 -0500, Chris Nighswonger wrote: On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) < brian.cutt...@health.ny.gov> wrote: Your syntax fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" { comp-tar include "./[a-f]*" estimate server } [...] Well, this fixes my problem, though why I do not know. fileserver CAMPUS_a-f /netdrives/CAMPUS { comp-tar exclude file "./[g-z]*" estimate server } 1 It seems a bit of work compared to the include directive. I tried "include file" to no avail. I haven't quite followed this whole thread, either, but have you taken a look at http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists ? It may help explain some of the nuances in how things work. As Stefan mentioned, it would be helpeful to know what amanda version you are using, and which dump program. If your client supports it, using APPLICATION amgtar (rather than GNUTAR) is definitely a good idea. But, especially if you are using GNUTAR, an important thing to keep in mind is that the exclude list is processed by passing the whole list using an --exclude option to GNU tar, which then processes that list as it does the dump -- while in contrast the "include" options are processed ahead of time by Amanda to build a list of directories to pass on the command line (thus making up tar's list of "what should I be backing up"?). The tricky thing is that the tar process runs with root priviledge, so it can see any directory out there... but the "include" work is done by an unprivileged process and so the directory *above* the include pattern must be readable by the amanda user... So, what are the permissions on the /netd