Dealing with a dump too big
I'm trying to deal with a problem which I've just noticed. I've completely overwritten
my level 0 backup of a disk called admin://db/f$. This is a SAMBA share from an NT
server. I think I have complete level one backups:
[EMAIL PROTECTED]:~ > amadmin DailySet1 find admin //db/ |fgrep //db/f$
2004-05-20 admin //db/f$ 1 DailySet109 8 OK
2004-05-21 admin //db/f$ 1 DailySet11113 OK
2004-05-24 admin //db/f$ 1 DailySet112 7 OK
2004-05-25 admin //db/f$ 1 DailySet114 8 OK
2004-05-26 admin //db/f$ 1 DailySet11510 OK
2004-05-27 admin //db/f$ 1 DailySet11610 OK
2004-05-28 admin //db/f$ 1 DailySet117 8 OK
2004-05-31 admin //db/f$ 1 DailySet118 7 OK
2004-06-01 admin //db/f$ 1 DailySet119 8 OK
2004-06-02 admin //db/f$ 1 DailySet121 8 OK
2004-06-04 admin //db/f$ 1 DailySet123 9 OK
2004-06-07 admin //db/f$ 1 DailySet12411 OK
2004-06-08 admin //db/f$ 1 DailySet125 9 OK
2004-06-09 admin //db/f$ 1 DailySet126 8 OK
2004-06-10 admin //db/f$ 1 DailySet12712 OK
2004-06-11 admin //db/f$ 1 DailySet12810 OK
2004-06-14 admin //db/f$ 1 DailySet101 9 OK
2004-06-15 admin //db/f$ 1 DailySet10212 OK
2004-06-16 admin //db/f$ 1 DailySet10410 OK
2004-06-17 admin //db/f$ 1 DailySet10510 OK
2004-06-18 admin //db/f$ 1 --- 0 FAILED (planner) [dumps way too
big, 5679359 KB, must skip incremental dumps]
2004-06-18 admin //db/f$/inetsrv 0 DailySet10613 OK
[EMAIL PROTECTED]:~ >
The last daily report I got, in the planner section, said:
planner: admin //db/f$ 20040618 0 [dump larger than tape, 13855177 KB, full dump
delayed]
The disk F: on the NT server is indeed 13G in size, but I didn't think that would be a
problem, since I excluded //db/f$/inetsrv, which is 9.1G. This filesystem, which I
just backup for the first time last run, backed up at level 0 just fine:
[EMAIL PROTECTED]:~ > amadmin DailySet1 info admin //db/
Current info for admin //db/f$:
Stats: dump rates (kps), Full: 4221.0, 4333.0, 4412.0
Incremental: 4607.0, 4309.0, 4723.0
compressed size, Full: -100.0%,-100.0%,-100.0%
Incremental: -100.0%,-100.0%,-100.0%
Dumps: lev datestmp tape file origK compK secs
0 20040514 DailySet10515 4621248 4622030 1095
1 20040617 DailySet10510 5662420 5662530 1229
Current info for admin //db/f$/inetsrv:
Stats: dump rates (kps), Full: 3642.0, -1.0, -1.0
Incremental: -1.0, -1.0, -1.0
compressed size, Full: -100.0%,-100.0%,-100.0%
Incremental: -100.0%,-100.0%,-100.0%
Dumps: lev datestmp tape file origK compK secs
0 20040618 DailySet10613 9174403 9193950 2524
[EMAIL PROTECTED]:~ >
The pertinent sections of my disklist and amanda.conf files are:
[EMAIL PROTECTED]:~ > grep admin /etc/amanda/DailySet1/disklist
admin //db/f$ db-f-nocomp-medpri-tar-exclude-inetsrv #DB server, Drive F: excluding
\inetsrv
admin //db/f$/inetsrv nocomp-medpri-tar #DB server, Drive F:\inetsrv\
[EMAIL PROTECTED]:~ >
>From amanda.conf:
# Special dumptypes for excluding directories
define dumptype db-f-nocomp-medpri-tar-exclude-inetsrv{
nocomp-medpri-tar
comment "Special for admin//db/f$, excluding /inetsrv/"
exclude "./inetsrv/"
}
My questions are:
1. What's the long term solution to this problem? Have I done something wrong in the
amanda.conf or disklist files?
2. Is there anything I can do right now, before the nightly normal run, to get a level
0 backup of just this share?
Thanks for all your help and suggestions.
-Kevin Zembower
-
E. Kevin Zembower
Unix Administrator
Johns Hopkins University/Center for Communications Programs
111 Market Place, Suite 310
Baltimore, MD 21202
410-659-6139
RE: Dealing with a dump too big
Title: RE: Dealing with a dump too big
In the UK, we woudl flush a dump down the toilet, but not if it's too big ;-)
-Original Message-
From: [EMAIL PROTECTED] on behalf of KEVIN ZEMBOWER
Sent: Mon 6/21/2004 4:46 PM
To: [EMAIL PROTECTED]
Cc:
Subject: Dealing with a dump too big
I'm trying to deal with a problem which I've just noticed. I've completely overwritten my level 0 backup of a disk called admin://db/f$. This is a SAMBA share from an NT server. I think I have complete level one backups:
[EMAIL PROTECTED]:~ > amadmin DailySet1 find admin //db/ |fgrep //db/f$
2004-05-20 admin //db/f$ 1 DailySet109 8 OK
2004-05-21 admin //db/f$ 1 DailySet111 13 OK
2004-05-24 admin //db/f$ 1 DailySet112 7 OK
2004-05-25 admin //db/f$ 1 DailySet114 8 OK
2004-05-26 admin //db/f$ 1 DailySet115 10 OK
2004-05-27 admin //db/f$ 1 DailySet116 10 OK
2004-05-28 admin //db/f$ 1 DailySet117 8 OK
2004-05-31 admin //db/f$ 1 DailySet118 7 OK
2004-06-01 admin //db/f$ 1 DailySet119 8 OK
2004-06-02 admin //db/f$ 1 DailySet121 8 OK
2004-06-04 admin //db/f$ 1 DailySet123 9 OK
2004-06-07 admin //db/f$ 1 DailySet124 11 OK
2004-06-08 admin //db/f$ 1 DailySet125 9 OK
2004-06-09 admin //db/f$ 1 DailySet126 8 OK
2004-06-10 admin //db/f$ 1 DailySet127 12 OK
2004-06-11 admin //db/f$ 1 DailySet128 10 OK
2004-06-14 admin //db/f$ 1 DailySet101 9 OK
2004-06-15 admin //db/f$ 1 DailySet102 12 OK
2004-06-16 admin //db/f$ 1 DailySet104 10 OK
2004-06-17 admin //db/f$ 1 DailySet105 10 OK
2004-06-18 admin //db/f$ 1 --- 0 FAILED (planner) [dumps way too big, 5679359 KB, must skip incremental dumps]
2004-06-18 admin //db/f$/inetsrv 0 DailySet106 13 OK
[EMAIL PROTECTED]:~ >
The last daily report I got, in the planner section, said:
planner: admin //db/f$ 20040618 0 [dump larger than tape, 13855177 KB, full dump delayed]
The disk F: on the NT server is indeed 13G in size, but I didn't think that would be a problem, since I excluded //db/f$/inetsrv, which is 9.1G. This filesystem, which I just backup for the first time last run, backed up at level 0 just fine:
[EMAIL PROTECTED]:~ > amadmin DailySet1 info admin //db/
Current info for admin //db/f$:
Stats: dump rates (kps), Full: 4221.0, 4333.0, 4412.0
Incremental: 4607.0, 4309.0, 4723.0
compressed size, Full: -100.0%,-100.0%,-100.0%
Incremental: -100.0%,-100.0%,-100.0%
Dumps: lev datestmp tape file origK compK secs
0 20040514 DailySet105 15 4621248 4622030 1095
1 20040617 DailySet105 10 5662420 5662530 1229
Current info for admin //db/f$/inetsrv:
Stats: dump rates (kps), Full: 3642.0, -1.0, -1.0
Incremental: -1.0, -1.0, -1.0
compressed size, Full: -100.0%,-100.0%,-100.0%
Incremental: -100.0%,-100.0%,-100.0%
Dumps: lev datestmp tape file origK compK secs
0 20040618 DailySet106 13 9174403 9193950 2524
[EMAIL PROTECTED]:~ >
The pertinent sections of my disklist and amanda.conf files are:
[EMAIL PROTECTED]:~ > grep admin /etc/amanda/DailySet1/disklist
admin //db/f$ db-f-nocomp-medpri-tar-exclude-inetsrv #DB server, Drive F: excluding \inetsrv
admin //db/f$/inetsrv nocomp-medpri-tar #DB server, Drive F:\inetsrv\
[EMAIL PROTECTED]:~ >
>From amanda.conf:
# Special dumptypes for excluding directories
define dumptype db-f-nocomp-medpri-tar-exclude-inetsrv{
nocomp-medpri-tar
comment "Special for admin//db/f$, excluding /inetsrv/"
exclude "./inetsrv/"
}
My questions are:
1. What's the long term solution to this problem? Have I done something wrong in the amanda.conf or disklist files?
2. Is there anything I can do right now, before the nightly normal run, to get a level 0 backup of just this share?
Thanks for all your help and suggestions.
-Kevin Zembower
-
E. Kevin Zembower
Unix Administrator
Johns Hopkins University/Center for Communications Programs
111 Market Place, Suite 310
Baltimore, MD 21202
410-659-6139
Re: Dealing with a dump too big
Hi, Kevin,
on Montag, 21. Juni 2004 at 17:46 you wrote to amanda-users:
KZ> The disk F: on the NT server is indeed 13G in size, but I
KZ> didn't think that would be a problem, since I excluded
KZ> //db/f$/inetsrv, which is 9.1G. This filesystem, which I just
KZ> backup for the first time last run, backed up at level 0 just fine:
KZ> The pertinent sections of my disklist and amanda.conf files are:
[EMAIL PROTECTED]:~ >> grep admin /etc/amanda/DailySet1/disklist
KZ> admin //db/f$ db-f-nocomp-medpri-tar-exclude-inetsrv #DB
KZ> server, Drive F: excluding \inetsrv
KZ> admin //db/f$/inetsrv nocomp-medpri-tar #DB server, Drive
F:\inetsrv\
[EMAIL PROTECTED]:~ >>
KZ> From amanda.conf:
KZ> # Special dumptypes for excluding directories
KZ> define dumptype db-f-nocomp-medpri-tar-exclude-inetsrv{
KZ>nocomp-medpri-tar
KZ>comment "Special for admin//db/f$, excluding /inetsrv/"
KZ>exclude "./inetsrv/"
KZ> }
KZ> My questions are:
KZ> 1. What's the long term solution to this problem? Have I done
KZ> something wrong in the amanda.conf or disklist files?
The usage of excludes is a bit different when backing up smb-shares.
Please give me the output of
amadmin DailySet1 disklist admin //db/f$
which should tell us more about how AMANDA interprets your cascading
of exclusions ...
Execute this before and after you edited the following =>
You can only use ONE exclusion-option with smbclient ...
AFAIK this should be:
exclude ".\inetsrv\*"
in this case (Win uses backslashes ...)
KZ> 2. Is there anything I can do right now, before the nightly
KZ> normal run, to get a level 0 backup of just this share?
You can do this (after editing your exclusion):
amadmin DailySet1 force admin //db/f$
amdump DailySet1 admin //db/f$
--
best regards,
Stefan
Stefan G. Weichinger
mailto:[EMAIL PROTECTED]
Re: Dealing with a dump too big
KEVIN ZEMBOWER wrote: The disk F: on the NT server is indeed 13G in size, but I didn't think that would be a problem, since I excluded //db/f$/inetsrv, which is 9.1G. This filesystem, which I just backup for the first time last run, backed up at level 0 just fine: There is a bug in the estimate for samba clients. It used to be that amanda started smbclient -T and just flushed the output in /dev/null for the estimate phase. That was very very slow. Then the implementation was changed to use the smbclient builtin 'du' command to do the estimate. That's very fast now. The bug you're hitting is that that builtin 'du' command does not know about excludes. The estimate for the full dump does include everything. The real run not :-) My questions are: 1. What's the long term solution to this problem? Have I done something wrong in the amanda.conf or disklist files? Jean-Louis is working on a generalized "quick-and-dirty" estimate, that is based on the statistics of the previous runs. That could would help in your case. I have no idea about the current status. (You asked for 'long term' :-) ) 2. Is there anything I can do right now, before the nightly normal run, to get a level 0 backup of just this share? As dirty trick would be to change your tapetype and fake a larger tape size. As long as you don't overflow your real tape, that would at least get around the problem "dump larger than tapesize" for this particular backup image. (But lying to Amanda is in general not such a good idea. Keep your eyes open for other unexpected results!) -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
Re: Dealing with a dump too big
Stefan, hi, thanks for your suggestions.
Here's the output before any changes are made:
[EMAIL PROTECTED]:~ > amadmin DailySet1 disklist admin //db/
line 100:
host admin:
interface default
disk //db/f$:
program "GNUTAR"
exclude file "./inetsrv/"
priority 1
dumpcycle 3
maxdumps 1
maxpromoteday 1
strategy STANDARD
compress NONE
auth BSD
kencrypt NO
holdingdisk YES
record YES
index NO
skip-incr NO
skip-full NO
line 101:
host admin:
interface default
disk //db/f$/inetsrv:
program "GNUTAR"
priority 1
dumpcycle 3
maxdumps 1
maxpromoteday 1
strategy STANDARD
compress NONE
auth BSD
kencrypt NO
holdingdisk YES
record YES
index NO
skip-incr NO
skip-full NO
[EMAIL PROTECTED]:~ >
[Why can't I enter 'amadmin DailySet1 disklist admin //db/f$'? I get "amadmin: no disk
matched", unless I trim it back to "amadmin DailySet1 disklist admin //db/".]
As Paul points out, the problem is a flaw in the way estimated sizes are computed: it
ignores the excluded files and directories.
Won't your suggestion to:
amadmin DailySet1 force admin //db/f$
amdump DailySet1 admin //db/f$
just force admin //db/f$ to do a level 0 dump, but all the rest of the backup targets
do whatever they were scheduled to do? This normally takes 4-6 hours on my system,
and wouldn't complete before the normally scheduled backup. I am doing 'amadmin
DailySet1 force admin //db/f$' for tonight's backup.
Thanks, again, for your suggestions.
-Kevin
>>> [EMAIL PROTECTED] 06/21/04 12:43PM >>>
Hi, Kevin,
on Montag, 21. Juni 2004 at 17:46 you wrote to amanda-users:
KZ> The disk F: on the NT server is indeed 13G in size, but I
KZ> didn't think that would be a problem, since I excluded
KZ> //db/f$/inetsrv, which is 9.1G. This filesystem, which I just
KZ> backup for the first time last run, backed up at level 0 just fine:
KZ> The pertinent sections of my disklist and amanda.conf files are:
[EMAIL PROTECTED]:~ >> grep admin /etc/amanda/DailySet1/disklist
KZ> admin //db/f$ db-f-nocomp-medpri-tar-exclude-inetsrv #DB
KZ> server, Drive F: excluding \inetsrv
KZ> admin //db/f$/inetsrv nocomp-medpri-tar #DB server, Drive
F:\inetsrv\
[EMAIL PROTECTED]:~ >>
KZ> From amanda.conf:
KZ> # Special dumptypes for excluding directories
KZ> define dumptype db-f-nocomp-medpri-tar-exclude-inetsrv{
KZ>nocomp-medpri-tar
KZ>comment "Special for admin//db/f$, excluding /inetsrv/"
KZ>exclude "./inetsrv/"
KZ> }
KZ> My questions are:
KZ> 1. What's the long term solution to this problem? Have I done
KZ> something wrong in the amanda.conf or disklist files?
The usage of excludes is a bit different when backing up smb-shares.
Please give me the output of
amadmin DailySet1 disklist admin //db/f$
which should tell us more about how AMANDA interprets your cascading
of exclusions ...
Execute this before and after you edited the following =>
You can only use ONE exclusion-option with smbclient ...
AFAIK this should be:
exclude ".\inetsrv\*"
in this case (Win uses backslashes ...)
KZ> 2. Is there anything I can do right now, before the nightly
KZ> normal run, to get a level 0 backup of just this share?
You can do this (after editing your exclusion):
amadmin DailySet1 force admin //db/f$
amdump DailySet1 admin //db/f$
--
best regards,
Stefan
Stefan G. Weichinger
mailto:[EMAIL PROTECTED]
Re: Dealing with a dump too big
Hi, Kevin, on Montag, 21. Juni 2004 at 19:19 you wrote to amanda-users: KZ> Stefan, hi, thanks for your suggestions. KZ> Here's the output before any changes are made: [EMAIL PROTECTED]:~ >> amadmin DailySet1 disklist admin //db/ KZ> KZ> line 100: KZ> host admin: KZ> interface default KZ> disk //db/f$: KZ> program "GNUTAR" KZ> exclude file "./inetsrv/" And now you modified it to the backslashes? KZ> [Why can't I enter 'amadmin DailySet1 disklist admin KZ> //db/f$'? I get "amadmin: no disk matched", unless I trim it back KZ> to "amadmin DailySet1 disklist admin //db/".] The character $ seems to be interpreted as the regex-metacharacter. KZ> As Paul points out, the problem is a flaw in the way KZ> estimated sizes are computed: it ignores the excluded files and KZ> directories. This way the estimates should be bigger than the actual dumps. KZ> Won't your suggestion to: KZ> amadmin DailySet1 force admin //db/f$ KZ> amdump DailySet1 admin //db/f$ KZ> just force admin //db/f$ to do a level 0 dump, but all the KZ> rest of the backup targets do whatever they were scheduled to do? KZ> This normally takes 4-6 hours on my system, and wouldn't complete KZ> before the normally scheduled backup. I am doing 'amadmin KZ> DailySet1 force admin //db/f$' for tonight's backup. the second line would just dump the DLE given, //db/f$. SYNOPSIS amdump config [ host [ disk ]* ]* Stefan.
Re: Dealing with a dump too big
Hi, Stefan, Thanks for pointing out the option on amdump to just backup a single host or partition; I never used that, and overlooked what you were trying to tell me in my original response. I've been working since yesterday to implement your suggestions. I'll modify the slashes to back-slashes today, when I'm able to make an individual run for just the admin //db/f$ share. WRT the '$' metacharacter, what I actually had to run was: amdump DailySet1 admin '//db/f\$' Thanks, again, for your help and suggestions. -Kevin >>> [EMAIL PROTECTED] 06/21/04 01:37PM >>> Hi, Kevin, on Montag, 21. Juni 2004 at 19:19 you wrote to amanda-users: KZ> Stefan, hi, thanks for your suggestions. KZ> Here's the output before any changes are made: [EMAIL PROTECTED]:~ >> amadmin DailySet1 disklist admin //db/ KZ> KZ> line 100: KZ> host admin: KZ> interface default KZ> disk //db/f$: KZ> program "GNUTAR" KZ> exclude file "./inetsrv/" And now you modified it to the backslashes? KZ> [Why can't I enter 'amadmin DailySet1 disklist admin KZ> //db/f$'? I get "amadmin: no disk matched", unless I trim it back KZ> to "amadmin DailySet1 disklist admin //db/".] The character $ seems to be interpreted as the regex-metacharacter. KZ> As Paul points out, the problem is a flaw in the way KZ> estimated sizes are computed: it ignores the excluded files and KZ> directories. This way the estimates should be bigger than the actual dumps. KZ> Won't your suggestion to: KZ> amadmin DailySet1 force admin //db/f$ KZ> amdump DailySet1 admin //db/f$ KZ> just force admin //db/f$ to do a level 0 dump, but all the KZ> rest of the backup targets do whatever they were scheduled to do? KZ> This normally takes 4-6 hours on my system, and wouldn't complete KZ> before the normally scheduled backup. I am doing 'amadmin KZ> DailySet1 force admin //db/f$' for tonight's backup. the second line would just dump the DLE given, //db/f$. SYNOPSIS amdump config [ host [ disk ]* ]* Stefan.
Re: Dealing with a dump too big
Hi, Kevin, on Dienstag, 22. Juni 2004 at 18:17 you wrote to amanda-users: KZ> Hi, Stefan, KZ> Thanks for pointing out the option on amdump to just backup a KZ> single host or partition; I never used that, and overlooked what KZ> you were trying to tell me in my original response. I've been KZ> working since yesterday to implement your suggestions. Is it that much I suggested ? ;-) KZ> I'll modify the slashes to back-slashes today, when I'm able KZ> to make an individual run for just the admin //db/f$ share. The back-slashes work out well in one of my client's installation. KZ> WRT the '$' metacharacter, what I actually had to run was: KZ> amdump DailySet1 admin '//db/f\$' Errm, yes, I could have told you that ... escaping meta-characters it is ... KZ> Thanks, again, for your help and suggestions. You're welcome ... -- best regards, Stefan Stefan G. Weichinger mailto:[EMAIL PROTECTED]
