Re: High CPU running backup
--On Friday, March 19, 2004 14:47:44 -0500 Jonathan Dill <[EMAIL PROTECTED]> wrote: > I would guess that the "ufsrestore" is making an "index" of one of the dumps. If > you don't care about interactive "amrecover" you could make a dumptype that doesn't > do "index" that should eliminate the ufsrestore process. Running fewer dumps in > parallel should help, too. > > I don't know a lot about Solaris. 0.0% swap should mean nothing is paging, but with > 729M swap used, I'd still check out high-memory processes, maybe something is > leaking memory, but any paging should still show up in swap and not kernel I would > think. > It looks to me like it wouldn't take a lot to make the system start paging heavily > from the state that it is in with only 13M of real memory free. Actually, in Solaris (and Linux) it is normal to not show much free RAM, as all reads are left in cache (to speed up any subsequent reads) until something else needs to use the RAM. The only real way to tell if you need more RAM is to watch for swapping activity. Frank > > Simon Lorenz wrote: > >> Any suggestions for stopping this would be much appreciated. >> >> load averages: 2.21, 2.21, 2.02 >> 17:48:54 >> 142 processes: 131 sleeping, 5 running, 1 zombie, 4 stopped, 1 on cpu >> CPU states: 0.0% idle, 13.7% user, 76.9% kernel, 9.3% iowait, 0.0% swap >> Memory: 768M real, 13M free, 729M swap in use, 3911M swap free >> >> PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND >> 10563 amanda 1 220 2232K 936K sleep5:32 35.03% sendbackup >> 10476 amanda 1 0 19 2640K 1920K run 2:26 15.04% dumper >> 10568 amanda 1 320 11M 9928K sleep1:49 12.55% ufsrestore >> 10572 amanda 1 480 11M 2864K run 0:47 5.20% ufsdump >> 10571 amanda 1 530 11M 2864K run 0:48 5.16% ufsdump >> 10573 amanda 1 480 11M 2872K run 0:47 4.64% ufsdump >> 10574 amanda 1 360 11M 3072K sleep0:32 3.53% ufsdump >> 10570 amanda 1 480 11M 7520K sleep0:20 1.89% ufsdump >> 10646 root 1 580 2104K 1208K cpu 0:01 0.35% top >> >> -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: High CPU running backup
I would guess that the "ufsrestore" is making an "index" of one of the dumps. If you don't care about interactive "amrecover" you could make a dumptype that doesn't do "index" that should eliminate the ufsrestore process. Running fewer dumps in parallel should help, too. I don't know a lot about Solaris. 0.0% swap should mean nothing is paging, but with 729M swap used, I'd still check out high-memory processes, maybe something is leaking memory, but any paging should still show up in swap and not kernel I would think. It looks to me like it wouldn't take a lot to make the system start paging heavily from the state that it is in with only 13M of real memory free. Simon Lorenz wrote: Any suggestions for stopping this would be much appreciated. load averages: 2.21, 2.21, 2.02 17:48:54 142 processes: 131 sleeping, 5 running, 1 zombie, 4 stopped, 1 on cpu CPU states: 0.0% idle, 13.7% user, 76.9% kernel, 9.3% iowait, 0.0% swap Memory: 768M real, 13M free, 729M swap in use, 3911M swap free PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 10563 amanda 1 220 2232K 936K sleep5:32 35.03% sendbackup 10476 amanda 1 0 19 2640K 1920K run 2:26 15.04% dumper 10568 amanda 1 320 11M 9928K sleep1:49 12.55% ufsrestore 10572 amanda 1 480 11M 2864K run 0:47 5.20% ufsdump 10571 amanda 1 530 11M 2864K run 0:48 5.16% ufsdump 10573 amanda 1 480 11M 2872K run 0:47 4.64% ufsdump 10574 amanda 1 360 11M 3072K sleep0:32 3.53% ufsdump 10570 amanda 1 480 11M 7520K sleep0:20 1.89% ufsdump 10646 root 1 580 2104K 1208K cpu 0:01 0.35% top
Re: High CPU running backup
--On Friday, March 19, 2004 18:00:01 + Simon Lorenz <[EMAIL PROTECTED]> wrote: > I have a Solaris 8 system running Amanda that ginds to a hault when the > backups are running. Amanda and the sub processes take all avliable CPU. > This is dispite having compression set to none (using tape device). It is > primarily the snedbackup, dumper and ufsrestore processes that are causing > trhe issue. > > Any suggestions for stopping this would be much appreciated. > > load averages: 2.21, 2.21, 2.02 > 17:48:54 > 142 processes: 131 sleeping, 5 running, 1 zombie, 4 stopped, 1 on cpu > CPU states: 0.0% idle, 13.7% user, 76.9% kernel, 9.3% iowait, 0.0% swap > Memory: 768M real, 13M free, 729M swap in use, 3911M swap free The part that looks odd to me is the high kernel CPU usage. I would expect to see mostly user and iowat usage while running the backups. Are you backing up NFS or vxfs filesystems? Both of those drivers use kernel time, although I wouldn't expect the usage to be quite that high. How many dumps are you running in parallel? Perhaps reducing that number would give you an overall speedup Frank > >PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND > 10563 amanda 1 220 2232K 936K sleep5:32 35.03% sendbackup > 10476 amanda 1 0 19 2640K 1920K run 2:26 15.04% dumper > 10568 amanda 1 320 11M 9928K sleep1:49 12.55% ufsrestore > 10572 amanda 1 480 11M 2864K run 0:47 5.20% ufsdump > 10571 amanda 1 530 11M 2864K run 0:48 5.16% ufsdump > 10573 amanda 1 480 11M 2872K run 0:47 4.64% ufsdump > 10574 amanda 1 360 11M 3072K sleep0:32 3.53% ufsdump > 10570 amanda 1 480 11M 7520K sleep0:20 1.89% ufsdump > 10646 root 1 580 2104K 1208K cpu 0:01 0.35% top > > > Simon -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
High CPU running backup
I have a Solaris 8 system running Amanda that ginds to a hault when the backups are running. Amanda and the sub processes take all avliable CPU. This is dispite having compression set to none (using tape device). It is primarily the snedbackup, dumper and ufsrestore processes that are causing trhe issue. Any suggestions for stopping this would be much appreciated. load averages: 2.21, 2.21, 2.02 17:48:54 142 processes: 131 sleeping, 5 running, 1 zombie, 4 stopped, 1 on cpu CPU states: 0.0% idle, 13.7% user, 76.9% kernel, 9.3% iowait, 0.0% swap Memory: 768M real, 13M free, 729M swap in use, 3911M swap free PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 10563 amanda 1 220 2232K 936K sleep5:32 35.03% sendbackup 10476 amanda 1 0 19 2640K 1920K run 2:26 15.04% dumper 10568 amanda 1 320 11M 9928K sleep1:49 12.55% ufsrestore 10572 amanda 1 480 11M 2864K run 0:47 5.20% ufsdump 10571 amanda 1 530 11M 2864K run 0:48 5.16% ufsdump 10573 amanda 1 480 11M 2872K run 0:47 4.64% ufsdump 10574 amanda 1 360 11M 3072K sleep0:32 3.53% ufsdump 10570 amanda 1 480 11M 7520K sleep0:20 1.89% ufsdump 10646 root 1 580 2104K 1208K cpu 0:01 0.35% top Simon
Re: flush full dumps only
Jonathan Dill wrote: almost so simple as to be a joke, but I don't have any idea what kind of bugs can be found. See below :-) #!/bin/tcsh -f set r=(/snap/amanda-hd) set i=(/snap/amanda-incr) cd $r set dl=(`find . -depth -type d -print`) cd $i foreach d ($dl) if (! -d $d) then echo mkdir -p $d endif end cd $r foreach d ($dl) set fl=(`find . -type f \! -name "*.0" -print`) foreach f ($fl) echo mv $f $i/$f end end The last "find" command is intended to get all but the level zero dumps, but it is not completely correct: Full dumps that were split into chunks have a dot + chunknumber appended after the level ".0". I can't come up with a perfect solution, but the find below should work fine if the chunks are not too small (if each filesystem is 100 chunks at most). find . -type f \! -name "*.0" \ -a \! -name "*.0.[0-9]" \ -a \! -name "*.0.[0-9][0-9]" \ -print at least if we don't have a DLE that ends in ".0", and where the incremental backup would end in ".0.1". :-( ps. You could replace the first "find, mkdir" with: cd $r; find . -type d | cpio -pd $i pps. If you don't mind flush the incrementals to tape too, the script could be: cd $r find . -type f \! -name "*.0" \ -a \! -name "*.0.[0-9]" \ -a \! -name "*.0.[0-9][0-9]" \ -print | tee /tmp/amanda/flushonly | cpio -pdml $i rm `cat /tmp/amanda/flushonly` and you should avoid DLE's with confusing names... -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
Re: flush full dumps only
Hi John,
This simple patch should do a better job, but it should be configurable
by an option in the configuration file:
eg. taping_level [full|incr|all]
Jean-Louis
On Fri, Mar 19, 2004 at 12:00:16PM -0500, Jonathan Dill wrote:
> I have attached a very simple but effective script that I set up to
> separate full and incremental holding disk files, so that full dumps can
> be flushed to tape while incrementals remain on disk. The script is
> almost so simple as to be a joke, but I don't have any idea what kind of
> bugs can be found.
>
> WARNING: The script is provided as-is with no warranty express or
> implied. If you screw up your amanda backups do not come crying to me.
> To really make the script work, you will need to change the values of $r
> and $i for your configuration, run it to see what it will do, and then
> remove the "echo" from the beginning of the two lines. If you can't
> figure out what that means, then you probably shouldn't be screwing
> around with this stuff anyway and should forget about it.
>
> For restores, I think I can flush the latest batch of full dumps, then
> use a modified amanda.conf that looks in the directory where the
> incrementals are stored, but I am not sure how amanda decides where
> holding disk files are located. I might have a use another script to
> move the incrementals back to the normal holding disk tree, but I will
> cross that bridge when I come to it, to use a rather hackneyed
> expression.
>
> --
> Jonathan Dill <[EMAIL PROTECTED]>
> #!/bin/tcsh -f
> set r=(/snap/amanda-hd)
> set i=(/snap/amanda-incr)
> cd $r
> set dl=(`find . -depth -type d -print`)
> cd $i
> foreach d ($dl)
> if (! -d $d) then
> echo mkdir -p $d
> endif
> end
> cd $r
> foreach d ($dl)
> set fl=(`find . -type f \! -name "*.0" -print`)
> foreach f ($fl)
> echo mv $f $i/$f
> end
> end
--
Jean-Louis Martineau email: [EMAIL PROTECTED]
Département IRO, Université de Montréal
C.P. 6128, Succ. CENTRE-VILLETel: (514) 343-6111 ext. 3529
Montréal, Canada, H3C 3J7Fax: (514) 343-5834
diff -u -r --show-c-function --exclude-from=amanda.diff
amanda-2.4.5b1.orig/server-src/amflush.c amanda-2.4.5b1.new/server-src/amflush.c
--- amanda-2.4.5b1.orig/server-src/amflush.c2003-06-05 13:06:20.0 -0400
+++ amanda-2.4.5b1.new/server-src/amflush.c 2004-03-19 12:40:17.0 -0500
@@ -276,6 +276,7 @@ char **main_argv;
dp = lookup_disk(file.name, file.disk);
if (dp->todo == 0) continue;
+ if (file.dumplevel != 0) continue;
fprintf(stderr,
"FLUSH %s %s %s %d %s\n",
diff -u -r --show-c-function --exclude-from=amanda.diff
amanda-2.4.5b1.orig/server-src/driver.c amanda-2.4.5b1.new/server-src/driver.c
--- amanda-2.4.5b1.orig/server-src/driver.c 2004-02-13 09:03:36.0 -0500
+++ amanda-2.4.5b1.new/server-src/driver.c 2004-03-19 12:40:32.0 -0500
@@ -1126,7 +1126,8 @@ void handle_dumper_result(fd)
dp->host->hostname, dp->name);
fflush(stdout);
- enqueue_disk(&tapeq, dp);
+ if(sched(dp)->level == 0)
+ enqueue_disk(&tapeq, dp);
dp = NULL;
startaflush();
diff -u -r --show-c-function --exclude-from=amanda.diff
amanda-2.4.5b1.orig/server-src/planner.c amanda-2.4.5b1.new/server-src/planner.c
--- amanda-2.4.5b1.orig/server-src/planner.c2004-02-13 09:01:08.0 -0500
+++ amanda-2.4.5b1.new/server-src/planner.c 2004-03-19 12:43:14.0 -0500
@@ -536,7 +536,9 @@ char **argv;
for(holding_file=holding_list->first; holding_file != NULL;
holding_file = holding_file->next) {
get_dumpfile(holding_file->name, &file);
-
+
+ if (file.dumplevel != 0) continue;
+
fprintf(stderr,
"FLUSH %s %s %s %d %s\n",
file.name,
Re: flush full dumps only
I have attached a very simple but effective script that I set up to separate full and incremental holding disk files, so that full dumps can be flushed to tape while incrementals remain on disk. The script is almost so simple as to be a joke, but I don't have any idea what kind of bugs can be found. WARNING: The script is provided as-is with no warranty express or implied. If you screw up your amanda backups do not come crying to me. To really make the script work, you will need to change the values of $r and $i for your configuration, run it to see what it will do, and then remove the "echo" from the beginning of the two lines. If you can't figure out what that means, then you probably shouldn't be screwing around with this stuff anyway and should forget about it. For restores, I think I can flush the latest batch of full dumps, then use a modified amanda.conf that looks in the directory where the incrementals are stored, but I am not sure how amanda decides where holding disk files are located. I might have a use another script to move the incrementals back to the normal holding disk tree, but I will cross that bridge when I come to it, to use a rather hackneyed expression. -- Jonathan Dill <[EMAIL PROTECTED]> #!/bin/tcsh -f set r=(/snap/amanda-hd) set i=(/snap/amanda-incr) cd $r set dl=(`find . -depth -type d -print`) cd $i foreach d ($dl) if (! -d $d) then echo mkdir -p $d endif end cd $r foreach d ($dl) set fl=(`find . -type f \! -name "*.0" -print`) foreach f ($fl) echo mv $f $i/$f end end
Re: Firewall and Portrange Settings
Thanks for all the help and comments. I have started using the ip_conntrack_amanda module in my iptables setup and that seems to work fine. The issue of stock packages vs. personalized compiles is an interesting philosophical debate I'd rather not get into at this point. Both approaches have their merits, and both are appropriate under certain circumstances. I prefer to use stock packages where they provide enough power and flexibility to solve the problem at hand. The combination of amanda, iptables, and ip_conntrack_amanda does that for me in this case. I understand that YMMV. Thanks again to all.
RE: I changed something in my disklist
Got that thanks.
>-Original Message-
>From: Paul Bijnens [mailto:[EMAIL PROTECTED]
>Sent: Friday, March 19, 2004 10:27 AM
>To: Gavin Henry
>Cc: [EMAIL PROTECTED]
>Subject: Re: I changed something in my disklist
>
>
>Gavin Henry wrote:
>
>>>You switch from gnutar to dump. Gnutar can do subdirectories, dump
>>>cannot do.
>>>Add "program GNUTAR" to the dumptype "nocomp-high" (and better name
>>>it "high-tar" if that one does not already exist).
>>>
>>
>>
>> I added the program bit. There is already a high-tar
>section, but what do
>> you mean "name it"
>
>It's up to you how to name a dumptype. But it is very confusing
>if the name doesn't match what it does.
>If you have dumptypes named "high" and "high-tar", then the name
>suggests that "high-tar" is similar to "high" but using program GNUTAR.
>
>It happens now that amanda has in the example directory a lot of
>dumptypes. Many people copy the example amanda.conf, and modify it
>to their needs. In the example there is "comp-high" and a
>"nocomp-high"
>and the only difference is the compression property.
>The example also show a different naming scheme, where, the priority
>is expressed by 'root' or 'user' (root filesystems don't
>change as much,
>and you can always reinstall from a CD, his has lower priority for
>backups, whereas user filesystems contain more uniq data, which is of
>higher priority to backup). So the names invented for dumptypes are
>"root-tar", "user-tar", "high-tar" (for even more important data),
>all three without compression, and there are two similar with
>compression "comp-root-tar" and "comp-user-tar".
>
>It are only examples, it's up to you to define your own.
>If you blindly copy the examples and then add 'program "GNUTAR"' to
>"high-tar", then the next person looking at the config with the 5
>dumptypes will probably get very confused.
>
>To really confuse (or to show off) you could name a dumptype:
>
>define dumptype legato {
> ...
>}
>
>And then show the disklist to your manager if you want a larger budget.
>
>--
>Paul Bijnens, XplanationTel +32 16 397.511
>Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512
>http://www.xplanation.com/ email: [EMAIL PROTECTED]
>***
>* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
>* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
>* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
>* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
>* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...*
>* ... "Are you sure?" ... YES ... Phew ... I'm out *
>***
>
>
Re: I changed something in my disklist
Gavin Henry wrote:
You switch from gnutar to dump. Gnutar can do subdirectories, dump
cannot do.
Add "program GNUTAR" to the dumptype "nocomp-high" (and better name
it "high-tar" if that one does not already exist).
I added the program bit. There is already a high-tar section, but what do
you mean "name it"
It's up to you how to name a dumptype. But it is very confusing
if the name doesn't match what it does.
If you have dumptypes named "high" and "high-tar", then the name
suggests that "high-tar" is similar to "high" but using program GNUTAR.
It happens now that amanda has in the example directory a lot of
dumptypes. Many people copy the example amanda.conf, and modify it
to their needs. In the example there is "comp-high" and a "nocomp-high"
and the only difference is the compression property.
The example also show a different naming scheme, where, the priority
is expressed by 'root' or 'user' (root filesystems don't change as much,
and you can always reinstall from a CD, his has lower priority for
backups, whereas user filesystems contain more uniq data, which is of
higher priority to backup). So the names invented for dumptypes are
"root-tar", "user-tar", "high-tar" (for even more important data),
all three without compression, and there are two similar with
compression "comp-root-tar" and "comp-user-tar".
It are only examples, it's up to you to define your own.
If you blindly copy the examples and then add 'program "GNUTAR"' to
"high-tar", then the next person looking at the config with the 5
dumptypes will probably get very confused.
To really confuse (or to show off) you could name a dumptype:
define dumptype legato {
...
}
And then show the disklist to your manager if you want a larger budget.
--
Paul Bijnens, XplanationTel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512
http://www.xplanation.com/ email: [EMAIL PROTECTED]
***
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...*
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***
RE: I changed something in my disklist
> >You switch from gnutar to dump. Gnutar can do subdirectories, dump >cannot do. >Add "program GNUTAR" to the dumptype "nocomp-high" (and better name >it "high-tar" if that one does not already exist). > I added the program bit. There is already a high-tar section, but what do you mean "name it" Cheers.
Re: I changed something in my disklist
Gavin Henry wrote: I changed my disklist the other day, as you might remember from previous post regarding data timeouts on my /var partition. It is now at nocomp-high I don't remember what is was before, was it "comp-high-tar" ? | DUMP: You can't update the dumpdates file when dumping a subdirectory | DUMP: The ENTIRE dump is aborted. You switch from gnutar to dump. Gnutar can do subdirectories, dump cannot do. Add "program GNUTAR" to the dumptype "nocomp-high" (and better name it "high-tar" if that one does not already exist). -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
I changed something in my disklist
Hi all, I changed my disklist the other day, as you might remember from previous post regarding data timeouts on my /var partition. It is now at nocomp-high I am not sure what the errors mean? thanks. /-- whitehat /var lev 0 FAILED [/sbin/dump returned 1] sendbackup: start [whitehat:/var level 0] sendbackup: info BACKUP=/sbin/dump sendbackup: info RECOVER_CMD=/sbin/restore -f... - sendbackup: info end | DUMP: You can't update the dumpdates file when dumping a subdirectory | DUMP: The ENTIRE dump is aborted. sendbackup: error [/sbin/dump returned 1] \
