Re: speed of amdump
>the speed of our hp surestore seems to be ok, but the amdump takes too much ti
>me
> (6 hours for 23 GB, we want to use now client fast compression).
There are several areas that can affect speed, and it's important to
understand the flow of data through Amanda to see where the potential
bottlenecks are:
* The client dump program reads data from the client disk.
- How fast is the disk?
- How fast is the disk -> system connection (e.g. SCSI bus/controller)?
- Are you backing up lots and lots of little files, which will
involve more seeking?
* If you have maxdumps set greater than one, more than one backup may
be going on at the same time. If these are backing up separate areas
on the same physical disk, seeking will slow things down. You can
control this with the "spindle" field in the disklist file.
* If you have client compression enabled, the dump image is run
through either "compress" or "gzip", depending on how Amanda was
built. Using "best" translates to "-9" on gzip, which is known to
take up to several times as long to compress the data, but with
minimal improvement in compression ratio (depending on the data,
of course).
Also, if maxdumps > 1 and multiple backups are running at the same
time, that will multiply the system load.
* The image is then sent over the network to the server. Are you
seeing collisions or errors in the network statistics on either side?
That's often an indication of a duplex mismatch.
What kind of transfer rate to you get if you do a test ftp from
client to /dev/null on the server of a few MBytes?
* On the server side, if you have compression turned on (which cannot
also be on if client side compression is enabled), the image is sent
through a compression program and the same points as on the client
apply.
* If there was enough space in the holding disk, the image is written
there. The same questions that were asked about the speed of the
client disk apply here.
In addition, if you have multiple clients or maxdumps > 1, multiple
images may be stored in the holding disk at the same time which will
increase seeking.
* If there is not enough holding disk space, the image will be written
directly to tape. How fast is the tape drive? How fast is the
interface to it?
If you are using software compression on either the client or the
server, make absolutely certain you do not also have hardware
compression turned on. In addition to expanding the amount of tape
space used, this can slow down the transfer.
As others have said, how hardware compression is enabled or disabled
is highly OS specific, and you didn't mention what your server
is running.
Note that I'm not asking you to post answers to these questions to
the list. They are being presented for you to ask yourself, although,
of course, if you continue to have trouble you might want to share
the details.
My guess from what you've said is that the client is doing "best"
compression and that's killing the throughput. I'd certainly suggest
dropping that back to "fast". You might also consider which system is
faster, the client or the server, and shift to doing compression on the
server if it's faster and the extra network traffic (because the whole
image will be transmitted, not the compressed version) warrant.
There are other factors (TCP stream buffer size, routers, bent cable pins,
etc), but the above should be checked out and eliminated first.
>In amanda.conf:
>netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
>But we have a 100mbit LAN, so there should be 10240 kbps, thats right? Or what
>
>is the meaning of this entry? Which values do you have?
This entry controls whether another backup may be started on a client.
For instance, if Amanda estimates a currently running backup is going
to use 500 Kbps (based on the estimated size and historical speed
information) and a candidate second backup is going to use 200 Kbps, the
second backup will be held up (500 + 200 > 600). If the first backup is
estimated to use 250, the second will not be held up due to the bandwidth
(interface) limit (250 + 200 <= 600).
Note that several things go into deciding whether a second (or third,
etc) backup may be started -- is there enough holding disk space, is there
an available dumper (inparallel), will the client support another backup
(maxdumps), etc. Network use is just one constraint.
The interface speed value does **not** control how fast Amanda moves data.
Once data starts flowing, all that is left entirely up to the OS and
hardware.
>In /etc/amanda/interfaces:
>define interface local {
> comment "a local disk"
> use 1000 kbps
>}
I use 1 kbps. Basically, this is to make sure the interface is not
a limit when dumping local disks (it could even be made much higher).
>define interface eth0 {
> comment "100 Mb
Re: speed of amdump
>the speed of our hp surestore seems to be ok, but the amdump takes too >much time (6 hours for 23 GB, we want to use now client fast >compression). How fast can you get data off of your disks? If that 23G is one filesystem, you might consider splitting it up so that Amanda can schedule it more evenly. If you're stuck using GNU tar, you've got filesystem overhead factored in there too.
Re: speed of amdump [really, speed of ethernet]
On Fri, Jan 24, 2003 at 08:31:19AM -0600, [EMAIL PROTECTED] wrote: > And if any of us can RELIABLY exceed 30 percent usage on > an Ethernet network, I want to see it. Well, one of my amanda clients does two dumps in parallel over 100BaseT at >=3MB/s dumper rate (each) every single night. Does that count? :-) 2*3MB/s -> 48 Mbit/s. Minimum. They average more like 3.5MB/s each. As long as the packet size is reasonable, any competently implemented 10BaseT or 100BaseT link can gracefully handle sustained payload peaks up to 90% of wire speed, no problem. I plan on 50% wire speed utilization for my 11pm-5am backup period, and I consider that very conservative. 1000BaseT is different, at least with stock interoperable HW and SW; the 1500-byte packet size limits you to something like 50-65% of wire speed. You need semi-proprietary, not-completely interoperable jumbo packet HW and SW to go faster. > As always, your mileage may vary ... Yup! -- Jay Lessert [EMAIL PROTECTED] Accelerant Networks Inc. (voice)1.503.439.3461 Beaverton OR, USA(fax)1.503.466.9472
Re: speed of amdump
On Fri, Jan 24, 2003 at 09:59:48AM +0100, dalton wrote: > Hi, > > the speed of our hp surestore seems to be ok, but the amdump takes too much > time (6 hours for 23 GB, we want to use now client fast compression). > > So we have a suspicion (someone just has installed it, but this person is > not here any more :-(: You can glean a lot of info from the amreport mailed to you. Like how much time spent in taping, how much in estimating, ... It may give a pointer to the culprit. Post a copy of the report from that run. One thing, you are using a holding disk, yes? What is the reserve value set to? Hopefully not the 100% default. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
RE: speed of amdump
And if any of us can RELIABLY exceed 30 percent usage on an Ethernet network, I want to see it. For networks, plan on using about 10 percent of the available bandwidth in your capacity planning and you should be safe. For 10 Mbit Ethernet, figure 100 KB/sec, for 100 Mbit Ethernet, figure 1 MB/sec, for 1 Gbit Ethernet, figure 10 MB/sec. If you are using a network switch instead of a hub, that helps, but it is still better to be on the safe side and plan conservatively. As always, your mileage may vary ... Don Ritchey [EMAIL PROTECTED] -Original Message- From: Gene Heskett [mailto:[EMAIL PROTECTED]] Sent: Friday, January 24, 2003 7:35 AM To: Scott Mcdermott; [EMAIL PROTECTED] Subject: Re: speed of amdump On Friday 24 January 2003 05:35, Scott Mcdermott wrote: >dalton on Fri 24/01 09:59 +0100: >> But we have a 100mbit LAN, so there should be 10240 kbps, thats >> right? > >just being picky, > >$ bc -ql >100 * (2^10)^2 / 8 / 2^10 >12800. > >:) of course if you actually hit that wire speed I'll be amazed. > I don't believe this is quite correct. Thats a serial protocol, and AFAIK it still uses start and stop bits, so the real /8/ divisor above should probably be /10/, giving a 10 megabytes or 100 megabaud a second rate, hence the naming convention. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.22% setiathome rank, not too shabby for a WV hillbilly This e-mail and any of its attachments may contain Exelon Corporation proprietary information, which is privileged, confidential, or subject to copyright belonging to the Exelon Corporation family of Companies. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. Thank You.
Re: speed of amdump
Gene Heskett on Fri 24/01 08:34 -0500: > > > But we have a 100mbit LAN, so there should be 10240 kbps, thats > > > right? > > > > $ bc -ql > > 100 * (2^10)^2 / 8 / 2^10 > > 12800. > > I don't believe this is quite correct. Thats a serial protocol, and > AFAIK it still uses start and stop bits, so the real /8/ divisor above > should probably be /10/, giving a 10 megabytes or 100 megabaud a > second rate, hence the naming convention. Ethernet is synchronous, and framed; the overhead is nowhere near that. http://sd.wareonearth.com/~phil/net/overhead/ looks like it comes out to about 12051 KBps, or 11.8MBps.
Re: speed of amdump
On Friday 24 January 2003 05:35, Scott Mcdermott wrote: >dalton on Fri 24/01 09:59 +0100: >> But we have a 100mbit LAN, so there should be 10240 kbps, thats >> right? > >just being picky, > >$ bc -ql >100 * (2^10)^2 / 8 / 2^10 >12800. > >:) of course if you actually hit that wire speed I'll be amazed. > I don't believe this is quite correct. Thats a serial protocol, and AFAIK it still uses start and stop bits, so the real /8/ divisor above should probably be /10/, giving a 10 megabytes or 100 megabaud a second rate, hence the naming convention. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.22% setiathome rank, not too shabby for a WV hillbilly
Re: speed of amdump
dalton on Fri 24/01 09:59 +0100: > But we have a 100mbit LAN, so there should be 10240 kbps, thats right? just being picky, $ bc -ql 100 * (2^10)^2 / 8 / 2^10 12800. :) of course if you actually hit that wire speed I'll be amazed.
Re: speed of amdump
Lars Segerlund schrieb:
have you checked the options for parrallell dumpers ?
Yes i have 4.
You should be able to have multiple connections to clients, and your
time seem's looong !
We have just one client (the file server) and the backup server.
And the interface-values and bandwith-values are ok?
At client site also an amanda-config section?
thx, dalton
Basicly as long as you have the bandwidth you should be able to get
down close to the backup time for the simgle slowest client.
/ regards, Lars Segerlund.
dalton wrote:
Hi,
the speed of our hp surestore seems to be ok, but the amdump takes too
much time (6 hours for 23 GB, we want to use now client fast
compression).
So we have a suspicion (someone just has installed it, but this person
is not here any more :-(:
In amanda.conf:
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
But we have a 100mbit LAN, so there should be 10240 kbps, thats right?
Or what is the meaning of this entry? Which values do you have?
In /etc/amanda/interfaces:
define interface local {
comment "a local disk"
use 1000 kbps
}
define interface eth0 {
comment "100 Mbps ethernet"
use 10240 kbps
}
For a local disk just 1000 kbps? Too less?
Should be much more, that's right?
Which values do you have?
One more question:
We have a backup server where amanda runs, and a file server.
amanda gets the files from the file server (by clients best
compression, now we want to use client fast).
Have to be an /etc/amanda/... - directory also on the client site (in
our case the file server?)
There is one, but we are not sure, if the client needs it.
Thank you for your help,
best regards
Dalton
speed of amdump
Hi,
the speed of our hp surestore seems to be ok, but the amdump takes too much time
(6 hours for 23 GB, we want to use now client fast compression).
So we have a suspicion (someone just has installed it, but this person is not
here any more :-(:
In amanda.conf:
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
But we have a 100mbit LAN, so there should be 10240 kbps, thats right? Or what
is the meaning of this entry? Which values do you have?
In /etc/amanda/interfaces:
define interface local {
comment "a local disk"
use 1000 kbps
}
define interface eth0 {
comment "100 Mbps ethernet"
use 10240 kbps
}
For a local disk just 1000 kbps? Too less?
Should be much more, that's right?
Which values do you have?
One more question:
We have a backup server where amanda runs, and a file server.
amanda gets the files from the file server (by clients best compression, now we
want to use client fast).
Have to be an /etc/amanda/... - directory also on the client site (in our case
the file server?)
There is one, but we are not sure, if the client needs it.
Thank you for your help,
best regards
Dalton
