Re: Performance degrading over time?
>or adding 5 internal 36 GB drives in a RAID 0 configuration as a holding >disk. I see that IBM has SCSI disks available up to at least 146G. I'd recommend plunking in two of those instead so you have room to grow.
Re: Performance degrading over time?
Hi! > > > nothing goes from holding disk to tape unless the dump to holding disk > > > has finished. Then it goes like cat'ting or dd'ing to tape. Small > > > likelyhood of too slow for tape. > > > > Aggreed - cat/dd >/dev/tape will surely be fast enough. > > But I need a holding disk at least as large as my largest FS to dump? > > So if I have one 170 GB RAID I need one 170 GB holding disk? > > Repeating, > > > > nothing goes from holding disk to tape unless the dump to holding disk > > > has finished. OK. Understood - finally. > > The customer won't like that ;-) > > Doesn't have to be high performance drives. > Cheap IDE drives are way fast enough. How do you fit cheap IDE drives into a Sun Enterprise 3500? Configuration of the machine: 1 internal 9 GB (system) drive 1 external Sun Storedge A1000 RAID enclosure - 170 GB net storage 1 external LTO drive Oooops :-))) Well, you can't argue against the facts - I'll suggest either getting rid of the small "trace" files on a nightly basis or disabling that "feature" altogether - or adding 5 internal 36 GB drives in a RAID 0 configuration as a holding disk. Hmmm ... possibly some cheap PC style external SCSI-to-IDE-RAID will do the trick just as well. Sun won't officially support it, but they don't support the HP LTO, either. Thanks to all, end of thread (hopefully) Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
Re: Performance degrading over time?
On Mon, Nov 04, 2002 at 03:55:13PM +0100, Patrick M. Hausen wrote: > But I need a holding disk at least as large as my largest FS to dump? > So if I have one 170 GB RAID I need one 170 GB holding disk? No - you need a holding disk preferrably as big as your TWO largest disklist entries, which may be <= your largest FS. From "Using Amanda" ] Ideally, there should be enough holding disk space for the two largest ] backup images simultaneously, so one image can be coming into the holding ] disk while the other is being written to tape. If that is not practical, any ] amount that holds at least a few of the smaller images helps. > The customer won't like that ;-) Customers rarely like anything which costs them money :-) > Or is this what the chunksize parameter is for - taper will start when > the first chunk is completely written to the holding disk? No - chunksize merely specifies how big are the chunks the backup is split into on disk. Really only needed on older OS which have a relatively small limit (often 2GB) to the size of a single file. > In this case I'm sure a holding disk will speed up things quite a > bit even in my "pathological" case of only one big FS. No - a holding disk will only help you at all if it's at least as big as your two smallest disklist entries. Kindest regards, Niall O Broin
Re: Performance degrading over time?
--On Monday, November 04, 2002 15:04:27 +0100 "Patrick M. Hausen" <[EMAIL PROTECTED]> wrote: Paul Bijnens wrote: Patrick M. Hausen wrote: > > Seems like Oracle likes to create a lot of small ".trc" files over > time. The filesystem in question is littered with thousands of them. > > Once we archived and deleted them, backup performance was back to normal. > > Would separate holding disk (we don't use one at all at the moment) > help in a configuration like this? Additionally I'd suggest deleting Probably, then your tapedrive can keep streaming (could half your backuptime!), and amanda can do much more parallel then it can do without holdingdisk (another doubling or more if you have many clients). Precisely. In all multi-client installations I run in my own network I have holding-disks, so dumps can be run in parallel and output buffered. But the machine in question is a one-server-client installation that only backs up itself. And in addition it mainly backs up one single file system. So my question is: does a holding-disk speed up this process? I mean, Amanda will start a dumper on the filesystem that starts filling the holding-disk. At the same time (?) a taper will start wrtiting the holding-disk's contents to tape. Now imagine the dumper getting to slow for the tape ... the holding-disk won't be filled quick enough either. Then there's a lot of seeks on the holding-disk itself, if it's read and written at the same time. Or is the mental picture I have about Amanda's operation incorrect? What should happen is that dumper will write the entire dump to the holding disk (either in one piece or in N chunksize pieces if you specify chunksize) and then when it is done stream the entire image to tape. If your holding disk is smaller than your dump then it will be bypassed and you will be dumping directly to tape with the pauses and reseeks you are seeing now. Frank Thanks, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de -- Frank Smith[EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Performance degrading over time?
Hello! Jon H. LaBadie wrote: > On Mon, Nov 04, 2002 at 03:04:27PM +0100, Patrick M. Hausen wrote: > > > > So my question is: does a holding-disk speed up this process? I mean, > > Amanda will start a dumper on the filesystem that starts filling > > the holding-disk. At the same time (?) a taper will start wrtiting > > the holding-disk's contents to tape. Now imagine the dumper getting > > to slow for the tape ... the holding-disk won't be filled quick > > enough either. Then there's a lot of seeks on the holding-disk > > itself, if it's read and written at the same time. > > > > Or is the mental picture I have about Amanda's operation incorrect? > > yes, incorrect. > > nothing goes from holding disk to tape unless the dump to holding disk > has finished. Then it goes like cat'ting or dd'ing to tape. Small > likelyhood of too slow for tape. Aggreed - cat/dd >/dev/tape will surely be fast enough. But I need a holding disk at least as large as my largest FS to dump? So if I have one 170 GB RAID I need one 170 GB holding disk? The customer won't like that ;-) Or is this what the chunksize parameter is for - taper will start when the first chunk is completely written to the holding disk? In this case I'm sure a holding disk will speed up things quite a bit even in my "pathological" case of only one big FS. A little bit of tape stop-and-go between chunks won't hurt as much as the current configuration does. Thanks, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
Re: Performance degrading over time?
Paul Bijnens wrote: > Patrick M. Hausen wrote: > > > > Seems like Oracle likes to create a lot of small ".trc" files over > > time. The filesystem in question is littered with thousands of them. > > > > Once we archived and deleted them, backup performance was back to normal. > > > > Would separate holding disk (we don't use one at all at the moment) > > help in a configuration like this? Additionally I'd suggest deleting > > Probably, then your tapedrive can keep streaming (could half your > backuptime!), and amanda can do much more parallel then it can do > without holdingdisk (another doubling or more if you have many > clients). Precisely. In all multi-client installations I run in my own network I have holding-disks, so dumps can be run in parallel and output buffered. But the machine in question is a one-server-client installation that only backs up itself. And in addition it mainly backs up one single file system. So my question is: does a holding-disk speed up this process? I mean, Amanda will start a dumper on the filesystem that starts filling the holding-disk. At the same time (?) a taper will start wrtiting the holding-disk's contents to tape. Now imagine the dumper getting to slow for the tape ... the holding-disk won't be filled quick enough either. Then there's a lot of seeks on the holding-disk itself, if it's read and written at the same time. Or is the mental picture I have about Amanda's operation incorrect? Thanks, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
Re: Performance degrading over time?
Hi all! > Since your dumper and taper times are always nearly identical you probably > aren't using a holding disk and are dumping directly to tape. And since > the rates for one filesystem have remained constant while the other one has > dropped I would look into possible recent changes on the larger (slower) > filesystem. Try doing a dump to /dev/null and see how fast (or slow) that is. > If data isn't fed to the tape fast enough the tape drive has to stop and > reposition itself every time it runs out of data, which will slow it down > considerably. We finally found the culprit, yet have to decide what to do about it. Seems like Oracle likes to create a lot of small ".trc" files over time. The filesystem in question is littered with thousands of them. Once we archived and deleted them, backup performance was back to normal. Would separate holding disk (we don't use one at all at the moment) help in a configuration like this? Additionally I'd suggest deleting the trace (?) files in a nightly run - seems like nobody needs them, anyway. Thanks for your help, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
Re: Performance degrading over time?
--On Tuesday, October 29, 2002 11:43:30 +0100 "Patrick M. Hausen" <[EMAIL PROTECTED]> wrote:
As you can see from April up until August the dumpsize increased
about 1.5 fold while the dump _time_ increased by a factor of
about 2.5. Dump rate dropped from 7.5 K/s to 4.6 K/s.
From August until now the dump size increased by about 10%
while dump time continues to grow dramatically.
I'll look into the "lots of small files added?" question.
Since your dumper and taper times are always nearly identical you probably
aren't using a holding disk and are dumping directly to tape. And since
the rates for one filesystem have remained constant while the other one has
dropped I would look into possible recent changes on the larger (slower)
filesystem. Try doing a dump to /dev/null and see how fast (or slow) that is.
If data isn't fed to the tape fast enough the tape drive has to stop and
reposition itself every time it runs out of data, which will slow it down
considerably.
Frank
Thanks,
Patrick
---
5. April 2002 15:29
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:54 1:53 0:00 (0:01 start, 0:00 idle)
Output Size (meg) 48155.548155.50.0
Original Size (meg) 48155.548155.50.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 24.1 24.10.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7284.9 7284.9--
Avg Tp Write Rate (k/s) 7282.8 7282.8--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1199200 1199200 -- 6:27 3100.76:28 3089.9
allianz01 c1t0d0s6 0 48112064 48112064 -- 106:22 7538.5 106:23 7537.7
1. Mai 2002 01:25
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:51 1:49 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 49187.349187.30.0
Original Size (meg) 49187.349187.30.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 24.6 24.60.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7700.0 7700.0--
Avg Tp Write Rate (k/s) 7698.5 7698.5--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1201184 1201184 -- 5:50 3427.75:52 3416.7
allianz01 c1t0d0s6 0 49166592 49166592 -- 103:11 7941.9 103:11 7941.6
1. Juni 2002 01:26
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:53 1:51 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 50107.250107.20.0
Original Size (meg) 50107.250107.20.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 25.1 25.10.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7674.4 7674.4--
Avg Tp Write Rate (k/s) 7672.9 7672.9--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1202656 1202656 -- 5:45 3487.65:46 3476.1
allianz01 c1t0d0s6 0 50107072 50107072 -- 105:41 7902.1 105:41 7901.9
1. Juli 2002 01:30
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:58 1:56 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 51813.251813.20.0
Original Size (meg) 51813.251813.20.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 25.9 25.90.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7640.7 7640.7--
Avg Tp Write Rate (k/s) 7639.2 7639.2--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1634400 1634400 -- 6:50 3985.36:51
Re: Performance degrading over time?
Hi all!
Jay Lessert wrote:
> So the first thing is to look at two AMANDA MAIL
> REPORT outputs: one from a "fast" run in June, and one from a "slow"
> run yesterday.
>
> Look at Estimate Time, Dump Time, Tape Time, Dump Rate, and Taper Rate
> for your one file system. Where is the increase happening?
>
> My personal bet is estimate time, and that your 6GB of growth is in the
> form of a large number of small files, but that is a wild guess.
The dump time is increasing way beyond the growth rate of data
to be dumped.
> If you give us actual data, we can do much better. :-)
See below.
As you can see from April up until August the dumpsize increased
about 1.5 fold while the dump _time_ increased by a factor of
about 2.5. Dump rate dropped from 7.5 K/s to 4.6 K/s.
>From August until now the dump size increased by about 10%
while dump time continues to grow dramatically.
I'll look into the "lots of small files added?" question.
Thanks,
Patrick
---
5. April 2002 15:29
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:54 1:53 0:00 (0:01 start, 0:00 idle)
Output Size (meg) 48155.548155.50.0
Original Size (meg) 48155.548155.50.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 24.1 24.10.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7284.9 7284.9--
Avg Tp Write Rate (k/s) 7282.8 7282.8--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1199200 1199200 -- 6:27 3100.76:28 3089.9
allianz01 c1t0d0s6 0 48112064 48112064 -- 106:22 7538.5 106:23 7537.7
1. Mai 2002 01:25
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:51 1:49 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 49187.349187.30.0
Original Size (meg) 49187.349187.30.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 24.6 24.60.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7700.0 7700.0--
Avg Tp Write Rate (k/s) 7698.5 7698.5--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1201184 1201184 -- 5:50 3427.75:52 3416.7
allianz01 c1t0d0s6 0 49166592 49166592 -- 103:11 7941.9 103:11 7941.6
1. Juni 2002 01:26
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:53 1:51 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 50107.250107.20.0
Original Size (meg) 50107.250107.20.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 25.1 25.10.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7674.4 7674.4--
Avg Tp Write Rate (k/s) 7672.9 7672.9--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1202656 1202656 -- 5:45 3487.65:46 3476.1
allianz01 c1t0d0s6 0 50107072 50107072 -- 105:41 7902.1 105:41 7901.9
1. Juli 2002 01:30
STATISTICS:
Total Full Daily
Dump Time (hrs:min)1:58 1:56 0:00 (0:02 start, 0:00 idle)
Output Size (meg) 51813.251813.20.0
Original Size (meg) 51813.251813.20.0
Avg Compressed Size (%) -- -- --
Tape Used (%) 25.9 25.90.0
Filesystems Dumped2 2 0
Avg Dump Rate (k/s) 7640.7 7640.7--
Avg Tp Write Rate (k/s) 7639.2 7639.2--
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--
allianz01 c0t0d0s0 0 1634400 1634400 -- 6:50 3985.36:51 3974.2
allianz01 c1t0d0s6
Re: Performance degrading over time?
Partially commenting to others replies too. On Mon, Oct 28, 2002 at 03:02:15PM +0100, Patrick M. Hausen wrote: > Hi all! > > We have a database server that is backed up daily using Amanda > to a dedicated tape drive. > > Sun E3500 > Sun E1000 RAID, ~170 GB of storage > LTO tape connected to LVD controller > > One and only application on this system is an Oracle database > server (8i). The tables are kept in one filesystem (UFS) and the > database is shut down nightly to accomodate for the Amanda run. > Despite of the dumps being "incremental", of course all the files > have changed during the day so we will get an "almost full dump". Any reason to not do full dumps only then. Might make estimate phase faster. > When I installed Amanda in June the nightly backup run took > about 4.5 hours. This has continuously increased to more than > 7 hours today - without any changes to hardware or configuration. > The size of the table files has increased from 72 to 78 GB > which doesn't quite qualify as the only reason ;-) Have you compared reports about which phase(s) are slower? > First thing I thought of was "UFS degradation when the FS is almost > full" - can't be, since the FS in question is around 170 GB and > thus only 45% full with the currrent tables. Fragmentation is much less a problem with ufs as compared with s5 fs. But still a problem. Unless defrag tools have been introduced, the standard way to defrag is ufsdump to tape, newfs (with same params) the partition, and ufsrestore. > Obviously with a single host connected to a single drive and > only so much storage we don't use a holding disk at the moment. Then might network, cpu, or spindle contention be a problem. Using software compression? In your case seems like hardware compression would be a good choice. Sorry Gene :) BTW, with Solaris 8, file system snapshots are possible even without the LVM. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Performance degrading over time?
Hi all! Andreas Baier wrote: > my answer seems to be a bit off-topic. but have you ever given a > Lgical Volume Manager a try - you could stop your oracle-instance, > issue the snapshot of the filesystem, startup your oracle and have > almost all-the time in the world for your backup while your users are > able to use the database with only a small performance impact. I do > this on a (smaller but 24/7) Oracle 8i database. But why - of course! Unfortunately the big expensive Sun/Oracle system was neither designed nor sold by us. We just "inherited" the system when problems started. The backup concept of the original seller/supporter was to use the internal DDS drive |-) So after we took over service and most of the budget already spent Amanda was an obvious solution - just add a "real" tape drive, they almost choked on the price of that already. We did offer volume management but for this component the price was prohibitive. Seems like we have a big sales opportunity now ;-)) > Fragmentation *might* be a cause for this performance ( 3,5 MB/s ) but > you would have noticed this with a poor performance of your > oracle-performance as-well. Well, the database is _only_ accessed by some web application. There are other limiting factors in this setup as far as responsiveness is concerned. I used to believe UFS wasn't prone to fragmentation as long as you keep your 10% reserve. Seems like very few very big files are sort of a pathological case. > What is about your network you told your performance in June was about > 4,5 MB/s - could there be a network-bottleneck? It's _local_ backup - no network communication involved. Well, OK, it _is_ a network socket and there will be quite a bit of context switching. 4,6 MB/s would be acceptable - it just keeps getting even slower. > Perhaps you´ll give the holdingdisk a try - you moght not regret. This is the first shot we will take - though I'm not very confident since the holding space will have to be on the same RAID in the same (only existing) partition as the database tables. I'll let you know the results. Thanks, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
Re: Performance degrading over time?
I'm not an expert in UFS, but could the drive be getting fragmented? Are you running with compression, and could the data in the database be getting more chaotic, thus increasing the size (as well as the time to compress) of the compressed file? --- "Patrick M. Hausen" <[EMAIL PROTECTED]> wrote: > Hi all! > > We have a database server that is backed up daily using Amanda > to a dedicated tape drive. > > Sun E3500 > Sun E1000 RAID, ~170 GB of storage > LTO tape connected to LVD controller > > One and only application on this system is an Oracle database > server (8i). The tables are kept in one filesystem (UFS) and the > database is shut down nightly to accomodate for the Amanda run. > Despite of the dumps being "incremental", of course all the files > have changed during the day so we will get an "almost full dump". > > When I installed Amanda in June the nightly backup run took > about 4.5 hours. This has continuously increased to more than > 7 hours today - without any changes to hardware or configuration. > The size of the table files has increased from 72 to 78 GB > which doesn't quite qualify as the only reason ;-) > > First thing I thought of was "UFS degradation when the FS is almost > full" - can't be, since the FS in question is around 170 GB and > thus only 45% full with the currrent tables. > > Obviously with a single host connected to a single drive and > only so much storage we don't use a holding disk at the moment. > > > Any ideas where to start searching? > > Thanks, > Patrick M. Hausen > Technical Director > -- > punkt.de GmbH Internet - Dienstleistungen - Beratung > Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 > 76135 Karlsruhe http://punkt.de > > > = --- Wayne Johnson, | There are two kinds of people: Those 3943 Penn Ave. N. | who say to God, "Thy will be done," Minneapolis, MN 55412-1908 | and those to whom God says, "All right, (612) 522-7003 | then, have it your way." --C.S. Lewis __ Do you Yahoo!? Y! Web Hosting - Let the expert host your web site http://webhosting.yahoo.com/
Performance degrading over time?
Hi all! We have a database server that is backed up daily using Amanda to a dedicated tape drive. Sun E3500 Sun E1000 RAID, ~170 GB of storage LTO tape connected to LVD controller One and only application on this system is an Oracle database server (8i). The tables are kept in one filesystem (UFS) and the database is shut down nightly to accomodate for the Amanda run. Despite of the dumps being "incremental", of course all the files have changed during the day so we will get an "almost full dump". When I installed Amanda in June the nightly backup run took about 4.5 hours. This has continuously increased to more than 7 hours today - without any changes to hardware or configuration. The size of the table files has increased from 72 to 78 GB which doesn't quite qualify as the only reason ;-) First thing I thought of was "UFS degradation when the FS is almost full" - can't be, since the FS in question is around 170 GB and thus only 45% full with the currrent tables. Obviously with a single host connected to a single drive and only so much storage we don't use a holding disk at the moment. Any ideas where to start searching? Thanks, Patrick M. Hausen Technical Director -- punkt.de GmbH Internet - Dienstleistungen - Beratung Scheffelstr. 17 a Tel. 0721 9109 -0 Fax: -100 76135 Karlsruhe http://punkt.de
