Re: Out of space problem
Op di 06mei08 om 10:28 schreef Paul Bijnens:
Or better, use amtapetape -e 35g /dev/yourtapedrive to get
an estimate (takes about 3-5 hours normally).
I got/use:
define tapetype HP-DAT72 {
comment HP-DAT72x10 Autoloader (AE313A)
# data provided by Gerrit A. Smit [EMAIL PROTECTED]
length 35480 mbytes
filemark 0 kbytes
speed 2992 kbytes
}
Re: Out of space problem
Thanks to everyone for the input - both serious and sarcastic.
Happy to report that hardware compression was enabled. Turned it off and
got a level 0 done last night.
Muy Gracias
N/
On 6/05/2008 10:30 AM, Nigel Allen wrote:
Hi All
I'm experiencing an odd problem with a USB DAT drive that keeps
running out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
Some dumps may have been left in the holding disk.
Run amflush to flush them to tape.
The next tape Amanda expects to use is: DailySet1-19.
FAILURE AND STRANGE DUMP SUMMARY:
mail..com.au mapper/VolGroup00-LogVol00 lev 0 FAILED
[out of tape]
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min)0:04
Run Time (hrs:min)14:44
Dump Time (hrs:min) 11:22 11:22 0:00
Output Size (meg) 33927.133927.10.0
Original Size (meg) 50710.150710.00.0
Avg Compressed Size (%)66.9 66.9 10.0 (level:#disks
...)
Filesystems Dumped2 1 1 (1:1)
Avg Dump Rate (k/s) 849.1 849.18.5
Tape Time (hrs:min)0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%) 0.00.00.0 (level:#disks
...)
Filesystems Taped 1 0 1 (1:1)
Chunks Taped 0 0 0
Avg Tp Write Rate (k/s) 7.4-- 7.4
USAGE BY TAPE:
Label Time Size %NbNc
DailySet1-18 0:00 32K0.0 1 0
NOTES:
planner: Forcing full dump of
mail..com.au:mapper/VolGroup00-LogVol00 as directed.
taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space
left on device
driver: Taper error: [writing file: No space left on device]
driver: going into degraded mode because of taper component error.
big estimate: mail..com.au sda1 1
est: 64Kout 32K
Here is the amanda.conf file with everything not used snipped out:
org DailySet1 # your organization name for reports
mailto amandabackup # space separated list of operators at your site
dumpuser amandabackup # the user to run dumps under
inparallel 4# maximum dumpers that will run in parallel
(max 63)
dumporder sssS# specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump
image to send
displayunit k # Possible values: k|m|g|t
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per
sec
dumpcycle 4 weeks # the number of days in the normal dump cycle
runspercycle 20 # the number of amdump runs in dumpcycle days
tapecycle 21 tapes # the number of tapes in rotation
bumpsize 20 Mb # minimum savings (threshold) to bump level 1
- 2
bumppercent 20 # minimum savings (threshold) to bump level 1
- 2
bumpdays 1 # minimum days at each level
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
etimeout 300# number of seconds per filesystem for
estimates.
dtimeout 1800 # number of idle seconds before a dump is
aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
tapebufs 20
runtapes 1 # number of tapes to be used in a single run
of amdump
tapedev /dev/nst0 # the no-rewind tape device to be used
changerfile /etc/amanda/DailySet1/changer.conf
changerdev /dev/null
maxdumpsize -1 # Maximum number of bytes the planner
will schedule
tapetype HP-DAT72 # what kind of tape it is (see
tapetypes below)
labelstr ^DailySet1-[0-9][0-9]*$ # label constraint regex: all
tapes must match
amrecover_do_fsf yes# amrecover will call amrestore with the
amrecover_check_label yes # amrecover will call amrestore with the
amrecover_changer null: # amrecover will use the changer if
you restore
holdingdisk hd1 {
comment main holding disk
directory /dumps/amanda # where the holding disk is
use -10 Gb # how much space can we use on it
chunksize 1Gb # size of chunk if you want big dump to be
}
autoflush no #
infofile /etc/amanda/DailySet1/curinfo# database DIRECTORY
logdir /etc/amanda/DailySet1# log directory
indexdir /etc/amanda/DailySet1/index # index directory
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
define dumptype global {
comment Global definitions
}
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
Re: Out of space problem
Hi Nigel!
Nigel Allen wrote:
I'm experiencing an odd problem with a USB DAT drive that keeps
running out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
...
Output Size (meg) 33927.133927.10.0
Original Size (meg) 50710.150710.00.0
Avg Compressed Size (%)66.9 66.9 10.0 (level:#disks
...)
...
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
...
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
compress client custom
client_custom_compress /usr/bin/bzip2
}
...
mail.airsolutions.com.aumapper/VolGroup00-LogVol00
custom-compress
mail.airsolutions.com.ausda1custom-compress
Hardware Software - compression together is allways a bad combination!
bzipped 34G would fit on d 36G tape but the hardware-commpression of the
tape will blow them up to much more so you run out of tape.
Disable hardware-compression - see howto's and wikis - or dont compress
your data
and rely on hardware-compression only (not recommended).
Bye, Peter
WOTLmade
Re: Out of space problem
Hi,
Nigel Allen schrieb:
Hi All
I'm experiencing an odd problem with a USB DAT drive that keeps running
out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Ok this means amanda can put 36 GB of data after softwarecompression on it,
forget the 72GB value, its marketing.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
Some dumps may have been left in the holding disk.
Run amflush to flush them to tape.
The next tape Amanda expects to use is: DailySet1-19.
FAILURE AND STRANGE DUMP SUMMARY:
mail..com.au mapper/VolGroup00-LogVol00 lev 0 FAILED [out
of tape]
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min)0:04
Run Time (hrs:min)14:44
Dump Time (hrs:min) 11:22 11:22 0:00
Output Size (meg) 33927.133927.10.0
Original Size (meg) 50710.150710.00.0
Avg Compressed Size (%)66.9 66.9 10.0 (level:#disks
...)
Filesystems Dumped2 1 1 (1:1)
Avg Dump Rate (k/s) 849.1 849.18.5
Tape Time (hrs:min)0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%) 0.00.00.0 (level:#disks
...)
Filesystems Taped 1 0 1 (1:1)
Chunks Taped 0 0 0
Avg Tp Write Rate (k/s) 7.4-- 7.4
USAGE BY TAPE:
Label Time Size %NbNc
DailySet1-18 0:00 32K0.0 1 0
NOTES:
planner: Forcing full dump of
mail..com.au:mapper/VolGroup00-LogVol00 as directed.
taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space
ok, according to this, it wrote about 30 GB to the tape,
then it ran into EOT
this smells to me as if you have hardwarecompression enabled and are doing
softwarecompression.
That is s BAD idea, as it will lead to what you see, a loss of roughly
6 GB of tape-space, as the already compressed dump-image will EXPAND
by the hardwarecompression.
So first disable hardwarecompression of your Tapedrive, then try again.
And it tells you, you'll have to splitt your dumps in chunks using tar
and exclude-/includelists, as the level 0's are to big for your tapes.
another solution could be using a Tapechanger and tape-spanning.
left on device
driver: Taper error: [writing file: No space left on device]
driver: going into degraded mode because of taper component error.
big estimate: mail..com.au sda1 1
est: 64Kout 32K
Here is the amanda.conf file with everything not used snipped out:
org DailySet1 # your organization name for reports
mailto amandabackup # space separated list of operators at your site
dumpuser amandabackup # the user to run dumps under
inparallel 4# maximum dumpers that will run in parallel
(max 63)
dumporder sssS# specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump
image to send
displayunit k # Possible values: k|m|g|t
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
dumpcycle 4 weeks # the number of days in the normal dump cycle
runspercycle 20 # the number of amdump runs in dumpcycle days
i would lower this, as it tells amanda:
do a level 0 of all disklist-entrys at least all 20 days.
have a nice day
Christoph
Re: Out of space problem
On Tue, 6 May 2008, Nigel Allen wrote:
I'm experiencing an odd problem with a USB DAT drive that keeps running out of
space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
GB or GiB?
Total Full Incr.
Estimate Time (hrs:min)0:04
Run Time (hrs:min)14:44
Dump Time (hrs:min) 11:22 11:22 0:00
Output Size (meg) 33927.133927.10.0
These are in KiB.
Original Size (meg) 50710.150710.00.0
Do you have hardware compression enabled on the drive?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say programmer or something like that.
-- Linus Torvalds
Re: Out of space problem
On 2008-05-06 02:30, Nigel Allen wrote:
I'm experiencing an odd problem with a USB DAT drive that keeps running
out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Ok. The meaning of those numbers is:
- around 36 gigabytes (in powers of 1000 not in powers of 1024) space
to put the bytes on.
- in the optimistic premise that the internal compressor can achieve
a reduction by 2, you can feed 72 GB data, which will fit in the
36 GB raw space on the tape.
You must understand that the compression is an algorithm, just like
gzip or bzip2 (actually a less optimised version of the older compress
program). The tape drive does NOT double the capacity by writing
the bits in higher density, or with double number of tracks or so.
In real life this means that those tapes can hold about 33 Gibyte
(in base 2^10=1024). This is the number that Amanda needs.
See http://en.wikipedia.org/wiki/SI_prefix#Computing for the difference
between GB and GiB. (Amanda uses the MiB, GiB, but writes it as MB, GB, etc)
Then also realize that the compressor built into the those DAT drives
actually EXPANDS data that is not compressable by about 20%. (Contrary
to the compressor built into the more modern LTO drives, which do not do
that.)
When feeding the tape drive with already compressed data (you use bzip2
already), and you use hardware compression, then you loose again
tapespace.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
[...]
taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space
left on device
driver: Taper error: [writing file: No space left on device]
The tape drive hit EOT after 1022240 KiB = 29.5 GiB.
Therefore I believe you have both hardware + software compression
enabled, resulting in a loss of about 3 GiB of space.
[...]
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
Especially when using software compression, use the native
capacity of the tape in the definition: Try with 34 GiB.
Or better, use amtapetape -e 35g /dev/yourtapedrive to get
an estimate (takes about 3-5 hours normally).
[...]
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
compress client custom
client_custom_compress /usr/bin/bzip2
}
When using bzip2, the hardware compressor can never compete with
that (except in speed). So better disable hardware compression.
mail.airsolutions.com.aumapper/VolGroup00-LogVol00
custom-compress
mail.airsolutions.com.ausda1custom-compress
When using gnutar, why do you insist on specifying a device name.
Why not use the mount point itself. (Amanda will do the conversion
for you, but why make it so obscure.)
Any idea where I can start would be appreciated (apart from bigger
tape or less data).
Another improvement is to break up the large dataset into
smaller ones, so that each one dump is much smaller than a the
capacity of your tape. That way, Amanda can better spread the dumps
over different tapes/days. See:
http://wiki.zmanda.com/index.php/How_To:Split_DLEs_With_Exclude_Lists
--
Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512
http://www.xplanation.com/ email: [EMAIL PROTECTED]
***
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... Are you sure? ... YES ... Phew ... I'm out *
***
Re: Out of space problem
T, 06 mai 2008 kirjutas Nigel Allen [EMAIL PROTECTED]:
I'm experiencing an odd problem with a USB DAT drive that keeps running
out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
[snip]
STATISTICS:
Output Size (meg) 33927.133927.10.0
[snip]
[snip]
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
[snip]
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
compress client custom
client_custom_compress /usr/bin/bzip2
}
[snip]
mail.airsolutions.com.aumapper/VolGroup00-LogVol00
custom-compress
mail.airsolutions.com.ausda1custom-compress
Any idea where I can start would be appreciated (apart from bigger
tape or less data).
Start by not believing that your 36/72 GB tapedrive can actually
record 72 GB :)
Continue by turning off hardware compression on the tape drive. As
your configuration indicates, you are using software compression. If
this compressed data is then sent to the tapedrive, it gets fed
through the hardware compression algorithm, but as the input data is
already compressed, the resulting hardware compressed data is
actually larger than the input.
Additionally, you would very likely benefit from splitting your
disklist into smaller entries, because your data size is pretty close
to what your tape can hold. By using more DLEs, Amanda has a better
chance to spread out the work throughout the dumpcycle and not
overfilling the tape.
--
Toomas Aas
Re: Out of space problem
On Tue, May 06, 2008 at 10:30:55AM +1000, Nigel Allen wrote: Hi All I'm experiencing an odd problem with a USB DAT drive that keeps running out of space. Apologies for the length of the post. The drive is supposed to be 36 / 72 GB. It is 36 GB. Maybe minus a bit for marketing liesexaggeration. But you are compressing the data with bzip2 then telling the hardware to try to compress it further. The hardware compressor actually expands already compressed data by 5-15%. Thus your tape can only hold about 31 GB of this type of processed data. And guess what? NOTES: planner: Forcing full dump of mail..com.au:mapper/VolGroup00-LogVol00 as directed. taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space left on ^^^ Amanda runs out of tape at about 31 GB. Any idea where I can start would be appreciated (apart from bigger tape or less data). See, you knew the solution all on your own. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 12027 Creekbend Drive (703) 787-0884 Reston, VA 20194 (703) 787-0922 (fax)
Out of space problem
Hi All
I'm experiencing an odd problem with a USB DAT drive that keeps running
out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
Some dumps may have been left in the holding disk.
Run amflush to flush them to tape.
The next tape Amanda expects to use is: DailySet1-19.
FAILURE AND STRANGE DUMP SUMMARY:
mail..com.au mapper/VolGroup00-LogVol00 lev 0 FAILED [out of tape]
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min)0:04
Run Time (hrs:min)14:44
Dump Time (hrs:min) 11:22 11:22 0:00
Output Size (meg) 33927.133927.10.0
Original Size (meg) 50710.150710.00.0
Avg Compressed Size (%)66.9 66.9 10.0 (level:#disks ...)
Filesystems Dumped2 1 1 (1:1)
Avg Dump Rate (k/s) 849.1 849.18.5
Tape Time (hrs:min)0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%) 0.00.00.0 (level:#disks ...)
Filesystems Taped 1 0 1 (1:1)
Chunks Taped 0 0 0
Avg Tp Write Rate (k/s) 7.4-- 7.4
USAGE BY TAPE:
Label Time Size %NbNc
DailySet1-18 0:00 32K0.0 1 0
NOTES:
planner: Forcing full dump of mail..com.au:mapper/VolGroup00-LogVol00
as directed.
taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space left on
device
driver: Taper error: [writing file: No space left on device]
driver: going into degraded mode because of taper component error.
big estimate: mail..com.au sda1 1
est: 64Kout 32K
Here is the amanda.conf file with everything not used snipped out:
org DailySet1 # your organization name for reports
mailto amandabackup # space separated list of operators at your site
dumpuser amandabackup # the user to run dumps under
inparallel 4# maximum dumpers that will run in parallel
(max 63)
dumporder sssS# specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump
image to send
displayunit k # Possible values: k|m|g|t
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
dumpcycle 4 weeks # the number of days in the normal dump cycle
runspercycle 20 # the number of amdump runs in dumpcycle days
tapecycle 21 tapes # the number of tapes in rotation
bumpsize 20 Mb # minimum savings (threshold) to bump level 1 - 2
bumppercent 20 # minimum savings (threshold) to bump level 1 - 2
bumpdays 1 # minimum days at each level
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
etimeout 300# number of seconds per filesystem for estimates.
dtimeout 1800 # number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
tapebufs 20
runtapes 1 # number of tapes to be used in a single run
of amdump
tapedev /dev/nst0 # the no-rewind tape device to be used
changerfile /etc/amanda/DailySet1/changer.conf
changerdev /dev/null
maxdumpsize -1 # Maximum number of bytes the planner
will schedule
tapetype HP-DAT72 # what kind of tape it is (see
tapetypes below)
labelstr ^DailySet1-[0-9][0-9]*$ # label constraint regex: all
tapes must match
amrecover_do_fsf yes# amrecover will call amrestore with the
amrecover_check_label yes # amrecover will call amrestore with the
amrecover_changer null: # amrecover will use the changer if
you restore
holdingdisk hd1 {
comment main holding disk
directory /dumps/amanda # where the holding disk is
use -10 Gb # how much space can we use on it
chunksize 1Gb # size of chunk if you want big dump to be
}
autoflush no #
infofile /etc/amanda/DailySet1/curinfo# database DIRECTORY
logdir /etc/amanda/DailySet1# log directory
indexdir /etc/amanda/DailySet1/index # index directory
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
define dumptype global {
comment Global definitions
}
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
compress client custom
client_custom_compress /usr/bin/bzip2
}
define interface local {
comment a local disk
use 1000 kbps
}
define interface le0 {
comment 10 Mbps ethernet
use 400 kbps
}
and finally the disklist:
