Re: Backing up PostgreSQL?
On Thu, Jun 13, 2002 at 04:57:16PM -0500, Kirk Strauser wrote: At 2002-06-13T21:26:22Z, Ragnar Kjørstad [EMAIL PROTECTED] writes: The easiest way to snapshot the filesystem is to use a logical volume manager (LVM or EVMS on linux) and then do: 1. take database offline 2. take snapshot 3. take database online 4. backup from snapshot 5. remove snapshot I'm on FreeBSD-STABLE right now, so that's unfortunately not an option at the moment. I'm interested that you include steps #1 and #3, though. On FreeBSD-CURRENT, the snapshot is atomic. There wouldn't be a need to stop or restart any services. Is it different on Linux? snapshot is atomic on linux as well, but by shutting the database down first you allow postgresql to shut down cleanly. If fsync is enabled and so on you should still be able to start with a consistant database, so I suppose it's not strictly required. -- Ragnar Kjørstad Big Storage
Re: How to Configure Tapeless Backup in 2.4.3
Anyone have a .conf for using the new tapeless backup techniques
introduced in 2.4.3? I too am stuck; my best guess is this:
There are 2 ways to do tapeless backup:
(a) The 2.4.2 and below way (just uses holding disk)
(b) the 2.4.3 way (if anyone has this working please post your .conf!)
(no holding disk needed I guess; uses new file driver, somehow)
Way (a) seems to be well described, and oughta works with 2.4.3 and up
The FAQ http://amanda.sourceforge.net/cgi-bin/fom?
_highlightWords=tapelessfile=191 states...
...I've set up a system using 2.4.2p2 to backup to nfs mounted drive
space just recently. If you are able to use 2.4.3, there is a
setting to save to disk. Check the docs or the mail archives...
I have checked the docs and the mail archives and have had no joy.
I'd would love to use the whiz-bang new 2.4.3 technique but
despite the optimistic language in the FAQ, the docs and the mail
archives do not adequately describe technique (b). Following are
2 long posts. The first is the somewhat sparse documentation in
the 2.4.3 docs. It may soon be time to give up on (b) and surrender
to the adequately documented (a) technique. People won't use new
features if they are not documented enough. The second long posting
details one man's use of technique (a).
We should all stop confusing the 2 different tapeless approaches.
And if anyone is successfully using the new 2.4.3 (b) technique,
could you post your .conf file? The helpful descriptions of the (a)
technique are masking the need for fuller documentation of the (b)
technique. I tried something like Mohamed in his earlier post with
the same bad results.
-Steve
New in 2.4.3 man page... but hard to grok... can't make it work
(See Mohamed's posting)...
OUTPUT DRIVERS
The normal value for the tapedev parameter, or for what a
tape changer returns, is a full path name to a non-rewind-
ing tape device, such as /dev/nst0 or /dev/rmt/0mn or
/dev/nst0.1 or whatever conventions the operating system
uses.Amanda provides additional application level
drivers that support non-tradition tape simulatation or
features. To access a specific output driver, set tapedev
(or configure your changer to return) a string of the form
driver:driver-info where driver is one of the supported
drivers and driver-info is optional additional information
needed by the driver.
The supported drivers are:
tape This is the default driver. The driver-info is
the tape device name. Entering /dev/rmt/0mn is
really a short hand for tape:/dev/rmt/0mn.
null This driver throws away anything written to it
and returns EOF for any reads except a special
case is made for reading a label, in which case
a fake value is returned that Amanda checks
for and allows through regardless of what you
have set in labelstr. The driver-info field is
not used and may be left blank:
tapedev null:
The length value from the associated tapetype is
used to limit the amount of data written. When
the limit is reached, the driver will simulate
end of tape.
NOTE: this driver should only be used for debug-
ging and testing, and probably only with the
record option set to no.
rait Redundant Array of Inexpensive (?)Tapes.
Reads and writes tapes mounted on multiple
drives by spreading the data across N-1 drives
and using the last drive for a checksum. See
docs/RAIT for more information.
The driver-info field describes the devices to
use. Curly braces indicate multiple replace-
ments in the string. For instance:
tapedev rait:/dev/rmt/tps0d{4,5,6}n
would use the following devices:
/dev/rmt/tps0d4n
/dev/rmt/tps0d5n
/dev/rmt/tps0d6n
file This driver emulates a tape device with a set of
files in a directory. The driver-info field
must be the name of an existing directory. The
driver will test for a subdirectory of that
named data and return offline until it is pre-
sent. When present, the driver uses two files
in the data subdirectory for each tape file.
One contains the actual data. The other con-
tains record length information.
The
Re: Backing up PostgreSQL?
On Thu, Jun 13, 2002 at 05:33:31PM -0500, Brandon D. Valentine wrote: The easiest way to snapshot the filesystem is to use a logical volume manager (LVM or EVMS on linux) and then do: 1. take database offline 2. take snapshot 3. take database online 4. backup from snapshot 5. remove snapshot I would like to comment that while this is a possible way to backup your database, it's not the way I would recommend going about it. There are a couple of caveats: 1) In order to take a filesystem snapshot you must have enough diskspace elsewhere to contain the filesystem snapshot. If your database resides on a large filesystem with other data[0] then you're unlikely to want to deal with caching an entire snapshot until amanda backs it up. You need extra space with both approaches (pg_dump and snapshot). Which solution requires the most space will depend on many factors, e.g. how much you write you your database. If it's mostly read-only, the snapshot will not require much space. 2) Backing up a database by grabbing the actual files off of the disk can introduce problems if trying to restore onto, for instance, an upgraded version of Postgres, which might have changed the ondisk representation slightly. There are also problems if you migrate to a different architecture since things like byte ordering can change. By generating a database dump with pg_dump you insure that you have a portable, plain text file full of valid query commands which can be read into any future version of Postgres and possibly even into other RDMBS products provided you choose a pg_dump format which is standards complaint enough. Yes, this is a backup-solution, not a migration-solution. pg_dumps can not always be imported into newer postgresql versions without modifications. 3) If your snapshot code is not atomic it means you must take your database server down everytime you make the snapshot, which on a large filesystem could be a non-trivial amount of time. With pg_dump you're just dumping the tables via the standard Postgres interface so you've got no issues with doing it on a running database. Hmm, is the pg_dump consistent? IOW, is it done in a single transaction? (even for multiple databases?) If yes, would a very long-running pg_dump not cause problems for the running server? I know postgresql doesn't lock whole tables, but it means that if data changes postgresql needs to keep two branches, and it will require extra diskspace and I suppose also introduce overhead to other processes? Of course, if the database is mostly read-only this is just a minor problem, but that's true for snapshot-backups as well. -- Ragnar Kjørstad Big Storage
remote backup still fails
Hi guys,
I still haven't solved this issue.. I hope you have an idea as to what
is wrong.
My guess, is that the following error message is the reason - but what
does it mean?
FAIL taper compaq.vsen.dk / 20020612 0 [input: can't open: ¼S^F: No such
file or director
y]
FAIL driver compaq.vsen.dk / 20020612 0 [dump to tape failed]
My disklist only contain:
localhost / always-full
compaq.vsen.dk / always-full
p.s. compaq.vsen.dk is in my hosts file - on both machines.
On Wed, 2002-06-12 at 19:00, klavs klavsen wrote:
Hi guys,
I'm setting up Amanda-2.4.3b3.
I've set it to log to my disk - and it seems to backup my localhost
fine.
If I run amcheck, it returns all fine - also on remote host.
I've put my amdump.1 log in the bottom - sorry I had to include it all,
but I didn't know what was important.
heres what I get in my backup-log:
DISK planner localhost /
DISK planner compaq.vsen.dk /
START driver date 20020612
START planner date 20020612
WARNING planner Last full dump of localhost:/ on tape DiskBackup11
overwritten on this run.
WARNING planner Last full dump of compaq.vsen.dk:/ on tape overwritten
in 1 run.
START taper datestamp 20020612 label DiskBackup11 tape 0
INFO planner Incremental of localhost:/ bumped to level 2.
FINISH planner date 20020612
STATS driver startup time 735.308
STRANGE dumper localhost / 0 [sec 995.448 kb 3039930 kps 3053.8 orig-kb
3039930]
sendbackup: start [localhost:/ level 0]
sendbackup: info BACKUP=/bin/tar
sendbackup: info RECOVER_CMD=/bin/tar -f... -
sendbackup: info end
| gtar: ./dev/gpmctl: socket ignored
[SNIP more socket ignored msgs]
| gtar: ./var/prelude/socket: socket ignored
| Total bytes written: 3112888320 (2.9GB, 3.0MB/s)
sendbackup: size 3039930
sendbackup: end
SUCCESS taper localhost / 20020612 0 [sec 995.770 kb 3039968 kps 3052.9
{wr: writers 94999 rdwait 956.222 wrwait 38.908 filemark 0.043}]
FAIL taper compaq.vsen.dk / 20020612 0 [input: can't open: ¼S^F: No such
file or director
y]
FAIL driver compaq.vsen.dk / 20020612 0 [dump to tape failed]
INFO taper tape DiskBackup11 kb 3039968 fm 2 [OK]
FINISH driver date 20020612 time 1921.265
amdump log:
amdump: start at Wed Jun 12 14:06:41 CEST 2002
amdump: datestamp 20020612
planner: pid 27386 executable /usr/lib/amanda/planner version 2.4.3b3
planner: build: VERSION=Amanda-2.4.3b3
planner:BUILT_DATE=Wed Jun 5 21:23:29 CEST 2002
planner:BUILT_MACH=Linux amd.vsen.dk 2.4.18 #1 Sat May 18
01:53:27 CEST 2002 i686 unknown
planner:CC=i586-mandrake-linux-gnu-gcc
planner:CONFIGURE_COMMAND='./configure' '--prefix=/usr'
'--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
'--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include'
'--libdir=/usr/lib' '--libexecdir=/usr/lib/amanda'
'--localstatedir=/var/lib/lib' '--sharedstatedir=/usr/com'
'--mandir=/usr/share/man' '--host=i586-mandrake-linux-gnu'
'--infodir=/usr/share/info' '--enable-shared'
'--with-index-server=localhost'
'--with-gnutar-listdir=/var/lib/lib/amanda/gnutar-lists'
'--with-smbclient=/usr/bin/smbclient' '--with-amandahosts'
'--with-user=amanda' '--with-group=disk' '--with-gnutar=/bin/tar'
planner: paths: bindir=/usr/bin sbindir=/usr/sbin
[SNIP planner: dirstuff]
planner: defs: DEFAULT_SERVER=localhost DEFAULT_CONFIG=DailySet1
planner:DEFAULT_TAPE_SERVER=localhost
planner:DEFAULT_TAPE_DEVICE=/dev/null HAVE_MMAP HAVE_SYSVSHM
planner:LOCKING=POSIX_FCNTL SETPGRP_VOID DEBUG_CODE
planner:AMANDA_DEBUG_DAYS=4 BSD_SECURITY USE_AMANDAHOSTS
planner:CLIENT_LOGIN=amanda FORCE_USERID HAVE_GZIP
planner:COMPRESS_SUFFIX=.gz COMPRESS_FAST_OPT=--fast
planner:COMPRESS_BEST_OPT=--best UNCOMPRESS_OPT=-dc
planner: dgram_bind: socket bound to 0.0.0.0.587
READING CONF FILES...
driver: pid 27387 executable /usr/lib/amanda/driver version 2.4.3b3
driver: send-cmd time 0.006 to taper: START-TAPER 20020612
startup took 0.022 secs
SETTING UP FOR ESTIMATES...
setting up estimates for localhost:/
localhost:/ overdue 1 day for level 0
setup_estimate: localhost:/: command 0, options:
last_level 1 next_level0 -1 level_days 1
getting estimates 0 (3047110) 1 (52140) 2 (0)
setting up estimates for compaq.vsen.dk:/
compaq.vsen.dk:/ overdue 11851 days for level 0
setup_estimate: compaq.vsen.dk:/: command 0, options:
last_level 0 next_level0 -11851 level_days 0
getting estimates 0 (-1) 1 (0) -1 (-1)
setting up estimates took 0.001 secs
GETTING ESTIMATES...
driver: started dumper0 pid 27390
driver: started dumper1 pid 27391
driver: started dumper2 pid 27392
driver: started dumper3 pid 27395
taper: pid 27388 executable taper version 2.4.3b3
taper: page size is 4096
taper: buffer size is 32768
taper: buffer[00] at 0x40265000
[SNIP more buffer stuff]
taper: buffer[19] at 0x402fd000
taper: buffer structures at 0x40305000 for 240 bytes
taper:
RE: ./var/adm not being excluded
It worked! Thank you very much.
It looks like there may be a bug in the exclude files... directive,
then, because no matter how I worked it, it wasn't working.
Michael Martinez
System Administrator (Contractor)
Information Systems and Technology Management
CSREES - United States Department of Agriculture
(202) 720-6223
-Original Message-
From: Gene Heskett [mailto:[EMAIL PROTECTED]]
Sent: Thursday, June 13, 2002 6:08 PM
To: Martinez, Michael - CSREES/ISTM; 'Jon LaBadie';
'[EMAIL PROTECTED]'
Subject: Re: ./var/adm not being excluded
On Thursday 13 June 2002 11:50, Martinez, Michael - CSREES/ISTM wrote:
Tried excluding the entire directory --- no go.
Another question for ya: if I use the exclude list option, does my
list file need to be in an amanda directory (var/amanda) or can it be
anywhere? My Sco Unix installation has amanda installed in
/usr/local. On the other hand, my amanda server has amanda in
/var/lib
The exclude file can be anyplace that the line in *your* backup config
(define dumptype etc) description in amanda.conf tells amanda to look.
By using that mechanism, one can have different exclude lists for
different backup styles or machines. Typically, it might look like
this snippet from amanda.conf:
define dumptype user-tar {
root-tar
comment user partitions dumped with tar
exclude list /usr/local/etc/amanda/DailySet1/exclude.gtar
compress server best
priority medium
}
Michael Martinez
System Administrator (Contractor)
Information Systems and Technology Management
CSREES - United States Department of Agriculture
(202) 720-6223
-Original Message-
From: Jon LaBadie [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 12, 2002 9:30 PM
To: '[EMAIL PROTECTED]'
Subject: Re: ./var/adm not being excluded
On Wed, Jun 12, 2002 at 11:32:47AM -0400, Martinez, Michael -
CSREES/ISTM
wrote:
I had posted this msg before, but I had given the wrong info.
I'm backing up / on one of my clients. I would like to exclude
/var/adm/syslog from the backup.
On my server:amanda.conf, I have defined a disklist entry with
exclude ./var/adm/syslog and I'm using this in my disklist file.
But it's not working. tar continues to give me an error (actually a
failure)
about /var/adm/syslog: changes as we read it.
It should not be a fatal error. I'm not sure why that should be anything
but an info advisory. FWIW, I don't see any messages about that in my
reports here, or at home. Do you have another cron job running
something that would cause this logfile to be appended to while amdump
is running? Amanda would rather have the run of the machine if it can.
Now, I know this isn't much of a problem. But since my Amanda report
gets mailed out every day to the IT director and the project
manager, I'd like
to
NUKE this FAILURE message, cuz they think the backup is failing.
Do you need to backup anything under /var/adm?
Maybe exclude the entire directory.
The entrys in the exclude file should be in the form of a relative to .
style, as in if you want to skip that directory tree, then the file
should contain a line that reads
./adm
and that whole tree will, or should be skipped. Of course it follows
that an identically named directory someplace else in the system will
also be skipped when using this relative syntax.
--
Cheers, Gene
CE @ WDTV-5, CBS for N. Central WV
Re: How to Configure Tapeless Backup in 2.4.3
i am using the 2.4.3 method, and i find it to be the most efficient. The only relevant line in the amanda.conf is the following: tapedev=file:/var/spool/whateveryourbackupdirectoryis After that, configure the rest as you would anything else, making a tape definition according to how much space you would need for the night's backups. To rotate 'tapes', i recommend creating sub-directories and symbolically linking them to a 'data' subdirectory, which amlabel et al will look for. The directories are labelled like tapes. I name the subdirectories according to date, then i have a shell script which creates a new directory every night, makes the symlink, labels the tape then runs the backup. works great! Amanda is wonderfull!! cheers, john. Steve Follmer wrote: Anyone have a .conf for using the new tapeless backup techniques introduced in 2.4.3? I too am stuck; my best guess is this: There are 2 ways to do tapeless backup: (a) The 2.4.2 and below way (just uses holding disk) (b) the 2.4.3 way (if anyone has this working please post your .conf!) (no holding disk needed I guess; uses new file driver, somehow) Way (a) seems to be well described, and oughta works with 2.4.3 and up The FAQ http://amanda.sourceforge.net/cgi-bin/fom? _highlightWords=tapelessfile=191 states... ...I've set up a system using 2.4.2p2 to backup to nfs mounted drive space just recently. If you are able to use 2.4.3, there is a setting to save to disk. Check the docs or the mail archives... I have checked the docs and the mail archives and have had no joy. I'd would love to use the whiz-bang new 2.4.3 technique but despite the optimistic language in the FAQ, the docs and the mail archives do not adequately describe technique (b). Following are 2 long posts. The first is the somewhat sparse documentation in the 2.4.3 docs. It may soon be time to give up on (b) and surrender to the adequately documented (a) technique. People won't use new features if they are not documented enough. The second long posting details one man's use of technique (a). We should all stop confusing the 2 different tapeless approaches. And if anyone is successfully using the new 2.4.3 (b) technique, could you post your .conf file? The helpful descriptions of the (a) technique are masking the need for fuller documentation of the (b) technique. I tried something like Mohamed in his earlier post with the same bad results. -Steve New in 2.4.3 man page... but hard to grok... can't make it work (See Mohamed's posting)... OUTPUT DRIVERS The normal value for the tapedev parameter, or for what a tape changer returns, is a full path name to a non-rewind- ing tape device, such as /dev/nst0 or /dev/rmt/0mn or /dev/nst0.1 or whatever conventions the operating system uses.Amanda provides additional application level drivers that support non-tradition tape simulatation or features. To access a specific output driver, set tapedev (or configure your changer to return) a string of the form driver:driver-info where driver is one of the supported drivers and driver-info is optional additional information needed by the driver. The supported drivers are: tape This is the default driver. The driver-info is the tape device name. Entering /dev/rmt/0mn is really a short hand for tape:/dev/rmt/0mn. null This driver throws away anything written to it and returns EOF for any reads except a special case is made for reading a label, in which case a fake value is returned that Amanda checks for and allows through regardless of what you have set in labelstr. The driver-info field is not used and may be left blank: tapedev null: The length value from the associated tapetype is used to limit the amount of data written. When the limit is reached, the driver will simulate end of tape. NOTE: this driver should only be used for debug- ging and testing, and probably only with the record option set to no. rait Redundant Array of Inexpensive (?)Tapes. Reads and writes tapes mounted on multiple drives by spreading the data across N-1 drives and using the last drive for a checksum. See docs/RAIT for more information. The driver-info field describes the devices to use. Curly braces indicate multiple replace- ments in the string. For
new amtoc doesn't work like the old one!
Hi everyone, I am having trouble with amtoc. In the earlier version of amtoc (Amanda 2.4.1) it asked for the configuration as an argument, i.e, Daily, Monthly, 4_Monthly and it worked. amtoc picked up the log file from the /tmp directory. However, with the new version (Amanda 2.4.2 patch 2), I have to give the entire path and log-filename as an argument. Does anybody have an idea as to how the latest version of amtoc can be made to behave like the previous version? Thanks in advance. Ashwin Bijur. Combustion Research and Flow Technology, Inc. 174 North Main Street Building 3, P.O.Box 1150 Dublin, PA 18917 Tel: (215) 249-9780 Fax: (215) 249-9796 www.craft-tech.com [EMAIL PROTECTED]
Re: Backing up PostgreSQL?
At 2002-06-14T05:40:24Z, Brandon D. Valentine [EMAIL PROTECTED] writes: Space permitting? Most natural language plain text files compress very well. A file full of database queries is going to have lots of easy to predict repetitive query statements and lots of natural language within the database entries itself. It should compress better than 2:1. n + n/2 can be something to contend with when n is on the order of 30GB and you're using commodity hardware. -- Kirk Strauser The Strauser Group - http://www.strausergroup.com/
Re: Backing up PostgreSQL?
At 2002-06-14T07:41:53Z, Ragnar Kjørstad [EMAIL PROTECTED] writes: Hmm, is the pg_dump consistent? From man 1 pg_dump: pg_dump makes consistent backups even if the database is being used concurrently. pg_dump does not block other users accessing the database (readers or writers). IOW, is it done in a single transaction? (even for multiple databases?) From what I can tell, pg_dump has no provision for concurrent dumping of multiple databases. -- Kirk Strauser The Strauser Group - http://www.strausergroup.com/
RE: Skipping a tape
Yes, that would be the right way to do it. This is why many people on the list have recommended having and extra tape or two in the rotation as spares. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 13, 2002 7:35 PM To: Bort, Paul Cc: [EMAIL PROTECTED] Subject: Re: Skipping a tape Thanks for the answer. However, I belive I am hitting the issue 1) that you mentioned. My configuration is as follows: dumpcycle 2 weeks # the number of days in the normal dump cycle runspercycle 10 # the number of amdump runs in dumpcycle days tapecycle 22 tapes # the number of tapes in rotation Could I perhaps change tapecycle to 21 tapes for a short time? Thanks again, d Bort, Paul wrote: No need to edit the tapelist file directly, almost ever. In this case, you can tell AMANDA not to use the tape, then tell her it is available again later. Remove a tape from rotation: amanda@tape$ amadmin YOURCONFIG no-reuse YOURTAPE Return a tape to rotation: amanda@tape$ amadmin YOURCONFIG reuse YOURTAPE Notes: 1) AMANDA may ask for a new tape if this reduces the number of active tapes below the tapecycle value in amanda.conf. 2) Index information is preserved for restores, so don't worry about being able to restore from this tape 3) If you're alternating between a set of on-site and off-site tapes, you may want to add a third set that sits idle at your office waiting to be used. This gives you time to change tapes off-site. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 13, 2002 4:38 PM To: [EMAIL PROTECTED] Subject: Skipping a tape Hi, I am wondering if it is possible to safely skip a tape that happens to be unavailable (it was moved offsite but hasn't returned). I've looked at the $AMANDA/tapelist file and am tempted to edit, but am hesitating because I don't actually understand how Amanda uses the information. In particular, does the line position matter? The tape that Amanda wants is the last tape in tapelist, and it also has the earliest last-use date (the first field). If I wanted Amanda to just move to the next tape, should I just edit the date field? Move the line to the top of the file? Both? tia, d
Re: Intermittent timeouts udp woes
Here is an FYI wrap-up of a problem I sent to users and hackers lists a week or so ago. Problem was that amcheck worked most of the time, but timed out occasionally to frequently, especially for a Linux host. Solution was to recheck what I (thought I) had done in setting up the /etc/services and /etc/inetd.conf files. When setting up the clients, I had mis-specified nowait in the following inetd.conf line: amanda dgram udp nowait amanda /usr/local/amanda/libexec/amandad amandad The specification (per the installation instructions) should have been wait instead of nowait. It was easy to overlook the error because the other amanda-related lines in the file appropriately used nowait. Of course, with the wonders of cut and paste, I propagated the error to all of the clients. When I fixed it, time outs disappeared for all of my clients *except* for the Linux box. Weird behavior persisting there was that amcheck routinely called into being nearly twenty instances of amandad and still timed out frequently. So... I just upgraded the Linux host from RedHat 6.2 to 7.3 and the problem went away. A side benefit is that the upgrade automatically took care of migrating everything from the inetd.conf style to the xinetd style of configuring services. I've now run through two consecutive days, backing up 21 partitions on 7 hosts, without a hiccup. One new question (there's always one more) concerns the port 10080 entries that RedHat 7.3 set up for amanda in the new /etc/services file: amanda 10080/tcp # amanda backup services amanda 10080/udp # amanda backup services I don't recall anything from the various versions of installation instructions about setting up amanda for tcp on 10080. Can anyone comment on the reason/effect of this entry? Robert L. Becker, Jr. Col, USAF, MC Department of Cellular Pathology Armed Forces Institute of Pathology Washington, DC 20306-6000 301-319-0300 On Thu, 6 Jun 2002, Robert L. Becker Jr. wrote: Folks -- I'm posting a request for help to hackers list because 1) I've seen no definitive help so far from other users, and 2) investigation is leading into packet dissection that is way over my head. Problem seems to be udp miscommunication during amcheck and (apparently) during amdump. Client/Server links through amandad work for various hosts one time, but not the next. Previous posts to amanda-users (see thread Intermittent time-outs) describe sporadic amcheck failures, with timeout errors reported at the level of dgram_recv, for example (to a host for which selfcheck succeeded minutes before): amandad: weird, it's not a proper ack addr: peer 10.20.30.55 dup 10.20.30.55, port: peer 559 dup 582 amandad: dgram_recv: recvfrom() failed: Connection refused amandad: waiting for ack: Connection refused, retrying amandad: dgram_recv: recvfrom() failed: Connection refused amandad: waiting for ack: Connection refused, retrying amandad: dgram_recv: recvfrom() failed: Connection refused amandad: waiting for ack: Connection refused, retrying amandad: dgram_recv: recvfrom() failed: Connection refused amandad: waiting for ack: Connection refused, giving up! amandad: pid 1143 finish time Wed Jun 5 10:42:27 2002 [snipping out the rest of prior post to hackers list, now overtaken by events... anyone interested in symptoms can find the full msg in the hacker archives]
GNU tar and SAMBA backups
I'm moving over to use SAMBA based AMANDA backups, and have to start using gnutar. If my amcheck tells me... ERROR: edinburgh: [GNUTAR program not available] ERROR: edinburgh: [can not read/write /usr/local/var/amanda/gnutar-lists/.: No such file or directory] Client check: 1 host checked in 1.178 seconds, 2 problems found Where am I going wrong? I have downloaded, configured, make'd and installed GNU tar from www.gnu.org, I have re-configured, make-d and installed Amanda with the additional --with-gnutar=/usr/local/bin/tar I see a few websites/FAQs refer to gtar and can someone also confirm if gnutar is the same as tar from GNU??? Regards Kenny -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 Wolfson Microelectronics Ltd. http://www.wolfsonmicro.com t: +44 131 272-7000 f: +44 131 272-7001 Registered in Scotland 89839 This message may contain confidential or proprietary information. If you receive this message in error, please immediately delete it, destroy all copies of it and notify the sender. You must not use or disclose any part of this message if you are not the intended recipient. We may monitor all Email communication through our networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise. We take reasonable precautions to ensure our Emails are virus free. However, we cannot accept responsibility for any virus transmitted by us and recommend that you subject any incoming Email to your own virus checking procedures. Wolfson Microelectronics Limited is a company incorporated in Scotland having its Registered Office at 20 Bernard Terrace, Edinburgh EH8 9NX
Re: Backing up PostgreSQL?
On Thu, Jun 13, 2002 at 10:29:33PM -0500, Kirk Strauser wrote: I would re-do the backup steps as 1) Make a snapshot 2) Use dump to back up that completely static filesystem image 3) Remove the snapshot This is NOT guaranteed to work - it may, if you're lucky. By doing this you're guaranteeing that the database files, no matter how active the database, are frozen in time via the snapshot. But the big issue that you're failing to address here is that any one point in time the database files are not internally consistent. The currently running database instance has a consistent view of the data because it has data in memory and on disk and of course it knows what's where. This is why there are programs such as pgdump etc. One means suggested to backup a database which can't be stopped is to have your database files on a mirrored disk pair (or pairs). Then to make a backup, you issue read locks on all the tables. Having got those locks, you break the disk mirror at an OS level, then relinquish the locks. Then you backup the broken part of the mirror which is not in use. I think you could do something similar with a snapshot supporting filesystem. You lock all the tables, then make a snapshot, then relinquish the locks. And you should then be able to backup a consistent view of the tables from the snapshot. The above, however, is entirely speculation and I haven't tried it. But even as speculation, it's more likely to work than simply backing up a snapshot. Bear in mind that the snapshot may often work but it won't always and your backup system should cover every eventuality. Regards, Niall O Broin
Re: GNU tar and SAMBA backups
Does /usr/local/bin/tar exist (and is it executable) on edinburgh ? Frank --On Friday, June 14, 2002 16:29:27 +0100 Kenny MacPherson [EMAIL PROTECTED] wrote: I'm moving over to use SAMBA based AMANDA backups, and have to start using gnutar. If my amcheck tells me... ERROR: edinburgh: [GNUTAR program not available] ERROR: edinburgh: [can not read/write /usr/local/var/amanda/gnutar-lists/.: No such file or directory] Client check: 1 host checked in 1.178 seconds, 2 problems found Where am I going wrong? I have downloaded, configured, make'd and installed GNU tar from www.gnu.org, I have re-configured, make-d and installed Amanda with the additional --with-gnutar=/usr/local/bin/tar I see a few websites/FAQs refer to gtar and can someone also confirm if gnutar is the same as tar from GNU??? Regards Kenny -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 Wolfson Microelectronics Ltd. http://www.wolfsonmicro.com t: +44 131 272-7000 f: +44 131 272-7001 Registered in Scotland 89839 This message may contain confidential or proprietary information. If you receive this message in error, please immediately delete it, destroy all copies of it and notify the sender. You must not use or disclose any part of this message if you are not the intended recipient. We may monitor all Email communication through our networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise. We take reasonable precautions to ensure our Emails are virus free. However, we cannot accept responsibility for any virus transmitted by us and recommend that you subject any incoming Email to your own virus checking procedures. Wolfson Microelectronics Limited is a company incorporated in Scotland having its Registered Office at 20 Bernard Terrace, Edinburgh EH8 9NX -- Frank Smith[EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
RE: GNU tar and SAMBA backups
Yes it does!! edinburgh.root(7)= ls -al /usr/local/bin/tar -rwxr-xr-x 1 root other 835584 Jun 14 13:08 /usr/local/bin/tar -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 -Original Message- From: Frank Smith [mailto:[EMAIL PROTECTED]] Sent: 14 June 2002 16:48 To: Kenny MacPherson; [EMAIL PROTECTED] Subject: Re: GNU tar and SAMBA backups Does /usr/local/bin/tar exist (and is it executable) on edinburgh ? Frank --On Friday, June 14, 2002 16:29:27 +0100 Kenny MacPherson [EMAIL PROTECTED] wrote: I'm moving over to use SAMBA based AMANDA backups, and have to start using gnutar. If my amcheck tells me... ERROR: edinburgh: [GNUTAR program not available] ERROR: edinburgh: [can not read/write /usr/local/var/amanda/gnutar-lists/.: No such file or directory] Client check: 1 host checked in 1.178 seconds, 2 problems found Where am I going wrong? I have downloaded, configured, make'd and installed GNU tar from www.gnu.org, I have re-configured, make-d and installed Amanda with the additional --with-gnutar=/usr/local/bin/tar I see a few websites/FAQs refer to gtar and can someone also confirm if gnutar is the same as tar from GNU??? Regards Kenny -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 Wolfson Microelectronics Ltd. http://www.wolfsonmicro.com t: +44 131 272-7000 f: +44 131 272-7001 Registered in Scotland 89839 This message may contain confidential or proprietary information. If you receive this message in error, please immediately delete it, destroy all copies of it and notify the sender. You must not use or disclose any part of this message if you are not the intended recipient. We may monitor all Email communication through our networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise. We take reasonable precautions to ensure our Emails are virus free. However, we cannot accept responsibility for any virus transmitted by us and recommend that you subject any incoming Email to your own virus checking procedures. Wolfson Microelectronics Limited is a company incorporated in Scotland having its Registered Office at 20 Bernard Terrace, Edinburgh EH8 9NX -- Frank Smith [EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Intermittent timeouts udp woes
On Fri, 2002-06-14 at 10:59, Robert L. Becker Jr. wrote: [..snip..] One new question (there's always one more) concerns the port 10080 entries that RedHat 7.3 set up for amanda in the new /etc/services file: amanda 10080/tcp # amanda backup services amanda 10080/udp # amanda backup services I don't recall anything from the various versions of installation instructions about setting up amanda for tcp on 10080. Can anyone comment on the reason/effect of this entry? Robert L. Becker, Jr. Col, USAF, MC Department of Cellular Pathology Armed Forces Institute of Pathology Washington, DC 20306-6000 301-319-0300 Assuming amanda hasn't been modified to handle tcp, the effect of the entry is zilch. If a tcp service where to start listening to 10080 with this /etc/services in place, then utilities like netstat may incorrectly report the service as amanda, but nothing else. /etc/services merely associates service names with port/protocol pairs. I suspect someone was a little overzealous and assigned both port/protocol pairs to the same service. My /etc/services file lists www-http as both 80/udp and 80/tcp, but I have yet to find an http server that can service udp queries.
amrecover no such file
anyone seen this? akita 25# ./amrecover AMRECOVER Version 2.4.2p2. Contacting server on amanda-server ... amrecover: cannot connect to amanda-server: No such file or directory thanks, Jonathan Murray
RE: GNU tar and SAMBA backups
Check /tmp/amanda/amandad.debug to see if GNUTAR=/usr/local/bin/tar appears in the list of configuration options near the top of the file. If not, I suspect you didn't clear the cached configure output from your original build before you re-ran configure with your new options. Frank --On Friday, June 14, 2002 16:51:47 +0100 Kenny MacPherson [EMAIL PROTECTED] wrote: Yes it does!! edinburgh.root(7)= ls -al /usr/local/bin/tar -rwxr-xr-x 1 root other 835584 Jun 14 13:08 /usr/local/bin/tar -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 -Original Message- From: Frank Smith [mailto:[EMAIL PROTECTED]] Sent: 14 June 2002 16:48 To: Kenny MacPherson; [EMAIL PROTECTED] Subject: Re: GNU tar and SAMBA backups Does /usr/local/bin/tar exist (and is it executable) on edinburgh ? Frank --On Friday, June 14, 2002 16:29:27 +0100 Kenny MacPherson [EMAIL PROTECTED] wrote: I'm moving over to use SAMBA based AMANDA backups, and have to start using gnutar. If my amcheck tells me... ERROR: edinburgh: [GNUTAR program not available] ERROR: edinburgh: [can not read/write /usr/local/var/amanda/gnutar-lists/.: No such file or directory] Client check: 1 host checked in 1.178 seconds, 2 problems found Where am I going wrong? I have downloaded, configured, make'd and installed GNU tar from www.gnu.org, I have re-configured, make-d and installed Amanda with the additional --with-gnutar=/usr/local/bin/tar I see a few websites/FAQs refer to gtar and can someone also confirm if gnutar is the same as tar from GNU??? Regards Kenny -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 Wolfson Microelectronics Ltd. http://www.wolfsonmicro.com t: +44 131 272-7000 f: +44 131 272-7001 Registered in Scotland 89839 This message may contain confidential or proprietary information. If you receive this message in error, please immediately delete it, destroy all copies of it and notify the sender. You must not use or disclose any part of this message if you are not the intended recipient. We may monitor all Email communication through our networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise. We take reasonable precautions to ensure our Emails are virus free. However, we cannot accept responsibility for any virus transmitted by us and recommend that you subject any incoming Email to your own virus checking procedures. Wolfson Microelectronics Limited is a company incorporated in Scotland having its Registered Office at 20 Bernard Terrace, Edinburgh EH8 9NX -- Frank Smith [EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501 -- Frank Smith[EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
RE: GNU tar and SAMBA backups
This certainly can be made to work. All I did was add this line to disklist: edge //viveka/C user-tar note that edge is the localhost, since the samba is there. No software needed on the remote windows box. Then I had to make sure my dumptype did not involve the exclude command. Then I had to set up the account on the windows box and the password on the amanda box. Looks like you will need to create the directory /usr/local/var/amanda/gnutar-lists Not sure about your main problem. Are you sure you have gnutar and not merely tar? Check out http://www.samag.com/print/documentID=24118 and also http://www.samag.com/print/documentID=24115 Maybe we need to see your disklist and amanda.conf. Which Unix are you running? -Steve -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Kenny MacPherson Sent: Friday, June 14, 2002 8:52 AM To: 'Frank Smith'; [EMAIL PROTECTED] Subject: RE: GNU tar and SAMBA backups Yes it does!! edinburgh.root(7)= ls -al /usr/local/bin/tar -rwxr-xr-x 1 root other 835584 Jun 14 13:08 /usr/local/bin/tar -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 -Original Message- From: Frank Smith [mailto:[EMAIL PROTECTED]] Sent: 14 June 2002 16:48 To: Kenny MacPherson; [EMAIL PROTECTED] Subject: Re: GNU tar and SAMBA backups Does /usr/local/bin/tar exist (and is it executable) on edinburgh ? Frank --On Friday, June 14, 2002 16:29:27 +0100 Kenny MacPherson [EMAIL PROTECTED] wrote: I'm moving over to use SAMBA based AMANDA backups, and have to start using gnutar. If my amcheck tells me... ERROR: edinburgh: [GNUTAR program not available] ERROR: edinburgh: [can not read/write /usr/local/var/amanda/gnutar-lists/.: No such file or directory] Client check: 1 host checked in 1.178 seconds, 2 problems found Where am I going wrong? I have downloaded, configured, make'd and installed GNU tar from www.gnu.org, I have re-configured, make-d and installed Amanda with the additional --with-gnutar=/usr/local/bin/tar I see a few websites/FAQs refer to gtar and can someone also confirm if gnutar is the same as tar from GNU??? Regards Kenny -- Kenny MacPherson IT Manager DDI: +44 131 272 7091 Wolfson Microelectronics Ltd. http://www.wolfsonmicro.com t: +44 131 272-7000 f: +44 131 272-7001 Registered in Scotland 89839 This message may contain confidential or proprietary information. If you receive this message in error, please immediately delete it, destroy all copies of it and notify the sender. You must not use or disclose any part of this message if you are not the intended recipient. We may monitor all Email communication through our networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise. We take reasonable precautions to ensure our Emails are virus free. However, we cannot accept responsibility for any virus transmitted by us and recommend that you subject any incoming Email to your own virus checking procedures. Wolfson Microelectronics Limited is a company incorporated in Scotland having its Registered Office at 20 Bernard Terrace, Edinburgh EH8 9NX -- Frank Smith [EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Backing up PostgreSQL?
[ On , June 14, 2002 at 09:23:33 (-0500), Kirk Strauser wrote: ] Subject: Re: Backing up PostgreSQL? At 2002-06-14T07:41:53Z, Ragnar Kjørstad [EMAIL PROTECTED] writes: Hmm, is the pg_dump consistent? From man 1 pg_dump: pg_dump makes consistent backups even if the database is being used concurrently. pg_dump does not block other users accessing the database (readers or writers). I believe a caveat is necessary -- the reliability of this feature depends on which version of PostgreSQL you're using -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
Re: Backing up PostgreSQL?
[ On , June 13, 2002 at 22:29:33 (-0500), Kirk Strauser wrote: ] Subject: Re: Backing up PostgreSQL? At 2002-06-13T23:07:54Z, Niall O Broin [EMAIL PROTECTED] writes: Nothing to do with BSD, Linux, or even, Gawd help us, Windows. If you don't take the database offline then the files which make up the database are not guaranteed to be in anything resembling a consistent state i.e. if you copy those files, in whatever manner, there's absolutely no guarantee that you'll be later able to use the database program to read the files. That's the really nice thing about FreeBSD's new snapshots (in the development tree). When you freeze the filesystem image, it really is a snapshot. If all of your database files are on the same filesystem then you're pretty much guaranteed a consistent backup, regardless of the amount of updates being committed to the database. Which is still really totally useless for backing up an RDBMS like PostgreSQL. Even if everything is fsync()ed all the time there's still going to be dozens of race conditions where a backup, even if supposedly nearly instantaneous like a snapshot, will result in an inconsistent database image. With PostgreSQL in particular (since it only uses filesystem files, not raw disk) you MUST stop the database -- i.e. completely stop the processes that have database files open so that those files are completely all closed and so that the kernel filesystem interface can present a consistent view of the data that's been written to the filesystem. This is becoming even more critical as PostgreSQL gains ever more features that allow the DBA to optimise when writes are done to the filesystem. The only good thing about filesystem snapshots is that you can in theory do them much quicker (than a full dump) so database downtime is much lower. I would re-do the backup steps as 0) Stop the database 1) Make a snapshot 1.5) restart the database 2) Use dump to back up that completely static filesystem image /^\ a level zero 3) Remove the snapshot Of course if you're concerned about the integrity of your data (and perhaps you give your data a higher value than even the application and support system it is stored and manipulated in), particularly for off-site backups, then the only really sure way to backup the database is to do a dump at the database level into a flat-file form, and of course PostgreSQL has pg_dump which has already been discussed. You should probably be doing regular pg_dumps anyway, even if you don't do them every time you run amanda. They're the only way you can be sure your disaster recovery plan works -- i.e. they let you reload a new test database to be sure everything works. -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
RE: How to Configure Tapeless Backup in 2.4.3
On Fri, 2002-06-14 at 19:17, Steve Follmer wrote: To recap, the poster Mohamed and myself have the same ERROR, viz: $ /usr/local/sbin/amcheck puredisk Amanda Tape Server Host Check - ERROR: file:/backup: not an amanda tape (expecting a new tape) NOTE: skipping tape-writable test just run amlabel DailySet1 label that did it for me - my amcheck runs 100% with 2.4.3beta3 and file:/backup. -- Regards, Klavs Klavsen -| This mail has been sent to you by: | Klavs Klavsen - Open Source Consultant [EMAIL PROTECTED] - http://www.vsen.dk Get PGP key from www.keyserver.net - Key ID: 0x586D5BCA Fingerprint = A95E B57B 3CE0 9131 9D15 94DA E1CD 641E 586D 5BCA [ I believe that... ]--- It is a myth that people resist change. People resist what other people make them do, not what they themselves choose to do... That's why companies that innovate successfully year after year seek their peopl's ideas, let them initiate new projects and encourage more experiments.-- Rosabeth Moss Kanter signature.asc Description: This is a digitally signed message part
Re: How to Configure Tapeless Backup in 2.4.3
Steve Follmer [EMAIL PROTECTED] wrote: To recap, the poster Mohamed and myself have the same ERROR, viz: $ /usr/local/sbin/amcheck puredisk Amanda Tape Server Host Check - ERROR: file:/backup: not an amanda tape (expecting a new tape) NOTE: skipping tape-writable test Looks like amanda found something, but it just didn't have an Amanda label. Try running `amlabel' on it. -- Mike Hicks [mailto:[EMAIL PROTECTED]] Unix Support Assistant| Carlson School of Management Office: 1-160 Phone: 6-7909 | University of Minnesota msg12918/pgp0.pgp Description: PGP signature
RE: Backing up PostgreSQL?
[ On Friday, June 14, 2002 at 12:20:21 (-0400), Bort, Paul wrote: ] Subject: RE: Backing up PostgreSQL? I don't know much about PostgresQL, but on MS SQL server and Oracle (IIRC), any update that would leave the database inconsistent should be inside a transaction. Any snapshot will not happen while a transaction is in progress; therefore the snapshot is consistent and restorable. I guess it depends on how sane your programmers are. Oracle, running on any snapshot-capable unix (including FreeBSD) and using a normal filesystem for storage, will not -- cannot possibly -- guarantee that a snapshot will not happen while a transaction is in progress. There is no possible interlock in any snapshot implementations I'm aware of between the kernel (which does the snapshot operation) and the user-land Oracle process(es). -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
RE: Backing up PostgreSQL?
Sorry, when the previous poster mentioned 'snapshot', I was thinking of SQL Server's 'dump', which is transactionally consistent, because it's done by the SQL engine. I thought Oracle had a similar method for producing a usable backup of the SQL server? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, June 14, 2002 2:53 PM To: Bort, Paul Cc: [EMAIL PROTECTED] Subject: RE: Backing up PostgreSQL? [ On Friday, June 14, 2002 at 12:20:21 (-0400), Bort, Paul wrote: ] Subject: RE: Backing up PostgreSQL? I don't know much about PostgresQL, but on MS SQL server and Oracle (IIRC), any update that would leave the database inconsistent should be inside a transaction. Any snapshot will not happen while a transaction is in progress; therefore the snapshot is consistent and restorable. I guess it depends on how sane your programmers are. Oracle, running on any snapshot-capable unix (including FreeBSD) and using a normal filesystem for storage, will not -- cannot possibly -- guarantee that a snapshot will not happen while a transaction is in progress. There is no possible interlock in any snapshot implementations I'm aware of between the kernel (which does the snapshot operation) and the user-land Oracle process(es). -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
RE: Backing up PostgreSQL?
[ On Friday, June 14, 2002 at 15:07:40 (-0400), Bort, Paul wrote: ] Subject: RE: Backing up PostgreSQL? Sorry, when the previous poster mentioned 'snapshot', I was thinking of SQL Server's 'dump', which is transactionally consistent, because it's done by the SQL engine. I thought Oracle had a similar method for producing a usable backup of the SQL server? yes, I believe Oracle also has a database dump utility even when using filesystem storage (it certainly does when raw disk is used for storage and I see no logical reason why the same tool wouldn't work for all storage mechanisms, but then again we are talking about Oracle! :-) -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
Re: Backing up PostgreSQL?
On Fri, Jun 14, 2002 at 04:32:20PM +0100, Niall O Broin wrote: 1) Make a snapshot 2) Use dump to back up that completely static filesystem image 3) Remove the snapshot This is NOT guaranteed to work - it may, if you're lucky. By doing this you're guaranteeing that the database files, no matter how active the database, are frozen in time via the snapshot. But the big issue that you're failing to address here is that any one point in time the database files are not internally consistent. They're always consistent. If they hadn't been, you would not be able to recover from a crash / powerfailure. -- Ragnar Kjørstad Big Storage
Re: Backing up PostgreSQL?
[ On Saturday, June 15, 2002 at 00:45:12 (+0200), Ragnar Kjørstad wrote: ] Subject: Re: Backing up PostgreSQL? snapshots (LVM snapshots) are not supposedly nearly instantaneous, but instantaneous. All write-access to the device is _locked_ while the snapshot is in progress (the process takes a few milliseconds, maybe a second on a big system), and there are _no_ races. That's the whole point of the snapshot! That's irrelevant from PostgreSQL's point of view. There's no sure way to tell the postgresql process(es) to make the on-disk database image consistent before you create the snapshot. The race condition is between the user-level process and the filesystem. The only sure way to guarantee a self-consistent backup is to shut down the process so as to ensure all the files it had open are now closed. PostgreSQL makes no claims that all data necessary to present a continually consistent view of the DB will be written in a single system call. In fact this is impossible since the average database consistes of many files and you can only write to one file at a time through the UNIX system interface. If your snapshot occurs between two writes where both are necessary to maintain DB consistency then the snapshot is in an inconsistent state. Yes there are other ways to recover consistency using other protection mechanisms maintained by PostgreSQL, but you don't want to be relying on those when you're doing backups -- you barely want to rely on those when you're doing crash recovery! If doing a snapshot really is that fast then there's almost no excuse _not_ to stop the DB -- just do it! Your DB downtime will not be noticable. To postgresql (or any other application) the rollback of an snapshot (or the backup of a snapshot) will be exactly like recovering from a crash. Database-servers need to write the data to disk (and fsync) before the transaction is completed. In practise they don't actually write it to the database-files but to a log, but that doesn't change the point. So, the only advantage of shutting down the database is that it doesn't have to recover like from a crash. Newer releases of PostgreSQL don't always use fsync(). I wouldn't trust recovery to be consistent without any loss implicitly. Because PostgreSQL uses the filesystem (and not raw disk) the only way to be 100% certain that what's written to disk is a consistent view of the DB is to close all the open DB files. You don't want the state of your backups to appear as if the system had crashed -- you want them to be fully self-consistent. At least that's what you want if you care about your data _and_ your application, and you care about getting a restored system back online ASAP. Of course if you really care most about your data in and of itself then you'll be doing pg_dump to back it up -- quick restoration of a given physical system isn't always what's most important! Maybe both are important -- do a pg_dump, back up the dump file, then stop the DB, do a snapshot, restart the DB, and then back up the snapshot too. -- Greg A. Woods +1 416 218-0098; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED]; VE3TCP; Secrets of the Weird [EMAIL PROTECTED]
Re: GNU tar and SAMBA backups
On Fri, 14 Jun 2002, Michael Hicks wrote: Before you get too far, make sure you are running a new version of tar (check with `tar --version). Version 1.13 (available from ftp://ftp.gnu.org/) causes Amanda to generate corrupt index files, so recovering files can be painful. If you need a new version, get a copy of tar from ftp://alpha.gnu.org/gnu/tar/. I started running samba backups a few days before I ran across the first reference to a problem with gnutar 1.13. It turns out that's the version that runs on my smbclient host, yet the indexes displayed by amrecover and the two or three test recoveries that I've tried have worked fine. Are there special circumstances in which gnutar 1.13 fails? Thanks. Robert L. Becker, Jr. Col, USAF, MC Department of Cellular Pathology Armed Forces Institute of Pathology Washington, DC 20306-6000 301-319-0300
