Re: [HACKERS] PITR Recovery Question

2010-06-04 Thread Florian Pflug
On Jun 4, 2010, at 7:05 , Gnanakumar wrote:
>> If some of those WAL segments still reside in pg_xlog, you'll either need
> to teach your restore_command to fetch them from there. Note that you cannot
> recover "in reverse".
> 
> My pg_xlog/ and walarchive/ directory locations are
> "/usr/local/pgsql/data/pg_xlog" and "/mnt/pitr/walarchive" respectively.
> 
> If my normal restore command is: restore_command='cp
> "/mnt/pitr/walarchive/%f "%p"', how should I instruct restore command to
> fetch?  Should I just replace this with something like restore_command='cp
> /usr/local/pgsql/data/pg_xlog/%f "%p"'.  Also you have mentioned that we
> cannot recover "in reverse", what I understand from this is that even though
> if I replace the restore command pointing to pg_xlog/ directory, this will
> not work out in this situation?  Is my understanding right?

If you point it at a cluster's own pg_xlog directory, it won't work.

You might want to re-ead the section on the recovery process in the PTITR 
documentation, at
http://www.postgresql.org/docs/8.4/interactive/continuous-archiving.html#BACKUP-PITR-RECOVERY

If you have further questions, please take this discussion to pgsql-general.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] PITR Recovery Question

2010-06-03 Thread Gnanakumar
Hi Florian,

Sure.  I'll post my future questions either in pgsql-general or pgsql-admin.

> If some of those WAL segments still reside in pg_xlog, you'll either need
to teach your restore_command to fetch them from there. Note that you cannot
recover "in reverse".

My pg_xlog/ and walarchive/ directory locations are
"/usr/local/pgsql/data/pg_xlog" and "/mnt/pitr/walarchive" respectively.

If my normal restore command is: restore_command='cp
"/mnt/pitr/walarchive/%f "%p"', how should I instruct restore command to
fetch?  Should I just replace this with something like restore_command='cp
/usr/local/pgsql/data/pg_xlog/%f "%p"'.  Also you have mentioned that we
cannot recover "in reverse", what I understand from this is that even though
if I replace the restore command pointing to pg_xlog/ directory, this will
not work out in this situation?  Is my understanding right?

-Original Message-
From: Florian Pflug [mailto:f...@phlo.org] 
Sent: Thursday, June 03, 2010 8:50 PM
To: gna...@zoniac.com
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] PITR Recovery Question

Hi,

I'll try to answer your questions below, but in the future please post
questions concerning the usage and administration of postgres to
pgsql-general or pgsql-admin. This list focus is the development of new
features and bugfixes.

On Jun 3, 2010, at 15:37 , Gnanakumar wrote:
> PITR SETUP DETAILS
> We've 2 drives.  Primary drive (pgsql/data/ directory resides) is 400 GB
and
> secondary drive (WAL archive) is 30 GB.  All WAL archives are written to
> secondary drive.
> 
> Base backup taken on: Aug03, 2009
> WAL archive drive become full (100% use) on: Sep05, 2009
> 
> Because this WAL archive drive has become full, all WAL archive segments
to
> be archived are accumulated into pg_xlog/ directory itself.  Eventually, 9
> months (as of today from Sep05, 2009) of WAL archives are residing in
> pg_xlog/ directory.

This is by design. WAL logs are only removed from pg_xlog once they have
been archived successfully. Since your archive_command fails due to the disk
being full, they remain in pg_xlog. Once you enlarge the filesystem holding
the WAL archive they should be copied and subsequently removed from pg_xlog.

Note that you'd usually take a new base backup once in a while to limit the
number of WAL segments you need to retain. You can take a base backup while
postgres is running by issuing pg_start_backup() before you start the copy
and pg_stop_backup() after it finished. Apart from creating additional IO
load, doing so won't interfere with normal query execution in any way.

> My question is, in case if I would like to perform recovery process as it
is
> in this situation, will this work out?  That is, I'm seeing/finding out
> whether recovery process would perform successfully anywhere between the
> date range Aug03, 2009 (my base backup date) and as of today - Jun03,
2009.
> Reason I'm asking this is still all my WAL archives are residing in
pg_xlog/
> directory.

For PITR, you'll obviously need the WAL segment starting from the time your
base backup started up until the point you want to recover to. If some of
those WAL segments still reside in pg_xlog, you'll either need to teach your
restore_command to fetch them from there. Note that you cannot recover "in
reverse". To recover up to a certain point in time you always need to start
from a base backup taken *before* that time.

best regards,
Florian Pflug




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] PITR Recovery Question

2010-06-03 Thread Florian Pflug
Hi,

I'll try to answer your questions below, but in the future please post 
questions concerning the usage and administration of postgres to pgsql-general 
or pgsql-admin. This list focus is the development of new features and bugfixes.

On Jun 3, 2010, at 15:37 , Gnanakumar wrote:
> PITR SETUP DETAILS
> We've 2 drives.  Primary drive (pgsql/data/ directory resides) is 400 GB and
> secondary drive (WAL archive) is 30 GB.  All WAL archives are written to
> secondary drive.
> 
> Base backup taken on: Aug03, 2009
> WAL archive drive become full (100% use) on: Sep05, 2009
> 
> Because this WAL archive drive has become full, all WAL archive segments to
> be archived are accumulated into pg_xlog/ directory itself.  Eventually, 9
> months (as of today from Sep05, 2009) of WAL archives are residing in
> pg_xlog/ directory.

This is by design. WAL logs are only removed from pg_xlog once they have been 
archived successfully. Since your archive_command fails due to the disk being 
full, they remain in pg_xlog. Once you enlarge the filesystem holding the WAL 
archive they should be copied and subsequently removed from pg_xlog.

Note that you'd usually take a new base backup once in a while to limit the 
number of WAL segments you need to retain. You can take a base backup while 
postgres is running by issuing pg_start_backup() before you start the copy and 
pg_stop_backup() after it finished. Apart from creating additional IO load, 
doing so won't interfere with normal query execution in any way.

> My question is, in case if I would like to perform recovery process as it is
> in this situation, will this work out?  That is, I'm seeing/finding out
> whether recovery process would perform successfully anywhere between the
> date range Aug03, 2009 (my base backup date) and as of today - Jun03, 2009.
> Reason I'm asking this is still all my WAL archives are residing in pg_xlog/
> directory.

For PITR, you'll obviously need the WAL segment starting from the time your 
base backup started up until the point you want to recover to. If some of those 
WAL segments still reside in pg_xlog, you'll either need to teach your 
restore_command to fetch them from there. Note that you cannot recover "in 
reverse". To recover up to a certain point in time you always need to start 
from a base backup taken *before* that time.

best regards,
Florian Pflug



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-07 Thread Bruce Momjian
Tom Lane wrote:
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > Yea, my question is if you choose "after", do you get everything that
> > happens until the "after" transaction commits, or just when it begins. 
> > If I stop after xid 125, and xid 126 starts and stops before 125
> > commits, does 126 get restored?
> 
> Yes.  You don't get to be selective about what to keep: it's everything
> up to a certain time instant, and nothing after that.  Stopping by XID
> is just a different way of identifying what that time instant is.
> 
> BTW, stopping "before" an XID actually means stopping just before its
> commit or abort record, so transactions that ended before it did will
> be included in the recovery.

OK, I added a mention of this in the docs.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-07 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes:
> Yea, my question is if you choose "after", do you get everything that
> happens until the "after" transaction commits, or just when it begins. 
> If I stop after xid 125, and xid 126 starts and stops before 125
> commits, does 126 get restored?

Yes.  You don't get to be selective about what to keep: it's everything
up to a certain time instant, and nothing after that.  Stopping by XID
is just a different way of identifying what that time instant is.

BTW, stopping "before" an XID actually means stopping just before its
commit or abort record, so transactions that ended before it did will
be included in the recovery.

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-07 Thread Bruce Momjian
Tom Lane wrote:
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > When we do a PITR recovery based on xid, does it stop recovery based on
> > the start of the xid or the commit of the xid?
> 
> You can stop either "before" or "after" that commit.  See
> recovery.conf.sample (I don't think it's documented anywhere else
> yet :-(),

Yea, my question is if you choose "after", do you get everything that
happens until the "after" transaction commits, or just when it begins. 
If I stop after xid 125, and xid 126 starts and stops before 125
commits, does 126 get restored?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-06 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes:
> When we do a PITR recovery based on xid, does it stop recovery based on
> the start of the xid or the commit of the xid?

You can stop either "before" or "after" that commit.  See
recovery.conf.sample (I don't think it's documented anywhere else
yet :-(),

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-06 Thread Bruce Momjian

When we do a PITR recovery based on xid, does it stop recovery based on
the start of the xid or the commit of the xid?  And if you say
recovery_target_inclusive =true, does it recover that xid while not
recoverying other xids that are higher but committed sooner than the
target xid?

---

Oliver Elphick wrote:
> On Wed, 2004-08-04 at 19:16, Tom Lane wrote:
> > Oliver Elphick <[EMAIL PROTECTED]> writes:
> > > How about adding a logging option to put the transaction id on the log
> > > for every statement that modifies the database?  Would that be a small
> > > enough change to be allowed into 8.0?
> > 
> > I think we could get away with adding transaction ID as one of the
> > available %-items in log_line_prefix.  I'm not sure how useful this
> > really is though --- timestamps are probably more useful overall to
> > have in your log.
> 
> Why not both?
> 
> You seem to be suggesting that using the id is less useful than the
> time, but surely it's going to be easier to say "this disaster happened
> in transaction 123 so lets do a PITR up to 122" than to say "this
> happened at time x so do PITR up to x - 1 second"; the latter might miss
> several tranactions.  Have I got the concepts wrong here?
> 
> >   The direction I was expecting we'd head in is to
> > provide WAL logfile examination tools.
> 
> But that's not going to happen for 8.0, so any means of getting the
> transaction id is better than none.
> 
> -- 
> Oliver Elphick  [EMAIL PROTECTED]
> Isle of Wight  http://www.lfix.co.uk/oliver
> GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
>  
>  "And not only so, but we glory in tribulations also; 
>   knowing that tribulation worketh patience; And  
>   patience, experience; and experience, hope."  
> Romans 5:3,4 
> 
> 
> ---(end of broadcast)---
> TIP 3: if posting/reading through Usenet, please send an appropriate
>   subscribe-nomail command to [EMAIL PROTECTED] so that your
>   message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-06 Thread Gaetano Mendola
G u i d o B a r o s i o wrote:
8.0 || 7.5??
8.0
Regards
Gaetano Mendola

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
  http://www.postgresql.org/docs/faqs/FAQ.html


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-04 Thread Bruce Momjian
Rod Taylor wrote:
> > You seem to be suggesting that using the id is less useful than the
> > time, but surely it's going to be easier to say "this disaster happened
> > in transaction 123 so lets do a PITR up to 122" than to say "this
> 
> Transaction IDs are assigned at transaction start but what you really
> want is some indicator of when the commit occurred.
> 
> Transaction 123 may have committed while 122 was still running.

True.  In fact this brings up a problem of using the xid for recovery
stop.  The interesting point is that you might recover to just before
xact 123, but that doesn't mean you get xact 122.

Still I think we need to add xid to the log_line_prefix for PITR and
make it clear that specifying a recovery xid doesn't always include
earlier xids.  I have added this to the open items list.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-04 Thread Rod Taylor
> You seem to be suggesting that using the id is less useful than the
> time, but surely it's going to be easier to say "this disaster happened
> in transaction 123 so lets do a PITR up to 122" than to say "this

Transaction IDs are assigned at transaction start but what you really
want is some indicator of when the commit occurred.

Transaction 123 may have committed while 122 was still running.



---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-04 Thread Oliver Elphick
On Wed, 2004-08-04 at 19:16, Tom Lane wrote:
> Oliver Elphick <[EMAIL PROTECTED]> writes:
> > How about adding a logging option to put the transaction id on the log
> > for every statement that modifies the database?  Would that be a small
> > enough change to be allowed into 8.0?
> 
> I think we could get away with adding transaction ID as one of the
> available %-items in log_line_prefix.  I'm not sure how useful this
> really is though --- timestamps are probably more useful overall to
> have in your log.

Why not both?

You seem to be suggesting that using the id is less useful than the
time, but surely it's going to be easier to say "this disaster happened
in transaction 123 so lets do a PITR up to 122" than to say "this
happened at time x so do PITR up to x - 1 second"; the latter might miss
several tranactions.  Have I got the concepts wrong here?

>   The direction I was expecting we'd head in is to
> provide WAL logfile examination tools.

But that's not going to happen for 8.0, so any means of getting the
transaction id is better than none.

-- 
Oliver Elphick  [EMAIL PROTECTED]
Isle of Wight  http://www.lfix.co.uk/oliver
GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
 
 "And not only so, but we glory in tribulations also; 
  knowing that tribulation worketh patience; And  
  patience, experience; and experience, hope."  
Romans 5:3,4 


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-04 Thread Tom Lane
Oliver Elphick <[EMAIL PROTECTED]> writes:
> How about adding a logging option to put the transaction id on the log
> for every statement that modifies the database?  Would that be a small
> enough change to be allowed into 8.0?

I think we could get away with adding transaction ID as one of the
available %-items in log_line_prefix.  I'm not sure how useful this
really is though --- timestamps are probably more useful overall to
have in your log.  The direction I was expecting we'd head in is to
provide WAL logfile examination tools.

regards, tom lane

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly


Re: [HACKERS] PITR - recovery to a particular transaction

2004-08-04 Thread G u i d o B a r o s i o
8.0 || 7.5??

g:)

> The PITR docs that have just been put up say:
> 
> But if you want to recover to some previous point in time (say,
> right before the junior DBA dropped your main transaction
> table), just specify the required stopping point in
> recovery.conf. You can specify the stop point either by
> date/time or by transaction ID. As of this writing only the
> date/time option is very usable, since there are no tools to
> help you identify which transaction ID to use.
> 
> How about adding a logging option to put the transaction id on the log
> for every statement that modifies the database?  Would that be a small
> enough change to be allowed into 8.0?
> -- 
> Oliver Elphick  [EMAIL PROTECTED]
> Isle of Wight  http://www.lfix.co.uk/oliver
> GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
>  
>  "And not only so, but we glory in tribulations also; 
>   knowing that tribulation worketh patience; And  
>   patience, experience; and experience, hope."  
> Romans 5:3,4 
> 
> 
> ---(end of broadcast)---
> TIP 6: Have you searched our list archives?
> 
>http://archives.postgresql.org


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster


Re: [HACKERS] PITR Recovery

2004-06-23 Thread Simon Riggs
On Thu, 2004-06-17 at 22:47, Simon Riggs wrote:
> On Wed, 2004-06-16 at 02:49, Tom Lane wrote:
> > Simon Riggs <[EMAIL PROTECTED]> writes:
> > > -finalaction refers to what to do when target is reached - the purpose
> > > of this is to allow recovery of a database to occur when we don't have
> > > enough space for all of the xlogs at once, so we need to do recovery in
> > > batches.
> > 
> > It seems to me that this is the only *essential* feature out of what
> > you've listed, and the others are okay to add later.  So I question
> > your priorities:
> > 
> > > In time for beta freeze, I think it is possible to do a limited subset
> > > of the above:
> > > - implement DATABASE only (whole instance, not specific database)
> > > - implement END OF LOGS and TO TIMESTAMP
> > > - implement THEN START only
> > > - implement using simple C, rather than bison
> > 
> > which seem to include everything except the one absolute must-have
> > for any serious installation.
> > 
> 
> OK. At first, I disagreed, for many reasons.
> 
> I discussion with Bruce, I believe a fairly neat streaming solution is
> possible.
> 
> During recovery, as each request for a new xlog is made, we can make a
> system(3) call to a user defined recovery_program to retrieve the next
> xlog and out it in place. As each xlog is closed the file will be
> removed. The result of this would be to stream the data files through
> recovery, so no more than 1-2 files would ever be required to perform
> what could be (and is touted as this by other vendors) an infinite
> recovery.
> 
> The result is that a backup tape (or other tape silo) could stream data
> straight through to recovery, and would completely circumvent and
> concern about insufficient disk space for recovery.
> 
> This would involve changes to XLogFileOpen() in xlog.c and far less
> complex than I had imagined such functionality could be.
> 
> This could be specified to PostgreSQL by using:
> - restore_program='cp %s %s' or similar
> 
> I'll work more on the design, but not tonight.
> 

Technically straightforward, though more complex I thought, but
streaming the xlog files during recovery works in prototype - great idea
Bruce and thanks for pushing for a solution in that area, Tom.
[It looks like we do need to have a separate command file dedicated to
recovery options, otherwise there's no way to tell difference between
crash and full media recovery - but I'll lose the pompous syntax.]

I'll include this (actually very few new/changed lines) and the xlog
refactoring (lots of moved lines, but few changes) in a single patch.

These changes are dependent upon, but otherwise independent of the PITR
Archival path submitted on 15th. If anybody has comments on that patch,
please pass them through ASAP, otherwise I may be building on sand.

My plan is to get this out ASAP (tonight, hopefully), then build on it
with a few extra tweaks, so we have a full set of options for PITR by
29th.

Thanks,

Best Regards, Simon Riggs



---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings


Re: [HACKERS] PITR Recovery

2004-06-17 Thread Simon Riggs
On Wed, 2004-06-16 at 02:49, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
> > -finalaction refers to what to do when target is reached - the purpose
> > of this is to allow recovery of a database to occur when we don't have
> > enough space for all of the xlogs at once, so we need to do recovery in
> > batches.
> 
> It seems to me that this is the only *essential* feature out of what
> you've listed, and the others are okay to add later.  So I question
> your priorities:
> 
> > In time for beta freeze, I think it is possible to do a limited subset
> > of the above:
> > - implement DATABASE only (whole instance, not specific database)
> > - implement END OF LOGS and TO TIMESTAMP
> > - implement THEN START only
> > - implement using simple C, rather than bison
> 
> which seem to include everything except the one absolute must-have
> for any serious installation.
> 

OK. At first, I disagreed, for many reasons.

I discussion with Bruce, I believe a fairly neat streaming solution is
possible.

During recovery, as each request for a new xlog is made, we can make a
system(3) call to a user defined recovery_program to retrieve the next
xlog and out it in place. As each xlog is closed the file will be
removed. The result of this would be to stream the data files through
recovery, so no more than 1-2 files would ever be required to perform
what could be (and is touted as this by other vendors) an infinite
recovery.

The result is that a backup tape (or other tape silo) could stream data
straight through to recovery, and would completely circumvent and
concern about insufficient disk space for recovery.

This would involve changes to XLogFileOpen() in xlog.c and far less
complex than I had imagined such functionality could be.

This could be specified to PostgreSQL by using:
- restore_program='cp %s %s' or similar

I'll work more on the design, but not tonight.

Best Regards, Simon Riggs


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match


Re: [HACKERS] PITR Recovery

2004-06-17 Thread Simon Riggs
On Wed, 2004-06-16 at 23:50, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
> > Is that something you'd be able to do as a starting point for the other
> > changes? It's easier for a committer to do this, than for me to do it
> > and then another to review it...
> 
> I'm up to my eyeballs in tablespaces right now, but if you can wait a
> couple days for this ...

Whatever minimises your time...seriously

Say the word and I'll do it.

Simon


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org


Re: [HACKERS] PITR Recovery

2004-06-16 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes:
> Is that something you'd be able to do as a starting point for the other
> changes? It's easier for a committer to do this, than for me to do it
> and then another to review it...

I'm up to my eyeballs in tablespaces right now, but if you can wait a
couple days for this ...

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]


Re: [HACKERS] PITR Recovery

2004-06-16 Thread Simon Riggs
On Wed, 2004-06-16 at 02:49, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:

> > Implementation wise, I would expect all of this code to go in xlog.c,
> > with the recovery target code living in ReadRecord().
> 
> I'd like to keep it out of there, as xlog.c is far too big and complex
> already.  Not sure where else though.  Maybe we need to break down
> xlog.c somehow.
> 

Yes, I would very much like to split out the recovery code into a
different file, so that all recovery code was all in one place.

Refactoring in this way would protect further PITR work from conflicting
with other changes in the last minute rush, as well as making most
future recovery changes a single file patch (well...nearly...)

xlogutils.c is already almost fully dedicated to recovery code, so it
seems like a good place to centralise, even though I don't like the
name!

Looking at the code, I would suggest:

--Move these code sections into void X (void) functions that would
reside in xlogutils.c but get called from StartupXLog in xlog.c,
currently within if (InRecovery) {} braces:
Add StartupRecovery() - main REDO recovery
from 
---
/* REDO */
if (InRecovery)
{
--- to
(errmsg("redo is not required")));
}

/*
 * Init xlog buffer cache using the block containing the last valid
---
Add CleanupRecovery() - cleanup after recovery
..similarly

that would then allow us to 
--Move the following to xlogutils.c
   XLogInitRelationCache
   RestoreBkpBlocks
   ReadRecord (may need still to be called from xlog.c)
  RecordIsValid
  ValidXLOGHeader
   XLogCloseRelationCache

--Remove from xlogutils.h
extern void XLogInitRelationCache(void);
extern void XLogCloseRelationCache(void);

replace with 

extern void StartupRecovery(void);
extern void CleanupRecovery(void);

Is that something you'd be able to do as a starting point for the other
changes? It's easier for a committer to do this, than for me to do it
and then another to review it...

Best Regards, Simon Riggs


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings


Re: [HACKERS] PITR Recovery

2004-06-15 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes:
> -finalaction refers to what to do when target is reached - the purpose
> of this is to allow recovery of a database to occur when we don't have
> enough space for all of the xlogs at once, so we need to do recovery in
> batches.

It seems to me that this is the only *essential* feature out of what
you've listed, and the others are okay to add later.  So I question
your priorities:

> In time for beta freeze, I think it is possible to do a limited subset
> of the above:
> - implement DATABASE only (whole instance, not specific database)
> - implement END OF LOGS and TO TIMESTAMP
> - implement THEN START only
> - implement using simple C, rather than bison

which seem to include everything except the one absolute must-have
for any serious installation.

(BTW, I doubt that single-database recovery is possible at all, ever.
You can't go hacking the clog and shared tables and not keep all the
databases in sync.  So I'd forget the "object" concept altogether.)

> Implementation wise, I would expect all of this code to go in xlog.c,
> with the recovery target code living in ReadRecord().

I'd like to keep it out of there, as xlog.c is far too big and complex
already.  Not sure where else though.  Maybe we need to break down
xlog.c somehow.

regards, tom lane

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html