Re: [HACKERS] Online base backup from the hot-standby
As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameters() quite confusing. Thought? I updated a patch for what you have suggested (that the change of FPW should be WAL-logged separately from that of HS parameters). I want to base on this patch if there are no other opinions. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-07fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
+ /* + * The backend writes WAL of FPW at checkpoint. However, The backend do + * not need to write WAL of FPW at checkpoint shutdown because it + * performs when startup finishes. + */ + XLogReportParameters(REPORT_ON_BACKEND); I'm still unclear why that WAL doesn't need to be written at shutdown checkpoint. Anyway, the first sentence in the above comments is not right. Not a backend but a bgwriter writes that WAL at checkpoint. The second also seems not to be right. It implies that a shutdown checkpoint is performed only at end of startup. But it may be done when smart or fast shutdown is requested. Okay. I change to the following messages. /* * The bgwriter writes WAL of FPW at checkpoint. But does not at shutdown. * Because XLogReportParameters() is always called at the end of startup * process, it does not need to be called at shutdown. */ In addition, I change macro name. REPORT_ON_BACKEND - REPORT_ON_BGWRITER Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
+ /* +* The backend writes WAL of FPW at checkpoint. However, The backend do +* not need to write WAL of FPW at checkpoint shutdown because it +* performs when startup finishes. +*/ + XLogReportParameters(REPORT_ON_BACKEND); I'm still unclear why that WAL doesn't need to be written at shutdown checkpoint. Anyway, the first sentence in the above comments is not right. Not a backend but a bgwriter writes that WAL at checkpoint. The second also seems not to be right. It implies that a shutdown checkpoint is performed only at end of startup. But it may be done when smart or fast shutdown is requested. Okay. I change to the following messages. /* * The bgwriter writes WAL of FPW at checkpoint. But does not at shutdown. * Because XLogReportParameters() is always called at the end of startup * process, it does not need to be called at shutdown. */ In addition, I change macro name. REPORT_ON_BACKEND - REPORT_ON_BGWRITER I have updated as above-comment. Please check this. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-06fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
As I suggested in the reply to Simon, I think that the change of FPW
should be WAL-logged separately from that of HS parameters. ISTM
packing them in one WAL record makes XLogReportParameters()
quite confusing. Thought?
I want to confirm the reply of Simon. I think we cannot decide how this
code should be if there is not the reply.
if (!shutdown XLogStandbyInfoActive())
+ {
LogStandbySnapshot(checkPoint.oldestActiveXid,
checkPoint.nextXid);
+ XLogReportParameters(REPORT_ON_BACKEND);
+ }
Why doesn't the change of FPW need to be WAL-logged when
shutdown checkpoint is performed? It's helpful to add the comment
explaining why.
Sure. I update the patch soon.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
if (!shutdown XLogStandbyInfoActive())
+ {
LogStandbySnapshot(checkPoint.oldestActiveXid,
checkPoint.nextXid);
+ XLogReportParameters(REPORT_ON_BACKEND);
+ }
Why doesn't the change of FPW need to be WAL-logged when
shutdown checkpoint is performed? It's helpful to add the comment
explaining why.
Sure. I update the patch soon.
Done.
Please check this.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-05fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid' since that is what is what the user sets it to. Sure. What about the following message? It sounds more precise to me. ERROR: WAL generated with full_page_writes=off was replayed since last restartpoint Okay, I changes the patch to this messages. If someone says there is a idea better than it, I will consider again. I updated to patch corresponded above-comments. Thanks for updating the patch! Here are the comments: * don't yet have the insert lock, forcePageWrites could change under us, * but we'll recheck it once we have the lock. */ - doPageWrites = fullPageWrites || Insert-forcePageWrites; + doPageWrites = Insert-fullPageWrites || Insert-forcePageWrites; The source comment needs to be modified. * just turned off, we could recompute the record without full pages, but * we choose not to bother.) */ - if (Insert-forcePageWrites !doPageWrites) + if ((Insert-fullPageWrites || Insert-forcePageWrites) !doPageWrites) Same as above. Sure. XLogReportParameters() should skip writing WAL if full_page_writes has not been changed by SIGHUP. XLogReportParameters() should skip updating pg_control if any parameter related to hot standby has not been changed. YES. In checkpoint, XLogReportParameters() is called only when wal_level is hot_standby. OTOH, in walwriter, it's always called even when wal_level is not hot_standby. Can't we skip calling XLogReportParameters() whenever wal_level is not hot_standby? Yes, It is possible. In do_pg_start_backup() and do_pg_stop_backup(), the spinlock must be held to see XLogCtl-lastFpwDisabledLSN. Yes. What about changing the error message to: ERROR: WAL generated with full_page_writes=off was replayed during online backup Okay, too. Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); + XLogCtl-Insert.fullPageWrites = fullPageWrites; + LWLockRelease(WALInsertLock); I don't think WALInsertLock needs to be hold here because there is no concurrently running process which can access Insert.fullPageWrites. For example, Insert-currpos and Insert-LogwrtResult are also changed without the lock there. Yes. The source comment of XLogReportParameters() needs to be modified. Yes, too. Done. I updated to patch corresponded above-comments. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-04fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Some testing notes
--
select pg_start_backup('x');
ERROR: full_page_writes on master is set invalid at least once since
latest checkpoint
I think this error should be rewritten as
ERROR: full_page_writes on master has been off at some point since
latest checkpoint
We should be using 'off' instead of 'invalid' since that is what is what
the user sets it to.
Sure.
Minor typo above at 'CHECKPOTNT'
Yes.
I updated to patch corresponded above-comments.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-03fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid' since that is what is what the user sets it to. Sure. What about the following message? It sounds more precise to me. ERROR: WAL generated with full_page_writes=off was replayed since last restartpoint Okay, I changes the patch to this messages. If someone says there is a idea better than it, I will consider again. I updated to patch corresponded above-comments. Thanks for updating the patch! Here are the comments: * don't yet have the insert lock, forcePageWrites could change under us, * but we'll recheck it once we have the lock. */ - doPageWrites = fullPageWrites || Insert-forcePageWrites; + doPageWrites = Insert-fullPageWrites || Insert-forcePageWrites; The source comment needs to be modified. * just turned off, we could recompute the record without full pages, but * we choose not to bother.) */ - if (Insert-forcePageWrites !doPageWrites) + if ((Insert-fullPageWrites || Insert-forcePageWrites) !doPageWrites) Same as above. Sure. XLogReportParameters() should skip writing WAL if full_page_writes has not been changed by SIGHUP. XLogReportParameters() should skip updating pg_control if any parameter related to hot standby has not been changed. YES. In checkpoint, XLogReportParameters() is called only when wal_level is hot_standby. OTOH, in walwriter, it's always called even when wal_level is not hot_standby. Can't we skip calling XLogReportParameters() whenever wal_level is not hot_standby? Yes, It is possible. In do_pg_start_backup() and do_pg_stop_backup(), the spinlock must be held to see XLogCtl-lastFpwDisabledLSN. Yes. What about changing the error message to: ERROR: WAL generated with full_page_writes=off was replayed during online backup Okay, too. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); + XLogCtl-Insert.fullPageWrites = fullPageWrites; + LWLockRelease(WALInsertLock); I don't think WALInsertLock needs to be hold here because there is no concurrently running process which can access Insert.fullPageWrites. For example, Insert-currpos and Insert-LogwrtResult are also changed without the lock there. Yes. The source comment of XLogReportParameters() needs to be modified. Yes, too. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I can't see a reason why we would use a new WAL record for this,
rather than modify the XLOG_PARAMETER_CHANGE record type which was
created for a very similar reason.
The code would be much simpler if we just extend
XLOG_PARAMETER_CHANGE, so please can we do that?
Sure.
The log message full_page_writes on master is set invalid more than
once during online backup should read at least once rather than
more than once.
Yes.
lastFpwDisabledLSN needs to be initialized.
I think it don't need because all values in XLogCtl is initialized 0.
Is there a reason to add lastFpwDisabledLSN onto the Control file? If
we log parameters after every checkpoint then we'll know the values
when we startup. If we keep logging parameters this way we'll end up
with a very awkward and large control file. I would personally prefer
to avoid that, but that thought could go either way. Let's see if
anyone else thinks that also.
Yes. I add to CreateCheckPoint().
Image:
CreateCheckPoint()
{
if (!shutdown XLogStandbyInfoActive())
{
LogStandbySnapshot()
XLogReportParameters()
}
}
XLogReportParameters()
{
if (fpw == 'off' || ... )
XLOGINSERT()
}
However, it'll write XLOG_PARAMETER_CHANGE every checkpoints when FPW is 'off'.
(It will increases the amount of WAL.)
Is it OK?
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I can't see a reason why we would use a new WAL record for this,
rather than modify the XLOG_PARAMETER_CHANGE record type which was
created for a very similar reason.
The code would be much simpler if we just extend
XLOG_PARAMETER_CHANGE, so please can we do that?
Sure.
The log message full_page_writes on master is set invalid more than
once during online backup should read at least once rather than
more than once.
Yes.
lastFpwDisabledLSN needs to be initialized.
I think it don't need because all values in XLogCtl is initialized 0.
Is there a reason to add lastFpwDisabledLSN onto the Control file? If
we log parameters after every checkpoint then we'll know the values
when we startup. If we keep logging parameters this way we'll end up
with a very awkward and large control file. I would personally prefer
to avoid that, but that thought could go either way. Let's see if
anyone else thinks that also.
Yes. I add to CreateCheckPoint().
Image:
CreateCheckPoint()
{
if (!shutdown XLogStandbyInfoActive())
{
LogStandbySnapshot()
XLogReportParameters()
}
}
XLogReportParameters()
{
if (fpw == 'off' || ... )
XLOGINSERT()
}
However, it'll write XLOG_PARAMETER_CHANGE every checkpoints when FPW is
'off'.
(It will increases the amount of WAL.)
Is it OK?
Done.
Updated patch attached.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-02fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Some testing notes
--
select pg_start_backup('x');
ERROR: full_page_writes on master is set invalid at least once since
latest checkpoint
I think this error should be rewritten as
ERROR: full_page_writes on master has been off at some point since
latest checkpoint
We should be using 'off' instead of 'invalid' since that is what is what
the user sets it to.
Sure.
I switched full_page_writes=on , on the master
did a pg_start_backup() on the slave1.
Then I switched full_page_writes=off on the master, did a reload +
checkpoint.
I was able to then do my backup of slave1, copy the control file, and
pg_stop_backup().
When I did the test slave2 started okay, but is this safe? Do we need a
warning from pg_stop_backup() that is printed if it is detected that
full_page_writes was turned off on the master during the backup period?
I also reproduced.
pg_stop_backup() fails in most cases.
However, it succeeds if both the following cases are true.
* checkpoint is done before walwriter recieves SIGHUP.
* slave1 has not received the WAL of 'off' by SIGHUP yet.
Minor typo above at 'CHECKPOTNT'
Yes.
If my concern about full page writes being switched to off in the middle
of a backup is unfounded then I think this patch is ready for a
committer. They can clean the two editorial changes when they apply the
patches.
Yes. I'll clean since these comments fix.
If do_pg_stop_backup is going to need some logic to recheck the full
page write status then an updated patch is required.
It already contains.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I created a patch corresponding FPW. Fujii's patch (ver 9) is based. Manage own FPW in shared-memory (on master) * startup and walwriter process update it. startup initializes it after REDO. walwriter updates it when started or received SIGHUP. Insert WAL including a value of current FPW (on master) * In the the same timing as update, they insert WAL (is named XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a value of the changed FPW. * When it creates CHECKPOINT, it adds a value of current FPW to the CHECKPOINT WAL. Manage master's FPW in local-memory in startup (on standby) * It takes a value of the master's FPW by reading XLOG_FPW_CHANGE at REDO. Check when pg_start_backup/pg_stop_backup (on standby) * It checks to use these two value. * master's FPW at latest CHECKPOINT * current master's FPW by XLOG_FPW_CHANGE Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base_01fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Attached is the updated version of the patch. I refactored the code, fixed some bugs, added lots of source code comments, improved the document, but didn't change the basic design. Please check this patch, and let's use this patch as the base if you agree with that. Thanks for update patch. Yes. I agree. In the current patch, there is no safeguard for preventing users from taking backup during recovery when FPW is disabled. This is unsafe. Are you planning to implement such a safeguard? Yes. I want to reference the following Fujii's comments. - Right. Let me explain again what I'm thinking. When FPW is changed, the master always writes the WAL record which contains the current value of FPW. This means that the standby can track all changes of FPW by reading WAL records. The standby has two flags: One indicates whether FPW has always been TRUE since last restartpoint. Another indicates whether FPW has always been TRUE since last pg_start_backup(). The standby can maintain those flags by reading WAL records streamed from the master. If the former flag indicates FALSE (i.e., the WAL records which the standby has replayed since last restartpoint might not contain required FPW), pg_start_backup() fails. If the latter flag indicates FALSE (i.e., the WAL records which the standby has replayed during the backup might not contain required FPW), pg_stop_backup() fails. If I'm not missing something, this approach can address the problem which you're concerned about. - Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Update patch.
Changes:
* set 'on' full_page_writes by user (in document)
* read FROM: XX in backup_label (in xlog.c)
* check status when pg_stop_backup is executed (in xlog.c)
Hi, Created a patch in response to comments.
* Procedure
1. Call pg_start_backup('x') on hot standby.
2. Take a backup of the data dir.
3. Copy the control file on hot standby to the backup.
4. Call pg_stop_backup() on hot standby.
* Behavior
(take backup)
If we execute pg_start_backup() on hot standby then execute restartpoint,
write a strings as FROM: slave in backup_label and change backup mode,
but do not change full_page_writes into on forcibly.
If we execute pg_stop_backup() on hot standby then rename backup_label
and change backup mode, but neither write backup end record and history
file nor wait to complete the WAL archiving.
pg_stop_backup() is returned this MinRecoveryPoint as result.
If we execute pg_stop_backup() on the server promoted then error
message is output since read the backup_label.
(recovery)
If we recover with the backup taken on hot standby, MinRecoveryPoint in
the control file copied by 3 of above-procedure is used instead of backup
end record.
If recovery starts as first, BackupEndPoint in the control file is written
a same value as MinRecoveryPoint. This is for remembering the value of
MinRecoveryPoint during recovery.
HINT message(If this has ...) is always output when we recover with the
backup taken on hot standby.
* Problem
full_page_writes's problem.
This has the following two problems.
* pg_start_backup() must set 'on' to full_page_writes of the master that
is actual writing of the WAL, but not the standby.
* The standby doesn't need to connect to the master that's actual
writing
WAL.
(Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2)
I'm worried how I should clear these problems.
Status: Considering
(Latest: http://archives.postgresql.org/pgsql-hackers/2011-08/msg00880.php)
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_07.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
* Procedure
1. Call pg_start_backup('x') on the standby.
2. Take a backup of the data dir.
3. Call pg_stop_backup() on the standby.
4. Copy the control file on the standby to the backup.
5. Check whether the control file is status during hot standby with
pg_controldata.
? - If the standby promote between 3. and 4., the backup can not recovery.
? ? ?- pg_control is that Minimum recovery ending location is equals 0/0.
? ? ?- backup-end record is not written.
What if we do #4 before #3? The backup gets corrupted? My guess is
that the backup is still valid even if we copy pg_control before executing
pg_stop_backup(). Which would not require #5 because if the standby
promotion happens before pg_stop_backup(), pg_stop_backup() can
detect that status change and cancel the backup.
#5 looks fragile. If we can get rid of it, the procedure becomes more
robust, I think.
Sure, you're right.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the full_page_write = on in all case (bypass configuration) ? what's the meaning? Thanks. This has the following two problems. * pg_start_backup() must set 'on' to full_page_writes of the master that is actual writing of the WAL, but not the standby. * The standby doesn't need to connect to the master that's actual writing WAL. (Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2) I'm worried how I should clear these problems. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the full_page_write = on in all case (bypass configuration) ? what's the meaning? Yeah. full_page_writes is a WAL generation parameter. Standbys don't generate WAL. I think you just have to insist that the master has it on. Thanks. This has the following two problems. * pg_start_backup() must set 'on' to full_page_writes of the master that is actual writing of the WAL, but not the standby. * The standby doesn't need to connect to the master that's actual writing WAL. (Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2) I'm worried how I should clear these problems. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
* Not correspond yet * full_page_write = off - If the primary is full_page_write = off, archive recovery may not act normally. Therefore the standby may need to check whether full_page_write = off to WAL. Isn't having a standby make the full_page_write = on in all case (bypass configuration) ? what's the meaning? Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
I will provide a patch which can exeute pg_start/stop_backup
including to solve above comment and conditions in next stage.
Then please review.
done.
* Procedure
1. Call pg_start_backup('x') on the standby.
2. Take a backup of the data dir.
3. Call pg_stop_backup() on the standby.
4. Copy the control file on the standby to the backup.
5. Check whether the control file is status during hot standby with
pg_controldata.
- If the standby promote between 3. and 4., the backup can not recovery.
- pg_control is that Minimum recovery ending location is equals 0/0.
- backup-end record is not written.
* Not correspond yet
* full_page_write = off
- If the primary is full_page_write = off, archive recovery may not act
normally. Therefore the standby may need to check whether
full_page_write
= off to WAL.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_05.patch
Description: Binary data
standby_online_backup_doc.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
This version of the patch adds a field into pg_controldata that tries to
store the source of the base backup while in recovery mode.
I think your ultimate goal with this patch is to be able to take a
backup of a running hot-standby slave and recover it as another
instance. This patch seems to provide the ability to have the second
slave stop recovery at minRecoveryPoint from the control file.
My understanding of the procedure you want to get to to take base
backups off a slave is
1. execute pg_start_backup('x') on the slave (*)
2. take a backup of the data dir
3. call pg_stop_backup() on the slave
4. Copy the control file on the slave
This patch only addresses the recovery portions.
Yes.
I don't think the above comment is very clear on what backupserver is.
Perhaps
/**
* backupserver is used while postgresql is in recovery mode to
* store the location of where the backup comes from.
* When Postgres starts recovery operations
* it is set to none. During recovery it is updated to either master,
or slave
* When recovery operations finish it is updated back to none
**/
Done.
Also shouldn't backupServer be the enum type of 'BackupServer' not int?
Other enums in the structure such as DBState are defined this way.
Now, this is a same as wal_level, not DBState. No?
Since I can't yet call pg_start_backup or pg_stop_backup() on the slave
I am calling them on the master.
(I also did some testing where I didn't put the system into backup
mode). I admit that I am not sure what to look for as an indication that
the system isn't recovering to the correct point. In much of my testing
I was just verifying that the slave started and my data 'looked' okay.
Updated patch as can execute pg_start/stop_backup() on standby server.
One-pass of above steps(from 1. to 4.) is now done on this.
However, there are conditions.
* Master's full_page_write = on.
* On the slave, do not execute stop/promote operation before pg_stop_backup()
is executed.
* the result of pg_start_backup() may exceed the result of pg_stop_backup().
I seem to get this warning in my logs when I start up the instance based
on the slave backup.
LOG: 0: database system was interrupted while in recovery at log
time 2011-07-08 18:40:20 EDT
HINT: If this has occurred more than once some data might be corrupted
and you might need to choose an earlier recovery target
I'm wondering if this warning is a bit misleading to users because it is
an expected message when starting up an instance based on a slave backup
(because the slave was already in recovery mode). If I shutdown this
instance and start it up again I keep getting the warning. My
understanding of your patch is that there shouldn't be any risk of
corruption in that case (assuming your patch has no bugs). Can/should we
be suppressing this message when we detect that we are recovering from a
slave backup?
This has not been supported yet.
I do not see what state of this message.
Always happens when backup is taken from slave.
What do you think about an approach to add context, unless take backup from
slave?
The direction of the patch has changed a bit during this commit fest. I
think it would be good to provide an update on the rest of the changes
you plan for this to be a complete useable feature. That would make it
easier to comment on something you
missed versus something your planning on dealing with in the next stage.
I see.
I will provide a patch which can exeute pg_start/stop_backup
including to solve above comment and conditions in next stage.
Then please review.
I change this patch status to Returned with feedback.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_04.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
As you proposed, adding new field which stores the backup end location taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_03.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
What about using backupStartPoint to check whether this recovery started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup from). Once recovery started, backupStartPoint is recorded to pg_control until recovery reaches backup end location, it is not related to any backup server. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
When the standby restarts after it crashes during recovery, it always checks whether recovery has reached the backup end location by using minRecoveryPoint even though the standby doesn't start from the backup. This looks odd. Certainly. But, in this case, the state before recovery starts is lost. Therefore, postgres can not see that the backup got from whether standby server or master. What should? Should use pg_control? Ex. * Add 'Where to get backup' to pg_control. (default 'none') * When recovery starts, it checks it whether 'none'. * When minRecoveryPoint equals 0/0, change 'master'. * When minRecoveryPoint do not equals 0/0, change 'slave'. * When it reached the end of recovery, change 'none' . - XLogRecPtrIsInvalid(ControlFile-backupStartPoint)) + (XLogRecPtrIsInvalid(ControlFile-backupStartPoint) || + reachedControlMinRecoveryPoint == true)) The flag 'reachedControlMinRecoveryPoint' is really required? When recovery reaches minRecoveryPoint, ControlFile-backupStartPoint is reset to zero. So we can check whether recovery has reached minRecoveryPoint or not by only doing XLogRecPtrIsInvalid(ControlFile-backupStartPoint). No? Yes. 'reachedControlMinRecoveryPoint' is unnecessary. We should check if recovery has reached minRecoveryPoint before calling CheckRecoveryConsistency() after reading new WAL record? Otherwise, even if recovery has reached minRecoveryPoint, the standby cannot think that it's in consistent state until it reads new WAL record. This is a same sequence with a case of backup end location. It should be no changed. + if (XLByteLT(ControlFile-minRecoveryPoint, EndRecPtr)) + ControlFile-minRecoveryPoint = EndRecPtr; Why does ControlFile-minRecoveryPoint need to be set to EndRecPtr? Yes. I delete it. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*
Who deletes the backup_label file created by pg_start_backup()?
Isn't pg_stop_backup() required to do that?
You need it to take the system out of backup mode as well, don't you?
Certainly so.
Add to the above process:
4. pg_stop_backup();
But I do not consider a case such as to promote in backup mode yet.
I need to think a little, including it.
On this commitfest, the goal of the patch is to be able to be
recovered using Minimum recovery ending location in the control file.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
On this commitfest, the goal of the patch is to be able to be recovered using Minimum recovery ending location in the control file. Done. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_02.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Considering everything that has been discussed on this thread so far.
Do you still think your patch is the best way to accomplish base backups
from standby servers?
If not what changes do you think should be made?
I reconsider the way to not use pg_stop_backup().
Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*
Behavior while restore:
* read Minimum recovery ending location of the copied pg_control.
* use the value with the same purposes as the end-of-backup location.
- When the value is equal to 0/0, this behavior do not do.
This situation is to acquire backup from master server.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
1) Today you can do a backup by just calling pg_start_backup('x'); copy
the data directory and
pg_stop_backup(); You do not need to use pg_basebackup to create a
backup. The solution you are proposing would require pg_basebackup to be
used to build backups from standby servers.
YES.
2) If I run pg_basebackup but do not specify '-x' then no pg_xlog
segments are included in the output. The relevant pg_xlog segments are
in the archive from the master. I can see situations where you are
already copying the archive to the remote site that the new standby will
be created in so you don't want to have to copy the pg_xlog segments
twice over your network.
No, I don't matter because of the same behavior as 9.1.
Please see the part of Before: of the following answer.
What Heikki is proposing will work both when you aren't using
pg_basebackup (as long the output of pg_stop_backup() is somehow
captured in a way that it can be read) and will also work with
pg_basebackup when '-x' isn't specified.
I receive this mail, so I notice I do wrong recognition to what
Heikki is proposing.
my recognition:
Before:
* I thought Heikki proposes, Execute SQL(pg_start_backup('x'); copy
the data directory and pg_stop_backup();) from the standby server
to the primary server.
- I disliked this.
Now:
* Heikki is proposing both No using pg_basebackup and Not specify -x.
So,
* Use the output of pg_stop_backup().
* Don't use backup history file.
he thinks.
Right?
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
What I think he is proposing would not require using pg_stop_backup() but you could also modify pg_stop_back() to work as well. What do you think of this idea? Do you see how the patch can be reworked to accomplish this? The logic that not use pg_stop_backup() would be difficult, because pg_stop_backup() is used to identify minRecoveryPoint. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
( Quotation from http://archives.postgresql.org/pgsql-hackers/2011-05/msg01396.php ) STEP1: Make startup process to acquire backup-end-position from not only backup-end record but also backup-history-file . * startup process allows to acquire backup-end-position from backup-history-file . I have created a patch to the above-mentioned content. Please check it. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_01.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I still think that's headed in the wrong direction. (http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php) Please check these mails, and teach the reason for content of the wrong direction. (http://archives.postgresql.org/pgsql-hackers/2011-06/msg00209.php) (http://archives.postgresql.org/pgsql-hackers/2011-05/msg01566.php) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I don't much like that approach. The standby would need to be able to write the backup history file to the archive at the end of backup, and we'd have to reintroduce the code to fetch it from archive and, when streaming, from the master. At the moment, the archiver doesn't even run in the standby. Please teach the reason for The standby would need to be able to write the backup history file to the archive at the end of backup . (I'd like to know why to only pg_xlog is wrong .) If the backup history file is not archived, the postgres process won't find it when you try to restore from the base backup. The new server has no access to the standby's pg_xlog directory. Thanks for the answer . But , it sends the backup history file to pg_xlog of new server (=backup server) when pg_basebackup is executed to the standby server , and so I was going to create the patch of such logic . I think it don't become the above-mentioned movement . Because there is the opinion of Cascade replication , I don't want to realize the function with the method which the standby requests to execute it on the primary server . (The opinion of Cascade replication： http://archives.postgresql.org/pgsql-hackers/2011-05/msg01150.php) I don't see how this helps. Hypothesis: * Online base backup was realized with the method which the standby requests to execute it on the primary server . * Cascade replication was developed , and user is using it . (Ex. Primary -- Standby1 -- Standby2) Situation: (1) Standby2 executes pg_basebackup . (2) Then, Standby2 accesses Standby1 . (3) But, it fails, because Standby2's primary is Standby1, not Primary . Result: * I don't want to realize the function with the method which the standby requests to execute it on the primary server . Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I don't much like that approach. The standby would need to be able to write the backup history file to the archive at the end of backup, and we'd have to reintroduce the code to fetch it from archive and, when streaming, from the master. At the moment, the archiver doesn't even run in the standby. Please teach the reason for The standby would need to be able to write the backup history file to the archive at the end of backup . (I'd like to know why to only pg_xlog is wrong .) Because there is the opinion of Cascade replication , I don't want to realize the function with the method which the standby requests to execute it on the primary server . (The opinion of Cascade replication： http://archives.postgresql.org/pgsql-hackers/2011-05/msg01150.php) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
Hi I would like to develop function for 'Online base backup from the hot-standby' in PostgreSQL 9.2. Todo : Allow hot file system backups on standby servers (http://wiki.postgresql.org/wiki/Todo) [GOAL] * Make pg_basebackup to execute to the hot-standby server and acquire online-base-backup . - pg_basebackup can be executed to only primary server in PostgreSQL 9.1 . - But physical-copy(etc) under processing of pg_basebackup raises the load of primary server . - Therefore , this function is necessary . [Problem] (There is the following problems when hot-standby acquires online-base-backup like executing pg_basebackup to the primary server .) * pg_start_backup() and pg_stop_backup() can't be executed to the hot-standby server . - hot-standby can't insert backup-end record to WAL-files and can't operate CHECKPOINT . - Because hot-standby can't write anything in WAL-files . * hot-standby can't send WAL-files to archive server. - when pg_stop_backup() is executed to the primary server , it waits for completing sending wal to archive server , but hot-standby can't do it. [Policy] (I create with the following Policy .) * This function doesn't affect primary server . - I don't adopt the way which hot-standby requests primary to execute pg_basebackup , because I think about many standbys is connected with a primary . [Approach] * When pg_basebackup is executed to the hot-standby server , it executes RESTARTPOINT instead of CHECKPOINT . backup_label is made from the RESTARTPOINT's results , and is sent to the designated backup server using pg_basebackup connection . * Instead of inserting backup-end record , hot-standby writes backup-end-position in backup-history-file and sends to the designated backup server using pg_basebackup connection . - In 9.1 , startup process knows backup-end-position from only backup-end record . In addition to its logic, startup process can know backup-end-position from backup-history-file . As a result , startup process can recovery certainly without backup-end record . [Precondition] (As a result of the above-mentioned Policy and Approach , there is the following restrictions .) * Immediately after backup starting of WAL must contain full page writes . But the above-mentioned Approach can't satisfy the restriction according to circumstances . Because full_page_writes of primary might equal 'off' . When standby recovery WAL which is removed full page writes by pg_lesslog , it is the same . * Because recovery starts from last CHECKPOINT , it becomes long . * I has not thought new process that become taking the place of waiting for completing sending wal to archive server , yet. [Working Step] STEP1: Make startup process to acquire backup-end-position from not only backup-end record but also backup-history-file . * startup process allows to acquire backup-end-position from backup-history-file . * When pg_basebackup is executed , backup-history-file is sent to the designated backup server . STEP2: Make pg_start_backup() and pg_stop_backup() to be executed by the hot-standby server. [Action until The first CommitFest (on June 15)] I will create a patch to STEP1 . (The patch will be able to settle a problem of Omnipitr-backup-slave.) (a problem of Omnipitr-backup-slave : http://archives.postgresql.org/pgsql-hackers/2011-03/msg01490.php) * Shedule of creating STEP2 is the next CommitFest (in September 15) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
