Re: [HACKERS] Allow wal_keep_segments to keep all segments
Simon Riggs wrote: On Wed, 2010-06-02 at 20:28 -0400, Bruce Momjian wrote: Simon Riggs wrote: On Wed, 2010-06-02 at 15:20 -0400, Bruce Momjian wrote: The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. It's not clear to me why keep all files until server breaks is a good setting. Surely you would set this parameter to the size of your disk. Why allow it to go higher? Well, the -1 allows them to set it temporarily without having to compute their free disk space. Frankly, because the disk space varies, it is impossible to know exactly how large the disk is at the time it would fill up. I think the normal computation would be: 1) How long is my file system backup and restore to standby going to take 2) How often do I generate a 16MB WAL file You would do some computation to figure that out, then maybe multiply it by 10x and set that for wal_keep_segments. I figured allowing a simple -1 would be easier. I think its much easier to find out your free disk space than it is to calculate how much WAL might be generated during backup. Disk space doesn't vary significantly on a production database. If we encourage that laziness then we will get reports that replication doesn't work and Postgres crashes. Well, we don't clean out the archive directory so I don't see this as anything new. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
On 03/06/10 15:15, Bruce Momjian wrote: Simon Riggs wrote: I think its much easier to find out your free disk space than it is to calculate how much WAL might be generated during backup. Disk space doesn't vary significantly on a production database. If we encourage that laziness then we will get reports that replication doesn't work and Postgres crashes. Well, we don't clean out the archive directory so I don't see this as anything new. We leave that up to the DBA to clean out one way or another. We provide restartpoint_command and the %r option in restore_command to help with that. Surely we don't expect DBAs to delete old files in pg_xlog? I agree with Simon here, I think it would be better to not provide -1 as an option here. At least you better document well that you should only do that temporarily or you will eventually run out of disk space. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Heikki Linnakangas wrote: On 03/06/10 15:15, Bruce Momjian wrote: Simon Riggs wrote: I think its much easier to find out your free disk space than it is to calculate how much WAL might be generated during backup. Disk space doesn't vary significantly on a production database. If we encourage that laziness then we will get reports that replication doesn't work and Postgres crashes. Well, we don't clean out the archive directory so I don't see this as anything new. We leave that up to the DBA to clean out one way or another. We provide restartpoint_command and the %r option in restore_command to help with that. Surely we don't expect DBAs to delete old files in pg_xlog? I agree with Simon here, I think it would be better to not provide -1 as an option here. At least you better document well that you should only do that temporarily or you will eventually run out of disk space. Using this only temporarily is mentioned in the doc patch. Do I need more? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + Index: doc/src/sgml/config.sgml === RCS file: /cvsroot/pgsql/doc/src/sgml/config.sgml,v retrieving revision 1.280 diff -c -c -r1.280 config.sgml *** doc/src/sgml/config.sgml 31 May 2010 15:50:48 - 1.280 --- doc/src/sgml/config.sgml 2 Jun 2010 19:19:18 - *** *** 1887,1893 Specifies the number of past log file segments kept in the filenamepg_xlog/ directory, in case a standby server needs to fetch them for streaming ! replication. Each segment is normally 16 megabytes. If a standby server connected to the primary falls behind by more than varnamewal_keep_segments/ segments, the primary might remove a WAL segment still needed by the standby, in which case the --- 1887,1893 Specifies the number of past log file segments kept in the filenamepg_xlog/ directory, in case a standby server needs to fetch them for streaming ! replication. Each segment is normally 16 megabytes. If a standby server connected to the primary falls behind by more than varnamewal_keep_segments/ segments, the primary might remove a WAL segment still needed by the standby, in which case the *** *** 1901,1908 is zero (the default), the system doesn't keep any extra segments for standby purposes, and the number of old WAL segments available for standbys is determined based only on the location of the previous ! checkpoint and status of WAL archiving. ! This parameter can only be set in the filenamepostgresql.conf/ file or on the server command line. /para /listitem --- 1901,1909 is zero (the default), the system doesn't keep any extra segments for standby purposes, and the number of old WAL segments available for standbys is determined based only on the location of the previous ! checkpoint and status of WAL archiving. If literal-1/ is ! specified, log file segments are kept indefinitely. This ! parameter can only be set in the filenamepostgresql.conf/ file or on the server command line. /para /listitem Index: doc/src/sgml/high-availability.sgml === RCS file: /cvsroot/pgsql/doc/src/sgml/high-availability.sgml,v retrieving revision 1.70 diff -c -c -r1.70 high-availability.sgml *** doc/src/sgml/high-availability.sgml 29 May 2010 09:01:10 - 1.70 --- doc/src/sgml/high-availability.sgml 2 Jun 2010 19:19:19 - *** *** 750,756 If you use streaming replication without file-based continuous archiving, you have to set varnamewal_keep_segments/ in the master to a value high enough to ensure that old WAL segments are not recycled ! too early, while the standby might still need them to catch up. If the standby falls behind too much, it needs to be reinitialized from a new base backup. If you set up a WAL archive that's accessible from the standby, wal_keep_segments is not required as the standby can always --- 750,760 If you use streaming replication without file-based continuous archiving, you have to set varnamewal_keep_segments/ in the master to a value high enough to ensure that old WAL segments are not recycled ! too early, while the standby might still need them to catch up. This ! is particularly important when performing a base backup because the ! standby will need all WAL segments generated since the start of the ! backup; consider setting varnamewal_keep_segments/ to ! literal-1/ temporarily in such cases. If the
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Heikki Linnakangas wrote:
On 03/06/10 15:15, Bruce Momjian wrote:
Simon Riggs wrote:
I think its much easier to find out your free disk space than it is to
calculate how much WAL might be generated during backup. Disk space
doesn't vary significantly on a production database.
If we encourage that laziness then we will get reports that replication
doesn't work and Postgres crashes.
Well, we don't clean out the archive directory so I don't see this as
anything new.
We leave that up to the DBA to clean out one way or another. We provide
restartpoint_command and the %r option in restore_command to help with that.
Surely we don't expect DBAs to delete old files in pg_xlog? I agree with
Simon here, I think it would be better to not provide -1 as an option
here. At least you better document well that you should only do that
temporarily or you will eventually run out of disk space.
I have updated the doc text to mention temporarily everywhere '-1' is
mentioned.
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ None of us is going to be here forever. +
Index: doc/src/sgml/config.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/config.sgml,v
retrieving revision 1.280
diff -c -c -r1.280 config.sgml
*** doc/src/sgml/config.sgml31 May 2010 15:50:48 - 1.280
--- doc/src/sgml/config.sgml3 Jun 2010 14:05:21 -
***
*** 1901,1908
is zero (the default), the system doesn't keep any extra segments
for standby purposes, and the number of old WAL segments available
for standbys is determined based only on the location of the previous
! checkpoint and status of WAL archiving.
! This parameter can only be set in the filenamepostgresql.conf/
file or on the server command line.
/para
/listitem
--- 1901,1909
is zero (the default), the system doesn't keep any extra segments
for standby purposes, and the number of old WAL segments available
for standbys is determined based only on the location of the previous
! checkpoint and status of WAL archiving. To temporarily keep
! all log file segments, use the value literal-1/. This
! parameter can only be set in the filenamepostgresql.conf/
file or on the server command line.
/para
/listitem
Index: doc/src/sgml/high-availability.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/high-availability.sgml,v
retrieving revision 1.70
diff -c -c -r1.70 high-availability.sgml
*** doc/src/sgml/high-availability.sgml 29 May 2010 09:01:10 - 1.70
--- doc/src/sgml/high-availability.sgml 3 Jun 2010 14:05:21 -
***
*** 750,756
If you use streaming replication without file-based continuous
archiving, you have to set varnamewal_keep_segments/ in the master
to a value high enough to ensure that old WAL segments are not recycled
! too early, while the standby might still need them to catch up. If the
standby falls behind too much, it needs to be reinitialized from a new
base backup. If you set up a WAL archive that's accessible from the
standby, wal_keep_segments is not required as the standby can always
--- 750,760
If you use streaming replication without file-based continuous
archiving, you have to set varnamewal_keep_segments/ in the master
to a value high enough to ensure that old WAL segments are not recycled
! too early, while the standby might still need them to catch up. This
! is particularly important when performing a base backup because the
! standby will need all WAL segments generated since the start of the
! backup; consider setting varnamewal_keep_segments/ to
! literal-1/ temporarily in such cases. If the
standby falls behind too much, it needs to be reinitialized from a new
base backup. If you set up a WAL archive that's accessible from the
standby, wal_keep_segments is not required as the standby can always
Index: src/backend/access/transam/xlog.c
===
RCS file: /cvsroot/pgsql/src/backend/access/transam/xlog.c,v
retrieving revision 1.415
diff -c -c -r1.415 xlog.c
*** src/backend/access/transam/xlog.c 2 Jun 2010 09:28:44 - 1.415
--- src/backend/access/transam/xlog.c 3 Jun 2010 14:05:22 -
***
*** 7337,7343
* Delete old log files (those no longer needed even for previous
* checkpoint or the standbys in XLOG streaming).
*/
! if (_logId || _logSeg)
{
/*
* Calculate the last segment that we need to retain because of
--- 7337,7343
* Delete old log files
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Bruce Momjian br...@momjian.us writes: Heikki Linnakangas wrote: Surely we don't expect DBAs to delete old files in pg_xlog? I agree with Simon here, I think it would be better to not provide -1 as an option here. At least you better document well that you should only do that temporarily or you will eventually run out of disk space. I have updated the doc text to mention temporarily everywhere '-1' is mentioned. FWIW, I've come to agree with Simon. Allowing -1 doesn't do anything that you can't do with a large positive setting, and what it does do is to encourage people to set the variable to an unsafe value as a substitute for thinking. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Excerpts from Bruce Momjian's message of jue jun 03 08:36:28 -0400 2010: Using this only temporarily is mentioned in the doc patch. Do I need more? Yeah, it's far too easy to miss. Besides, I think the wording you used is ambiguous -- it can be read as the server will temporarily keep all segments if you set it to -1, which is not the same thing at all. If you can't add a 20-point-font red blinking warning with a pink dancing elephant in a tutu, maybe it's best to not offer the dangerous setting in the first place. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of jue jun 03 08:36:28 -0400 2010: Using this only temporarily is mentioned in the doc patch. Do I need more? Yeah, it's far too easy to miss. Besides, I think the wording you used is ambiguous -- it can be read as the server will temporarily keep all segments if you set it to -1, which is not the same thing at all. If you can't add a 20-point-font red blinking warning with a pink dancing elephant in a tutu, maybe it's best to not offer the dangerous setting in the first place. Well, it seems enough people don't want this features that I am not going to add it. If we decide we want it later, we can add it. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Heikki Linnakangas wrote: We leave that up to the DBA to clean out one way or another. We provide restartpoint_command and the %r option in restore_command to help with that. I was in fact just looking into this, and I see that there is no example restartpoint_comand script given in the docs, nor in the wiki. A sample of such a command would be useful. This is all going to feel a bit strange to lots of users, and the more we can hold their hands the better off we and they will be. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
On Wed, Jun 2, 2010 at 3:20 PM, Bruce Momjian br...@momjian.us wrote: Bruce Momjian wrote: Robert Haas wrote: On Sat, May 8, 2010 at 10:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Uh, did we decide that 'wal_keep_segments' was the best name for this GUC setting? ?I know we shipped beta1 using that name. I thought min_wal_segments was a reasonable proposal, but it wasn't clear if there was consensus or not. I think most people thought it was another reasonable choice, but I think the consensus position is probably something like it's about the same rather than it's definitely better. We had one or two people with stronger opinions than that on either side, I believe. Agreed the current name seems OK. However, was there agreement that wal_keep_segments = -1 should keep all WAL segements? I can see that as useful for cases where you are doing a dump to be transfered to the slave, and not using archive_command. This avoids the need for the set a huge value solution. The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. +1 for the patch, but why wait until after beta2? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
On Wed, 2010-06-02 at 15:20 -0400, Bruce Momjian wrote: The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. It's not clear to me why keep all files until server breaks is a good setting. Surely you would set this parameter to the size of your disk. Why allow it to go higher? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Simon Riggs wrote: On Wed, 2010-06-02 at 15:20 -0400, Bruce Momjian wrote: The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. It's not clear to me why keep all files until server breaks is a good setting. Surely you would set this parameter to the size of your disk. Why allow it to go higher? Well, the -1 allows them to set it temporarily without having to compute their free disk space. Frankly, because the disk space varies, it is impossible to know exactly how large the disk is at the time it would fill up. I think the normal computation would be: 1) How long is my file system backup and restore to standby going to take 2) How often do I generate a 16MB WAL file You would do some computation to figure that out, then maybe multiply it by 10x and set that for wal_keep_segments. I figured allowing a simple -1 would be easier. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
Robert Haas wrote: The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. ?I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. +1 for the patch, but why wait until after beta2? I wanted to give people enough time to review/discuss it. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Allow wal_keep_segments to keep all segments
On Wed, 2010-06-02 at 20:28 -0400, Bruce Momjian wrote: Simon Riggs wrote: On Wed, 2010-06-02 at 15:20 -0400, Bruce Momjian wrote: The attached patch allows wal_keep_segments = -1 to keep all segements; this is particularly useful for taking a base backup, where you need all the WAL files during startup of the standby. I have documented this usage in the patch as well. I am thinking of applying this after 9.0 beta2 if there is no objection. It's not clear to me why keep all files until server breaks is a good setting. Surely you would set this parameter to the size of your disk. Why allow it to go higher? Well, the -1 allows them to set it temporarily without having to compute their free disk space. Frankly, because the disk space varies, it is impossible to know exactly how large the disk is at the time it would fill up. I think the normal computation would be: 1) How long is my file system backup and restore to standby going to take 2) How often do I generate a 16MB WAL file You would do some computation to figure that out, then maybe multiply it by 10x and set that for wal_keep_segments. I figured allowing a simple -1 would be easier. I think its much easier to find out your free disk space than it is to calculate how much WAL might be generated during backup. Disk space doesn't vary significantly on a production database. If we encourage that laziness then we will get reports that replication doesn't work and Postgres crashes. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
