Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > On Sat, May 8, 2010 at 10:40 PM, Tom Lane wrote: > > Bruce Momjian writes: > >> Uh, did we decide that 'wal_keep_segments' was the best name for this > >> GUC setting? ?I know we shipped beta1 using that name. > > > > I thought min_wal_segments was a reasonable proposal, but it wasn't > > clear if there was consensus or not. > > I think most people thought it was another reasonable choice, but I > think the consensus position is probably something like "it's about > the same" rather than "it's definitely better". We had one or two > people with stronger opinions than that on either side, I believe. Agreed the current name seems OK. However, was there agreement that wal_keep_segments = -1 should keep all WAL segements? I can see that as useful for cases where you are doing a dump to be transfered to the slave, and not using archive_command. This avoids the need for the "set a huge value" solution. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Sat, 2010-05-08 at 23:55 -0400, Robert Haas wrote: > On Sat, May 8, 2010 at 10:40 PM, Tom Lane wrote: > > Bruce Momjian writes: > >> Uh, did we decide that 'wal_keep_segments' was the best name for this > >> GUC setting? I know we shipped beta1 using that name. > > > > I thought min_wal_segments was a reasonable proposal, but it wasn't > > clear if there was consensus or not. > > I think most people thought it was another reasonable choice, but I > think the consensus position is probably something like "it's about > the same" rather than "it's definitely better". We had one or two > people with stronger opinions than that on either side, I believe. It's only a name and not worth a long discussion on. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Sat, May 8, 2010 at 10:40 PM, Tom Lane wrote: > Bruce Momjian writes: >> Uh, did we decide that 'wal_keep_segments' was the best name for this >> GUC setting? I know we shipped beta1 using that name. > > I thought min_wal_segments was a reasonable proposal, but it wasn't > clear if there was consensus or not. I think most people thought it was another reasonable choice, but I think the consensus position is probably something like "it's about the same" rather than "it's definitely better". We had one or two people with stronger opinions than that on either side, I believe. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian writes: > Uh, did we decide that 'wal_keep_segments' was the best name for this > GUC setting? I know we shipped beta1 using that name. I thought min_wal_segments was a reasonable proposal, but it wasn't clear if there was consensus or not. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian wrote: > Simon Riggs wrote: > > On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: > > > > > > > > (wal_keep_segments can be changed without restarting, right?) > > > > > > Should we allow -1 to mean "keep all segments"? > > > > Why is that not called "max_wal_segments"? wal_keep_segments sounds like > > its been through Google translate. > > LOL, good one. > > I assume it was done so it would start with 'wal', but I see > 'max_wal_senders', which doesn't start with 'wal' and would match your > suggestion exactly. I think we should either rename 'wal_keep_segments' > or 'max_wal_senders'. Uh, did we decide that 'wal_keep_segments' was the best name for this GUC setting? I know we shipped beta1 using that name. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
> It's really both of those things, so we could call it > wal_min_keep_segments, but I think an even better name would be > bikeshed_segments. Speaking from my UI perspective, I don't think users will care what we call it. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Mon, May 3, 2010 at 2:54 PM, Kevin Grittner wrote: > Simon Riggs wrote: >> On Fri, 2010-04-30 at 13:41 -0500, Kevin Grittner wrote: > >>> Surely it would confuse people to see they have fewer than >>> min_wal_segments WAL segments. >> >> That does sound like a reasonable argument, though it also applies >> to wal_keep_segments, so isn't an argument either way. The user >> will be equally confused to see fewer WAL files than they have >> asked to "keep". > > The definitions of "keep" in my dictionary include "to restrain from > removal" and "to retain in one's possession". It defines "minimum" > as "the least quantity assignable, admissible, or possible". It's really both of those things, so we could call it wal_min_keep_segments, but I think an even better name would be bikeshed_segments. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Fri, 2010-04-30 at 13:41 -0500, Kevin Grittner wrote: >> Surely it would confuse people to see they have fewer than >> min_wal_segments WAL segments. > > That does sound like a reasonable argument, though it also applies > to wal_keep_segments, so isn't an argument either way. The user > will be equally confused to see fewer WAL files than they have > asked to "keep". The definitions of "keep" in my dictionary include "to restrain from removal" and "to retain in one's possession". It defines "minimum" as "the least quantity assignable, admissible, or possible". If I'm understanding the semantics of this GUC (which I'll grant is not a sure thing), "keep" does a better job of conveying the meaning, since fewer than that are initially possible, but at least that many will be *kept* once they exist. I'm sure I'll figure it out at need, but the assertions that "minimum" more clearly defines the purpose is shaking *my* confidence that I understand what the GUC is for. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, 2010-04-30 at 13:41 -0500, Kevin Grittner wrote: > Heikki Linnakangas wrote: > > > Yeah, min_wal_segments or something would make sense. > > Surely it would confuse people to see they have fewer than > min_wal_segments WAL segments. That does sound like a reasonable argument, though it also applies to wal_keep_segments, so isn't an argument either way. The user will be equally confused to see fewer WAL files than they have asked to "keep". min_wal_segments is much clearer, IMHO. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas wrote: > Kevin Grittner wrote: >> Surely it would confuse people to see they have fewer than >> min_wal_segments WAL segments. > > they wouldn't see that, that's the point of the setting. I was thinking, in particular, about beginners poking around to see how things look after an initdb. Perhaps that state is too transient to matter, but it struck me that you'd have fewer than the minimum at the precise time a beginner might be likely to take a look. Unless on startup (and reload?) we created min_wal_segments WAL segments if they didn't already exist. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Kevin Grittner wrote: > Heikki Linnakangas wrote: > >> Yeah, min_wal_segments or something would make sense. > > Surely it would confuse people to see they have fewer than > min_wal_segments WAL segments. Umm, they wouldn't see that, that's the point of the setting. The segments are not removed/recycled until there is min_wal_segments segments in pg_xlog. Except in the beginning when you set or increase the setting, when there isn't that many segments generated yet. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, 2010-04-30 at 14:42 -0400, Tom Lane wrote: > Robert Haas writes: > > On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: > >> Why is that not called "max_wal_segments"? wal_keep_segments sounds like > >> its been through Google translate. > > > Because it's not a maximum? > > Indeed. It would really be more like min_wal_segments, if we wanted to > name it that way. Yeh, agreed: min_wal_segments. I realised while having dinner it was the opposite, so I'm pleased everybody else got there at same time. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Alvaro Herrera writes: > Bruce Momjian escribió: >> Which is where my 'wal_keep_segments = -1' idea came from. > Are you suggesting that -1 should mean "keep all segments that fit on > disk, but if creating a new segment fails with ENOSPC, recycle the > oldest one"? No, keep means keep. Even if there were some arguable use for "keep if you can", a scheme like that would render the machine unusable --- everything else on the same filesystem would be falling over. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Michael Tharp wrote: > On 04/30/2010 01:53 PM, Robert Haas wrote: >> >> Well, one of us is. Why would you want to retain all of your WAL logs >> in pg_xlog forever? > > To create or re-synchronize SR slaves, one could change > wal_keep_segments to -1, run a backup, wait for the slaves to catch up, > and change it back to the default. This way no segments would be deleted > until the system has reached a stable state. A slave can fall behind at any time, though. You would have to know to set wal_keep_segments to -1 before that happens. I've been thinking that in the future (read 9.1 or above), we would have a system for registering slaves in the primary server. The primary would keep track of how far each slave is, and refrain from removing WAL segments that it knows to be still needed by a slave. On the flip-side, the master wouldn't need to keep WAL around that it knows is no longer needed by any slaves. If someone has the energy, it would be possible to write a stand-alone application to do that too. It could serve old WAL files from the archive and rely recent ones from the real master. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian escribió: > Which is where my 'wal_keep_segments = -1' idea came from. Are you suggesting that -1 should mean "keep all segments that fit on disk, but if creating a new segment fails with ENOSPC, recycle the oldest one"? -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas writes: > Bruce Momjian wrote: >> Should we allow -1 to mean "keep all segments"? > Umm, you can't keep all segments around forever, can you? Surely you > have to recycle them sooner or later or you will run out of disk space. You couldn't use that as a permanent setting, but it can make sense as a transient setting, rather than having to guess how much WAL you'll need to keep while setting up a new standby. > In practice, you can just set wal_keep_segments to some ridiculously > high value to achieve the same result. True. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, Apr 30, 2010 at 2:08 PM, Simon Riggs wrote: > On Fri, 2010-04-30 at 13:52 -0400, Robert Haas wrote: >> On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: >> > On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: >> >> > >> >> > (wal_keep_segments can be changed without restarting, right?) >> >> >> >> Should we allow -1 to mean "keep all segments"? >> > >> > Why is that not called "max_wal_segments"? wal_keep_segments sounds like >> > its been through Google translate. >> >> Because it's not a maximum? > > I see the thinking, but why would you ever set it to be something that > is *less* than the existing numbers? That would be pointless and indeed, > does nothing. The only time you touch it at all is when you set it to be > a value higher than the number of files that would normally be kept, and > when that is the case it *will* be the maximum. > > So I say, max_wal_segments = 0 (default) meaning no limit, we just > rotate as needed. We put a comment in the docs to say that if a value is > selected less than 2*checkpoint_segments+1 then the value is overridden. As you were quick to point out to me earlier this week, I am not an expert on our write-ahead logging system; however, I think you are mistaken. Perhaps Heikki could speak to the point more definitively, but I believe that the number of segments that the system retains for WAL archiving or crash recovery is variable. The purpose of this variable is to put a floor under the number of segments that are retained so that SR slaves can catch up if they fall behind. Of course, if archiving is configured, they can do that anyway using restore_command, but you might be running SR without archiving, or you might just want to set this to a small value so that the slaves don't have to keep switching between SR and archive recovery if segments get archived or checkpointed away at inconvenient times. It doesn't make a whole lot of sense to set the floor on the number of segments retained to positive infinity, except in one specific case: archiving is disabled, and you're trying to hang on to enough segments in pg_xlog to take a hot backup. As Tom said, it would be nice to have a more elegant solution to that problem, but we can do that in a future release; it's not really the primary purpose of wal_keep_segments, anyway. It certainly would not be a good idea to make the default configuration "retain all WAL forever". If you did that, a user who sets up PostgreSQL and is not using SR or HS or hot backups will eventually and inevitably fill up their hard disk. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas wrote: > Bruce Momjian wrote: > > Tom Lane wrote: > >> Heikki Linnakangas writes: > >>> Tom Lane wrote: > If you aren't archiving then there's no guarantee that you'll still have > a continuous WAL series starting from the start of the backup. > >>> I wasn't really thinking of this use case, but you could set > >>> wal_keep_segments "high enough". > >> Ah. Okay, that seems like a workable approach, at least for people with > >> reasonably predictable WAL loads. We could certainly improve on it > >> later to make it more bulletproof, but it's usable now --- if we relax > >> the error checks. > >> > >> (wal_keep_segments can be changed without restarting, right?) > > > > Should we allow -1 to mean "keep all segments"? > > Umm, you can't keep all segments around forever, can you? Surely you > have to recycle them sooner or later or you will run out of disk space. > > I guess you could move that responsibility to a user-written script, but > we haven't traditionally encouraged or supported people to mess with the > contents of pg_xlog. That would require some more thinking IMHO, not 9.0 > material. > > In practice, you can just set wal_keep_segments to some ridiculously > high value to achieve the same result. Which is where my 'wal_keep_segments = -1' idea came from. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas writes: > On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: >> Why is that not called "max_wal_segments"? wal_keep_segments sounds like >> its been through Google translate. > Because it's not a maximum? Indeed. It would really be more like min_wal_segments, if we wanted to name it that way. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: WAL page magic number (was Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct)
Bruce Momjian wrote: > Heikki Linnakangas wrote: >> Tom Lane wrote: >>> Heikki Linnakangas writes: Browsing the history of XLOG_PAGE_MAGIC, it used to be incremented by one whenever the format changes, at least as long as it has been been in xlog_internal.h. It started at 0xD05B in 2005, and was incremented by one by each commit till 0xD062. But then the hot standby patch bumped it to 0xD166, and on March 19th Simon changed it to 0x9002, and on March 28th to 0x9003. Is there a plan somewhere on all that, or was it just random whacking? >>> Random whacking. Simon seems to have decided to try to make the number >>> mean something, but it never meant anything before. It's not a version >>> number, it's a magic identifier. We might need to avoid certain values >>> to prevent collisions with other file formats, so I'm suspicious of >>> trying to make it match up with some expectation. >> Ok, for better or worse, I whacked it back to the old series. > > Agreed. Making it match the version would be odd because you then have > t bump it for every major release. Not necessarily. It would make sense to me to just keep it the same if there has been no changes. The magic number would indicate the first version that used that format. I'm not advocating a switch to that system, but it would be sensible as well. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas wrote: > Robert Haas wrote: > > On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: > >> On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: > (wal_keep_segments can be changed without restarting, right?) > >>> Should we allow -1 to mean "keep all segments"? > >> Why is that not called "max_wal_segments"? wal_keep_segments sounds like > >> its been through Google translate. > > > > Because it's not a maximum? > > Yeah, min_wal_segments or something would make sense. It sounds about as > good or bad as wal_keep_segments to me. I admit I never liked "keep" but couldn't think of better wording. I do like the proposed wording better. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas wrote: > Yeah, min_wal_segments or something would make sense. Surely it would confuse people to see they have fewer than min_wal_segments WAL segments. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian wrote: > Tom Lane wrote: >> Heikki Linnakangas writes: >>> Tom Lane wrote: If you aren't archiving then there's no guarantee that you'll still have a continuous WAL series starting from the start of the backup. >>> I wasn't really thinking of this use case, but you could set >>> wal_keep_segments "high enough". >> Ah. Okay, that seems like a workable approach, at least for people with >> reasonably predictable WAL loads. We could certainly improve on it >> later to make it more bulletproof, but it's usable now --- if we relax >> the error checks. >> >> (wal_keep_segments can be changed without restarting, right?) > > Should we allow -1 to mean "keep all segments"? Umm, you can't keep all segments around forever, can you? Surely you have to recycle them sooner or later or you will run out of disk space. I guess you could move that responsibility to a user-written script, but we haven't traditionally encouraged or supported people to mess with the contents of pg_xlog. That would require some more thinking IMHO, not 9.0 material. In practice, you can just set wal_keep_segments to some ridiculously high value to achieve the same result. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: >> On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: (wal_keep_segments can be changed without restarting, right?) >>> Should we allow -1 to mean "keep all segments"? >> Why is that not called "max_wal_segments"? wal_keep_segments sounds like >> its been through Google translate. > > Because it's not a maximum? Yeah, min_wal_segments or something would make sense. It sounds about as good or bad as wal_keep_segments to me. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, 2010-04-30 at 13:58 -0400, Bruce Momjian wrote: > Robert Haas wrote: > > On Fri, Apr 30, 2010 at 1:39 PM, Bruce Momjian wrote: > > > Robert Haas wrote: > > >> On Fri, Apr 30, 2010 at 12:22 PM, Bruce Momjian wrote: > > >> > Tom Lane wrote: > > >> >> Heikki Linnakangas writes: > > >> >> > Tom Lane wrote: > > >> >> >> If you aren't archiving then there's no guarantee that you'll > > >> >> >> still have > > >> >> >> a continuous WAL series starting from the start of the backup. > > >> >> > > >> >> > I wasn't really thinking of this use case, but you could set > > >> >> > wal_keep_segments "high enough". > > >> >> > > >> >> Ah. ?Okay, that seems like a workable approach, at least for people > > >> >> with > > >> >> reasonably predictable WAL loads. ?We could certainly improve on it > > >> >> later to make it more bulletproof, but it's usable now --- if we relax > > >> >> the error checks. > > >> >> > > >> >> (wal_keep_segments can be changed without restarting, right?) > > >> > > > >> > Should we allow -1 to mean "keep all segments"? > > >> > > >> If that's what you want to do, use archive_mode. > > > > > > Uh, I assume that will require me to store the WAL files somewhere else, > > > rather than keeping them in /pg_xlog, which I thought was the goal. ?Am > > > I missing something? > > > > Well, one of us is. Why would you want to retain all of your WAL logs > > in pg_xlog forever? > > Well, this email thread mentioned a case where you needed to increase > wal_keep_segments to a sufficiently-high value, and of course figuring > out such a value is harder than just having a way of turning off > recycling with -1. I think the only sensible setting is "as big as my (available) disk space". Any higher and you're going to crash, any lower and you'll invalidate your backup for no reason. -1 emulates current behaviour, BTW Still think we should rename it, in which case 0 is same as "no maximum". -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, 2010-04-30 at 13:52 -0400, Robert Haas wrote: > On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: > > On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: > >> > > >> > (wal_keep_segments can be changed without restarting, right?) > >> > >> Should we allow -1 to mean "keep all segments"? > > > > Why is that not called "max_wal_segments"? wal_keep_segments sounds like > > its been through Google translate. > > Because it's not a maximum? I see the thinking, but why would you ever set it to be something that is *less* than the existing numbers? That would be pointless and indeed, does nothing. The only time you touch it at all is when you set it to be a value higher than the number of files that would normally be kept, and when that is the case it *will* be the maximum. So I say, max_wal_segments = 0 (default) meaning no limit, we just rotate as needed. We put a comment in the docs to say that if a value is selected less than 2*checkpoint_segments+1 then the value is overridden. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > On Fri, Apr 30, 2010 at 1:39 PM, Bruce Momjian wrote: > > Robert Haas wrote: > >> On Fri, Apr 30, 2010 at 12:22 PM, Bruce Momjian wrote: > >> > Tom Lane wrote: > >> >> Heikki Linnakangas writes: > >> >> > Tom Lane wrote: > >> >> >> If you aren't archiving then there's no guarantee that you'll still > >> >> >> have > >> >> >> a continuous WAL series starting from the start of the backup. > >> >> > >> >> > I wasn't really thinking of this use case, but you could set > >> >> > wal_keep_segments "high enough". > >> >> > >> >> Ah. ?Okay, that seems like a workable approach, at least for people with > >> >> reasonably predictable WAL loads. ?We could certainly improve on it > >> >> later to make it more bulletproof, but it's usable now --- if we relax > >> >> the error checks. > >> >> > >> >> (wal_keep_segments can be changed without restarting, right?) > >> > > >> > Should we allow -1 to mean "keep all segments"? > >> > >> If that's what you want to do, use archive_mode. > > > > Uh, I assume that will require me to store the WAL files somewhere else, > > rather than keeping them in /pg_xlog, which I thought was the goal. ?Am > > I missing something? > > Well, one of us is. Why would you want to retain all of your WAL logs > in pg_xlog forever? Well, this email thread mentioned a case where you needed to increase wal_keep_segments to a sufficiently-high value, and of course figuring out such a value is harder than just having a way of turning off recycling with -1. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On 04/30/2010 01:53 PM, Robert Haas wrote: Well, one of us is. Why would you want to retain all of your WAL logs in pg_xlog forever? ...Robert To create or re-synchronize SR slaves, one could change wal_keep_segments to -1, run a backup, wait for the slaves to catch up, and change it back to the default. This way no segments would be deleted until the system has reached a stable state. -- m. tharp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, Apr 30, 2010 at 1:39 PM, Bruce Momjian wrote: > Robert Haas wrote: >> On Fri, Apr 30, 2010 at 12:22 PM, Bruce Momjian wrote: >> > Tom Lane wrote: >> >> Heikki Linnakangas writes: >> >> > Tom Lane wrote: >> >> >> If you aren't archiving then there's no guarantee that you'll still >> >> >> have >> >> >> a continuous WAL series starting from the start of the backup. >> >> >> >> > I wasn't really thinking of this use case, but you could set >> >> > wal_keep_segments "high enough". >> >> >> >> Ah. ?Okay, that seems like a workable approach, at least for people with >> >> reasonably predictable WAL loads. ?We could certainly improve on it >> >> later to make it more bulletproof, but it's usable now --- if we relax >> >> the error checks. >> >> >> >> (wal_keep_segments can be changed without restarting, right?) >> > >> > Should we allow -1 to mean "keep all segments"? >> >> If that's what you want to do, use archive_mode. > > Uh, I assume that will require me to store the WAL files somewhere else, > rather than keeping them in /pg_xlog, which I thought was the goal. Am > I missing something? Well, one of us is. Why would you want to retain all of your WAL logs in pg_xlog forever? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, Apr 30, 2010 at 1:44 PM, Simon Riggs wrote: > On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: >> > >> > (wal_keep_segments can be changed without restarting, right?) >> >> Should we allow -1 to mean "keep all segments"? > > Why is that not called "max_wal_segments"? wal_keep_segments sounds like > its been through Google translate. Because it's not a maximum? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: > > > > > > (wal_keep_segments can be changed without restarting, right?) > > > > Should we allow -1 to mean "keep all segments"? > > Why is that not called "max_wal_segments"? wal_keep_segments sounds like > its been through Google translate. LOL, good one. I assume it was done so it would start with 'wal', but I see 'max_wal_senders', which doesn't start with 'wal' and would match your suggestion exactly. I think we should either rename 'wal_keep_segments' or 'max_wal_senders'. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: > > > > (wal_keep_segments can be changed without restarting, right?) > > Should we allow -1 to mean "keep all segments"? Why is that not called "max_wal_segments"? wal_keep_segments sounds like its been through Google translate. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > On Fri, Apr 30, 2010 at 12:22 PM, Bruce Momjian wrote: > > Tom Lane wrote: > >> Heikki Linnakangas writes: > >> > Tom Lane wrote: > >> >> If you aren't archiving then there's no guarantee that you'll still have > >> >> a continuous WAL series starting from the start of the backup. > >> > >> > I wasn't really thinking of this use case, but you could set > >> > wal_keep_segments "high enough". > >> > >> Ah. ?Okay, that seems like a workable approach, at least for people with > >> reasonably predictable WAL loads. ?We could certainly improve on it > >> later to make it more bulletproof, but it's usable now --- if we relax > >> the error checks. > >> > >> (wal_keep_segments can be changed without restarting, right?) > > > > Should we allow -1 to mean "keep all segments"? > > If that's what you want to do, use archive_mode. Uh, I assume that will require me to store the WAL files somewhere else, rather than keeping them in /pg_xlog, which I thought was the goal. Am I missing something? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Fri, Apr 30, 2010 at 12:22 PM, Bruce Momjian wrote: > Tom Lane wrote: >> Heikki Linnakangas writes: >> > Tom Lane wrote: >> >> If you aren't archiving then there's no guarantee that you'll still have >> >> a continuous WAL series starting from the start of the backup. >> >> > I wasn't really thinking of this use case, but you could set >> > wal_keep_segments "high enough". >> >> Ah. Okay, that seems like a workable approach, at least for people with >> reasonably predictable WAL loads. We could certainly improve on it >> later to make it more bulletproof, but it's usable now --- if we relax >> the error checks. >> >> (wal_keep_segments can be changed without restarting, right?) > > Should we allow -1 to mean "keep all segments"? If that's what you want to do, use archive_mode. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: WAL page magic number (was Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct)
Heikki Linnakangas wrote: > Tom Lane wrote: > > Heikki Linnakangas writes: > >> Browsing the history of XLOG_PAGE_MAGIC, it used to be incremented by > >> one whenever the format changes, at least as long as it has been been in > >> xlog_internal.h. It started at 0xD05B in 2005, and was incremented by > >> one by each commit till 0xD062. But then the hot standby patch bumped it > >> to 0xD166, and on March 19th Simon changed it to 0x9002, and on March > >> 28th to 0x9003. > > > >> Is there a plan somewhere on all that, or was it just random whacking? > > > > Random whacking. Simon seems to have decided to try to make the number > > mean something, but it never meant anything before. It's not a version > > number, it's a magic identifier. We might need to avoid certain values > > to prevent collisions with other file formats, so I'm suspicious of > > trying to make it match up with some expectation. > > Ok, for better or worse, I whacked it back to the old series. Agreed. Making it match the version would be odd because you then have t bump it for every major release. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Tom Lane wrote: > Heikki Linnakangas writes: > > Tom Lane wrote: > >> If you aren't archiving then there's no guarantee that you'll still have > >> a continuous WAL series starting from the start of the backup. > > > I wasn't really thinking of this use case, but you could set > > wal_keep_segments "high enough". > > Ah. Okay, that seems like a workable approach, at least for people with > reasonably predictable WAL loads. We could certainly improve on it > later to make it more bulletproof, but it's usable now --- if we relax > the error checks. > > (wal_keep_segments can be changed without restarting, right?) Should we allow -1 to mean "keep all segments"? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 16:18 -0400, Tom Lane wrote: > * rename recovery_connections to hot_standby (and the underlying > variable from XLogRequestRecoveryConnections to something like > EnableHotStandby); OK > * change its default value to 'off'. OK -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 4:18 PM, Tom Lane wrote: > This train is leaving the station. An enthusiastic +1 for train departure. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Okay, this thread seems to be running out of steam, and we are running out of hours before the beta1 wrap. I am going to take it on my own authority to do the following: * rename recovery_connections to hot_standby (and the underlying variable from XLogRequestRecoveryConnections to something like EnableHotStandby); * change its default value to 'off'. I believe the only other code changes needed for beta1 are the pg_start/stop_backup error check changes that Heikki proposed at <4bd953a6.70...@enterprisedb.com>. If he doesn't commit that within an hour or two, I will. This train is leaving the station. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas writes: > Tom Lane wrote: >> Yeah. ISTM the real bottom line here is that we have only a weak grasp >> on how these features will end up being used; or for that matter what >> the common error scenarios will be. I think that for the time being >> we should err on the side of being permissive. We can tighten things >> up and add more nanny-ism in the warnings later on, when we have >> more field experience. > Ok, here's a proposed patch. Per discussion, it relaxes the checks in > pg_start/stop_backup() so that they can be used as long as wal_level >= > 'archive', even if archiving is disabled. This patch seems reasonably noncontroversial (except possibly for message wording, which we can fine-tune later anyway). Please apply. 9.0beta1 is going to get wrapped in only a few hours. BTW, the documentation for these functions might need a bit of adjustment. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > [most significant extra WAL logging for hot standby is:] > * one extra WAL record every checkpoint, containing a full current > snapshot's worth of running xids 100-400 bytes typically, could go > up from there to 4000 bytes in very extreme write workloads that > also have many, many subtransactions The most convincing evidence that there's no significant performance hit for those not needing the feature (and not a bad thing from a testing point of view anyway) would be for someone to run some benchmarks comparing the archive and hot_standby logging levels, with no archive script or SR. If that hasn't been done yet, maybe you could find somebody who knows his way around pgbench-tools, who could construct a reasonable test and produce graphs and all that cool stuff. ;-) IMO, isolating the cost of generating and writing the extra WAL would be very valuable, and it would be reassuring to know that it worked with a large number of clients without exposing any dire race conditions. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
"Kevin Grittner" writes: > Simon Riggs wrote: >> 9.0 without HS enabled is fairly untested on the standby, so it's >> not clear to me turning it off makes you any safer. > I think you've just made the strongest possible case for not > defaulting it on during beta testing. Yes. If we've had it on so far, flipping it at the start of beta seems sensible from a test coverage standpoint. We will certainly have a reasonable number of beta testers who are interested in the feature and turn it on to test it; but we need to make sure the "off" position isn't accidentally broken. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 21:27 +0300, Heikki Linnakangas wrote: > I wasn't really following the join removal discussions, but it seems > much safer and less complex. And it would not have been feasible to have > both implementations of FSM around and have the user to choose. I have > been relieved by the lack of bug reports on the new FSM, that was a big > change that impacted all installations. I agree that Simon's argument doesn't hold a lot of water in comparison to the features he is citing. I say leave it on for Beta just so we can hopefully get some inherit testing but other than that, its off for release. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > What Heikki hasn't said is that setting max_prepared_transactions > incorrectly has always caused a fatal error at failover, and still does > so now when HS is disabled. Hmm, good point. That should probably be documented somewhere. Now that we WAL-log the max_prepared_xacts value, I suppose we could throw a WARNING too. > If these concerns were real ones, then that > situation would not have existed for years and would not still do so. There's very few people using two-phase commit out there, and even less so in a standby configuration. I wouldn't draw any conclusions from that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 13:01 -0500, Kevin Grittner wrote: > Simon Riggs wrote: > > > I've asked for evidence that recovery is any slower as a result of > > HS. > > Can you quantify the impact on the number of bytes written to WAL > when switching from the archiving level to the hot standby level? Sure, done that a few times. Extra WAL data is written for these actions, listed in order of increasing size * commit records contain a variable length list of relcacheinvalidations, mostly applies only to DDL * one extra WAL record in most VACUUMs, fairly small, optimised away in some cases * a transaction issues more than 64 subtransactions it will issue a record approx 256 bytes plus header * one extra WAL record every checkpoint, containing a full current snapshot's worth of running xids 100-400 bytes typically, could go up from there to 4000 bytes in very extreme write workloads that also have many, many subtransactions -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 19:55 +0300, Heikki Linnakangas wrote: >> It's safer to run >> with recovery_connections off if you don't need the feature. > > Presumably your advice is also that people should not run with Streaming > Replication if they don't need that feature? Umm, yes. Why would you bother to set it up if you don't need it? > And that we should also > have an enable_joinremoval flag so the risk it poses can be minimized? > You didn't argue this way with regard to vacuum/FSM in 8.4, which was > much more critical; we're just talking about a standby server. This is getting bizarre... I wasn't really following the join removal discussions, but it seems much safer and less complex. And it would not have been feasible to have both implementations of FSM around and have the user to choose. I have been relieved by the lack of bug reports on the new FSM, that was a big change that impacted all installations. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > 9.0 without HS enabled is fairly untested on the standby, so it's > not clear to me turning it off makes you any safer. I think you've just made the strongest possible case for not defaulting it on during beta testing. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 13:20 -0400, Robert Haas wrote: > This is a pretty good argument, and Heikki's argument just upthread > that a mismatched max_connections setting could bollix things is an > even better one. So I'm now changing my mind and thinking this should > be off by default, also. What Heikki hasn't said is that setting max_prepared_transactions incorrectly has always caused a fatal error at failover, and still does so now when HS is disabled. If these concerns were real ones, then that situation would not have existed for years and would not still do so. 9.0 without HS enabled is fairly untested on the standby, so it's not clear to me turning it off makes you any safer. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs writes: > Can we keep the default=on during beta and collect evidence before > making a final decision at release? I'd be for doing that if it didn't have adverse effects on the use of pg_start_backup (which is where we started this thread). But it would be to get more testing, not because we would consider beta results as even possibly giving sufficient confidence to justify shipping 9.0.0 with the default = on. I realize that you've sweated blood over a long period to get this stuff in there, and if I were you I'd probably be wishing it could be on by default too. But from a project management standpoint it's just way too risky to even consider. HS/SR at this point have to be seen as being about as trustworthy as the Windows port was in 8.0.0. No DBA is going to be happy with that stuff getting turned on unless he specifically asks for it. Maybe in 9.1 or 9.2, but not in 9.0. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 19:55 +0300, Heikki Linnakangas wrote: > And Aidan and me are human > beings, included in "everybody". Yes, that was too loose. I dislike that kind of argument and I'm sorry that slipped in here. > You know that there is cases where it causes problems in the standby, > even ignoring the possibility of bugs. For example, if you increase > max_connections in the master, the standby will abort. That behaviour was suggested by you. I don't think its anywhere near necessary that it does that and would like to remove that restriction. The likelihood we'll ever run out of slots is small even with large increases in max_connections. Example: if we increase from 100 to 500 we'd only hit the limit if we had all 500 connections on the master simultaneously running write transactions with an average 12 subtransactions each. > It's safer to run > with recovery_connections off if you don't need the feature. Presumably your advice is also that people should not run with Streaming Replication if they don't need that feature? And that we should also have an enable_joinremoval flag so the risk it poses can be minimized? You didn't argue this way with regard to vacuum/FSM in 8.4, which was much more critical; we're just talking about a standby server. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > I've asked for evidence that recovery is any slower as a result of > HS. Can you quantify the impact on the number of bytes written to WAL when switching from the archiving level to the hot standby level? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > Can we keep the default=on during beta and collect evidence before > making a final decision at release? That's tempting to get more (inadvertent) testing of HS, but I don't think it would be fair to the beta testers. The expectation is that what's in beta is what's in the release, unless some new issue pops up. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 12:16 -0500, Kevin Grittner wrote: > Simon Riggs wrote: > > > Which is what everybody has requested. > > You continue to use the term "everybody" rather loosely. Sorry, that was imprecise, I didn't mean to exaggerate. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 13:03 -0400, Tom Lane wrote: > There is not only a performance penalty, but a reliability penalty. > Enabling these switches turns on a whole lot of code that, with all > due respect to those who have worked on it, is absolutely positively > guaranteed to be full of bugs. Not all of which are going to be flushed > out during beta. If we ship 9.0 with these things on by default, it > will result in an immediate reliability downgrade for installations that > are simply doing what they did before and not even interested in HS/SR. > > Maybe by 9.1 or 9.2 it would be sensible to have some of this code > turned on by default. But it is absolutely not in keeping with this > project's mindset or historical practice to enable it by default now. > > regards, tom lane +1 Joshua D. Drake > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Aidan Van Dyk wrote: It's all about the path of least *suprise*. My "least suprise" would have been that a similar config I have now with my PITR slaves would pretty much still work, and not suddenly start accepting connections, and even worse, at some later date when I've already verified that it was doing what I expected with the configuration. The first time I setup a system to test HS, when I got to the point where the slave was up and running as a standard warm standby, I expected there to be something else I had to do in order for it to be available for queries. When I fired up psql and I was able to run queries without doing anything extra, I was surprised--but it was that fun, everything just works when I expected it to be harder than that kind of surprise. One of the reasons the version number was bumped up to 9.0 was to put people on warning that they should not assume their old setups would port forward without behavioral changes. The fact that existing warm-standby server users will be surprised to find they can run queries without doing anything special could be considered under that banner. If you feel that's not obvious enough, that could argue for more prominent documentation of that fact, rather than turning it off. The idea that it should be made harder to enable just to protect the expectations current users, and therefore introduce yet another place where PostgreSQL is less friendly to get started with than it could be, is backwards from the perspective of making things as easy as possible for new users. Arguing from a usability standpoint needs to consider both new and existing user requirements, and those are quite opposed to one another in terms of what default makes more sense IMHO. Now, if the argument is from the perspective of "this adds performance/reliability issues that weren't there before", and those go away if the feature is disabled by default, that's a respectable and indisputable reason to do so. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 13:03 -0400, Tom Lane wrote: > There is not only a performance penalty I've asked for evidence that recovery is any slower as a result of HS. If we had some, I'm guessing we'd be tuning rather than discussing this. > Enabling these switches turns on a whole lot of code that, with all > due respect to those who have worked on it, is absolutely positively > guaranteed to be full of bugs. I understand that view and don't take offence at all, thank you for your concern. I wish to take seriously the possibility you may be correct, so I want to squash unfounded rumours that will scare people into keeping HS turned off, because that will keep the bugs in, rather than flush them out so we can fix them. Can we keep the default=on during beta and collect evidence before making a final decision at release? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 1:03 PM, Tom Lane wrote: >>> That said, I'ld probalby be happy with PG 9 having a "default" config >>> of: >>> wal_mode = hot_standby >>> recovery_connections = on > >> That would be a bad idea - there's a significant performance penalty >> from setting wal_level to anything other than minimal (just as there >> is for turning on archive_mode in 8.4). > > There is not only a performance penalty, but a reliability penalty. > Enabling these switches turns on a whole lot of code that, with all > due respect to those who have worked on it, is absolutely positively > guaranteed to be full of bugs. Not all of which are going to be flushed > out during beta. If we ship 9.0 with these things on by default, it > will result in an immediate reliability downgrade for installations that > are simply doing what they did before and not even interested in HS/SR. This is a pretty good argument, and Heikki's argument just upthread that a mismatched max_connections setting could bollix things is an even better one. So I'm now changing my mind and thinking this should be off by default, also. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > Which is what everybody has requested. You continue to use the term "everybody" rather loosely. I don't begrudge people features they want, even when *I* don't need them; but please don't take my lack of argument against adding this feature as a silent request for it. SR has a place in our shop, especially if it can create traditional WAL file segments on the receiving end. (I understand that will not be a built-in feature for 9.0, but perhaps there will be a pgfoundry project or some such.) HS is no use to us, and I would rather not pay a performance penalty or take the risks of exercising complex new code paths because others need it. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 11:48 -0500, Kevin Grittner wrote: > Simon Riggs wrote: > > > There is no big use case for "run with HS turned off". > > There was mention of running the live database and a warm standby, > with a connection pooler pointed to both so that it could > automatically reconnect on failover. pgpool already copes by design and has been working against HS for months. If the connection pooler doesn't have some way of knowing the server is a standby, then it will try to connect and fail, which presumably it would retry. This seems like a hypothetical connection pooler rather than a real one, to me. HS can continue to be used with people's existing Warm Standby setup, just now you can read the database as well. Which is what everybody has requested. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas writes: > Nobody is proposing otherwise. What Simon and I are proposing is that > if the master is configured to support HS, it comes up on the slave by > default without requiring additional configuration. Now maybe that's > too much spooky action at a distance, but I suspect it IS the behavior > most people will want. If Tom and Heikki get their way and change the > default behavior, it'll just mean (nearly) everyone flips one extra > configuration switch. We already bought into the "one extra switch" penalty when we agreed to invent a separate wal_level parameter. The entire point of that was to have more, but simpler-to-understand, parameters with fewer hidden interactions. Arguing that there are now too many knobs to twiddle amounts to trying to revisit that discussion, which we don't have time for now. >> That said, I'ld probalby be happy with PG 9 having a "default" config >> of: >> wal_mode = hot_standby >> recovery_connections = on > That would be a bad idea - there's a significant performance penalty > from setting wal_level to anything other than minimal (just as there > is for turning on archive_mode in 8.4). There is not only a performance penalty, but a reliability penalty. Enabling these switches turns on a whole lot of code that, with all due respect to those who have worked on it, is absolutely positively guaranteed to be full of bugs. Not all of which are going to be flushed out during beta. If we ship 9.0 with these things on by default, it will result in an immediate reliability downgrade for installations that are simply doing what they did before and not even interested in HS/SR. Maybe by 9.1 or 9.2 it would be sensible to have some of this code turned on by default. But it is absolutely not in keeping with this project's mindset or historical practice to enable it by default now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 11:37 -0400, Aidan Van Dyk wrote: >> It's all about the path of least *suprise*. My "least suprise" would >> have been that a similar config I have now with my PITR slaves would >> pretty much still work, and not suddenly start accepting connections, >> and even worse, at some later date when I've already verified that it >> was doing what I expected with the configuration. > > If people upgrade to 9.0 and then are surprised that the features listed > are actually available, I too would be surprised. Being available is not the same as being on by default. > There is no big use case for "run with HS turned off". Everybody wants > it on, as long as it works and don't cause problems. So far, there is no > evidence to make anyone think that it would and I wish to avoid that > implication. The use case I explained earlier exists, the one with a master and a warm standby server with pgbouncer in front. And Aidan and me are human beings, included in "everybody". You know that there is cases where it causes problems in the standby, even ignoring the possibility of bugs. For example, if you increase max_connections in the master, the standby will abort. It's safer to run with recovery_connections off if you don't need the feature. > Heikki previously argued strongly against having any switch at all, so > it could be turned on all the time. Nothing's changed. I argued that we should always WAL-log the information required to enable hot standby, on the grounds that the overhead is small. I.e. that wal_level would be a two-state switch, either 'minimal' or 'hot_standby'. But I'm happy with how that is now. But that's not what we're discussing now, don't confuse things. We're talking about recovery_connections, not wal_level. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > There is no big use case for "run with HS turned off". There was mention of running the live database and a warm standby, with a connection pooler pointed to both so that it could automatically reconnect on failover. > Everybody wants it on, as long as it works and don't cause > problems. I don't see any immediate use cases for our shop, unless it is the only way to get WAL copied to our backup server through SR. Franky, I'd much rather have a WAL receiver which assembled the WAL file segments (at the "archive" level of logging) as they arrived and then fed them to our warm standby clusters. (Of course, I've never claimed to have a "typical" environment.) I would be uncomfortable with a default of "auto" for a replica to allow connections based on WAL file contents. If someone explicitly sets it to "auto", then it's up to them to understand the implications. IMO, If someone sets it to "on", they should get a highly visible failure if it can't run in that mode. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 12:17 PM, Aidan Van Dyk wrote: > * Simon Riggs [100429 12:06]: > >> Repeating the same thing when its been refuted doesn't help. What you >> say has not been proposed. >> >> If there is a case for HS-off-by-default, make it. If you want to change >> code, arguing directly against your own position, mentioned many times, >> we need a reason. How else can we know which argument of yours to >> believe? > > I'm not against HS being on-by-default. But if it is, and the WAL > it's consuming doesn't have the HS-records by default, then I want PG to > consider that a problem, make sure I absolutely know it's a problem... Nobody is proposing otherwise. What Simon and I are proposing is that if the master is configured to support HS, it comes up on the slave by default without requiring additional configuration. Now maybe that's too much spooky action at a distance, but I suspect it IS the behavior most people will want. If Tom and Heikki get their way and change the default behavior, it'll just mean (nearly) everyone flips one extra configuration switch. > I agree with Heikki that the action-at-a-distance of HS > trying-to-work-but-maybe-not-this-time-depending-on-the-master is an > undesirable state... > > Like everything else in PG, I'ld like it to "work completely", or tell > me there is a problem. > > That said, I'ld probalby be happy with PG 9 having a "default" config > of: > > wal_mode = hot_standby > recovery_connections = on > > Make it set to generate enough WAL and actually do recovery connections. That would be a bad idea - there's a significant performance penalty from setting wal_level to anything other than minimal (just as there is for turning on archive_mode in 8.4). > But also make the recover_connections boolean really mean what it s > called. It's not called try_recovery_connections Well, sure. But setting work_mem to 1GB doesn't force the planner to use a gigabyte of memory for every sort, either. It just gives permission. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
* Simon Riggs [100429 12:14]: HS is the most requested feature across multiple polls and *not* being > able to connect to it is likely to come as a huge surprise to many > people. HS is just as secure as the main database. > > There is no big use case for "run with HS turned off". Everybody wants > it on, as long as it works and don't cause problems. So far, there is no > evidence to make anyone think that it would and I wish to avoid that > implication. > > Heikki previously argued strongly against having any switch at all, so > it could be turned on all the time. Nothing's changed. I'm not arguing against having it on by default. What I'm against (and that's strong, it should probably be "prefer not to have") is having it configured "on", but having it "not work". If it's been configured to run in a state it can't, I would prefer it didn't run, not that it ran, but in a slightly different state... But I know that's just a preference... And one from an old-school unix admin too... a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 18:40 +0300, Heikki Linnakangas wrote: > >> It's error-prone to control what happens in the standby from the master. >> That's the "action at a distance" effect I mentioned earlier. The master >> should be configured in the master, and each standby should configured >> in the standby. > > Repeating the same thing when its been refuted doesn't help. What you > say has not been proposed. I was responding to your mail where you said that there is two settings for turning hot standby off, and asking why we need two. What I'm saying is that you shouldn't think of wal_level as a setting to turn hot standby on or off. It would be error-prone to control that from the master. So there is only one setting to turn hot standby on/off, recovery_connections in the standby. > If there is a case for HS-off-by-default, make it. If you want to change > code, arguing directly against your own position, mentioned many times, > we need a reason. How else can we know which argument of yours to > believe? Now you lost me. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
* Simon Riggs [100429 12:06]: > Repeating the same thing when its been refuted doesn't help. What you > say has not been proposed. > > If there is a case for HS-off-by-default, make it. If you want to change > code, arguing directly against your own position, mentioned many times, > we need a reason. How else can we know which argument of yours to > believe? I'm not against HS being on-by-default.But if it is, and the WAL it's consuming doesn't have the HS-records by default, then I want PG to consider that a problem, make sure I absolutely know it's a problem... I agree with Heikki that the action-at-a-distance of HS trying-to-work-but-maybe-not-this-time-depending-on-the-master is an undesirable state... Like everything else in PG, I'ld like it to "work completely", or tell me there is a problem. That said, I'ld probalby be happy with PG 9 having a "default" config of: wal_mode = hot_standby recovery_connections = on Make it set to generate enough WAL and actually do recovery connections. But also make the recover_connections boolean really mean what it s called. It's not called try_recovery_connections a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 11:37 -0400, Aidan Van Dyk wrote: > * Simon Riggs [100429 11:24]: > > > > What is the reason for the *extra* "off" switch? Why two? Why "off" > > twice? > > Because I run a PITR slave off a WAL archive... And I'm going to > continue to try and use this setup... I expect to try and experiment > with wal-streaming too when everything is up to 9, and be a little upset > when the "old faithfull" backup I've got and am betting my job on > suddenly starts acting differentlyl, or accepting connections and > causing my db clients to start thowing errors because I'm trying to get > a streaming replication w/ hot standby up on a *different* slave... > > It's all about the path of least *suprise*. My "least suprise" would > have been that a similar config I have now with my PITR slaves would > pretty much still work, and not suddenly start accepting connections, > and even worse, at some later date when I've already verified that it > was doing what I expected with the configuration. If people upgrade to 9.0 and then are surprised that the features listed are actually available, I too would be surprised. If we all believe that these radical surprises are a problem, then we should also turn off join removal and loads of other features in 9.0 that will also cause surprises. HS is the most requested feature across multiple polls and *not* being able to connect to it is likely to come as a huge surprise to many people. HS is just as secure as the main database. There is no big use case for "run with HS turned off". Everybody wants it on, as long as it works and don't cause problems. So far, there is no evidence to make anyone think that it would and I wish to avoid that implication. Heikki previously argued strongly against having any switch at all, so it could be turned on all the time. Nothing's changed. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 18:40 +0300, Heikki Linnakangas wrote: > It's error-prone to control what happens in the standby from the master. > That's the "action at a distance" effect I mentioned earlier. The master > should be configured in the master, and each standby should configured > in the standby. Repeating the same thing when its been refuted doesn't help. What you say has not been proposed. If there is a case for HS-off-by-default, make it. If you want to change code, arguing directly against your own position, mentioned many times, we need a reason. How else can we know which argument of yours to believe? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 09:48 -0400, Tom Lane wrote: >> Simon Riggs writes: >>> recovery_connections was on by default and unanimous agreement until >>> recently and I don't want to change that now, >> Uh, it was on by default only because a lot of us hadn't noticed that. >> I agree with Heikki's position here: it should not be on by default. > > We're talking about the case where somebody has set up a standby > database. It's not like they happen on this accidentally. > > * HS on by default, in the standby, via recovery_connections. > * HS off by default, in the master, via wal_level. > > Overall, that *is* off by default. (Note: I said nothing about that). > > We don't need it off *twice*, nor do we even need two switches. wal_level is not supposed to control if Hot Standby is on or off. It controls what information is written to WAL. If you have wal_level='archive' and recovery_connections='on' in the standby, that's a configuration error, just like setting wal_level='minimal' and archive_mode='on'. You wanted to enable Hot Standby, but the information required isn't in the WAL. It's error-prone to control what happens in the standby from the master. That's the "action at a distance" effect I mentioned earlier. The master should be configured in the master, and each standby should configured in the standby. The right mental model is that wal_mode controls what is written to the WAL. In fact, I might recommend always setting it to 'hot_standby' instead of 'archive', even if you have no standbys and you're only doing WAL archival - it's a lot easier to do PITR with hot standby. To control whether hot standby is enabled in the standby, use recovery_connections. And I'd prefer it to be off by default because off is the safer option. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
* Simon Riggs [100429 11:24]: > > What is the reason for the *extra* "off" switch? Why two? Why "off" > twice? Because I run a PITR slave off a WAL archive... And I'm going to continue to try and use this setup... I expect to try and experiment with wal-streaming too when everything is up to 9, and be a little upset when the "old faithfull" backup I've got and am betting my job on suddenly starts acting differentlyl, or accepting connections and causing my db clients to start thowing errors because I'm trying to get a streaming replication w/ hot standby up on a *different* slave... It's all about the path of least *suprise*. My "least suprise" would have been that a similar config I have now with my PITR slaves would pretty much still work, and not suddenly start accepting connections, and even worse, at some later date when I've already verified that it was doing what I expected with the configuration. There has got to be *something* that tells me it is trying to allow connectins during recovery, but failing... And i think that just loging a WARNING and continuing isn't enough, because I'll admit to having upgraded PG and just "testing" it, not looking thoroughly through the logs for any new messages that might peak my interest... That said, since I follow -hackers, that "something" could be considered this email thread, and I know I will be extra careful about my deployment of PITR slaves w/ 9 wrt making sure I'm explicit in all the settings I can find... And I'll make sure I look more carefully at logs when deploying 9 as well ;-) a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 09:48 -0400, Tom Lane wrote: > Simon Riggs writes: > > recovery_connections was on by default and unanimous agreement until > > recently and I don't want to change that now, > > Uh, it was on by default only because a lot of us hadn't noticed that. > I agree with Heikki's position here: it should not be on by default. We're talking about the case where somebody has set up a standby database. It's not like they happen on this accidentally. * HS on by default, in the standby, via recovery_connections. * HS off by default, in the master, via wal_level. Overall, that *is* off by default. (Note: I said nothing about that). We don't need it off *twice*, nor do we even need two switches. Last week we had one switch and it was on by default, now we're looking at two switches and off by default. I haven't yet heard a good reason for the change being proposed here by Heikki. The use cases are rare, if they truly exist at all. Monitoring the standby is much easier with HS on, for example. What is the reason for the *extra* "off" switch? Why two? Why "off" twice? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs writes: > recovery_connections was on by default and unanimous agreement until > recently and I don't want to change that now, Uh, it was on by default only because a lot of us hadn't noticed that. I agree with Heikki's position here: it should not be on by default. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas writes: > On Thu, Apr 29, 2010 at 7:19 AM, Heikki Linnakangas > wrote: >> If you're setting up a reporting server, and hot standby can't start, >> the server is not functioning properly. I would like to get an error in >> that case. > > Presumably you will actually try connecting to it, no? [...] > I think this is a problem that should be solved by monitoring, rather > than by automatically taking the server down. +1 on both counts. Is that a +2? Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 9:07 AM, Heikki Linnakangas wrote: > Robert Haas wrote: >> How about this one? The administrator sets up a master and a slave. >> She's heard about this new Hot Standby feature and so decides to >> enable it on the slave just to play around with it. Subsequently, she >> takes a better job at another company and they hire a new >> administrator, who thinks the Hot Standby WAL may be causing a >> performance problem on the master, so he switches wal_mode to archive. >> Six months later the primary fails. > > Umm, I don't see the problem. For high availability purposes, the > standby works just as well with wal_mode='archive'. Or are you saying > that the standby was configured with recovery_connections='on', and > failed to start for those six months because hot standby could not be > enabled, and no-one noticed? Yep. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 9:00 AM, Heikki Linnakangas wrote: > Robert Haas wrote: >> On Thu, Apr 29, 2010 at 7:19 AM, Heikki Linnakangas >> wrote: >>> Robert Haas wrote: I kind of agree with Simon on this one, except I would probably choose to have just on and off and make on work like his auto. In other words, recovery_connections=on means, give me recovery connections if possible, otherwise don't worry about it. >>> If you're setting up a reporting server, and hot standby can't start, >>> the server is not functioning properly. I would like to get an error in >>> that case. >> >> Presumably you will actually try connecting to it, no? > > Sure. I guess it would be acceptable if 'on' meant 'on, if possible', as > long as 'off' is the default. Otherwise it's too surprising. I disagree. I think on should mean 'on, if possible' and 'on' should be the default. I think that's how it was before this round of changes - but maybe I'm mistaken? It seems like it makes sense, at any rate. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > How about this one? The administrator sets up a master and a slave. > She's heard about this new Hot Standby feature and so decides to > enable it on the slave just to play around with it. Subsequently, she > takes a better job at another company and they hire a new > administrator, who thinks the Hot Standby WAL may be causing a > performance problem on the master, so he switches wal_mode to archive. > Six months later the primary fails. Umm, I don't see the problem. For high availability purposes, the standby works just as well with wal_mode='archive'. Or are you saying that the standby was configured with recovery_connections='on', and failed to start for those six months because hot standby could not be enabled, and no-one noticed? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > On Thu, Apr 29, 2010 at 7:19 AM, Heikki Linnakangas > wrote: >> Robert Haas wrote: >>> I kind of agree with Simon on this one, except I would probably choose >>> to have just on and off and make on work like his auto. >>> >>> In other words, recovery_connections=on means, give me recovery >>> connections if possible, otherwise don't worry about it. >> If you're setting up a reporting server, and hot standby can't start, >> the server is not functioning properly. I would like to get an error in >> that case. > > Presumably you will actually try connecting to it, no? Sure. I guess it would be acceptable if 'on' meant 'on, if possible', as long as 'off' is the default. Otherwise it's too surprising. > And what happens when someone changes the setting on the master from > hot_standby back to archive? I'd rather have the reporting server > continue recovery without being able to accept connections rather than > die in its tracks. As the code stands, if wal_level is switched from 'hot_standby' to 'archive' in the primary, and it's restarted, the standby will die. There is currently no way to stop to accepting read-only connections once they're allowed. It could restart with connections disallowed, but it needs a restart. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 7:55 AM, Heikki Linnakangas wrote: > What happened, I don't find that story very compelling because there are an infinite number of ways to have high-availability not work if you start by supposing that the person who sets them up doesn't test them properly and verify that everything actually works as expected. You could do all sorts of things wrong in that case. How about this one? The administrator sets up a master and a slave. She's heard about this new Hot Standby feature and so decides to enable it on the slave just to play around with it. Subsequently, she takes a better job at another company and they hire a new administrator, who thinks the Hot Standby WAL may be causing a performance problem on the master, so he switches wal_mode to archive. Six months later the primary fails. I think you can construct a scenario where just about any default setting causes a problem, but I like the idea of having this enabled by default, and I think it works fine if you just treat the case where recovery_connections=on but wal_mode=archive as a LOG or WARNING rather than an ERROR. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > recovery_connections was on by default and unanimous agreement until > recently and I don't want to change that now, just because a change > somewhere else appears to be forcing that but need not be so. We never really had that discussion, until now. It has always been 'on' by default, and that has been useful to get more testing during the development cycle, but it's not clear that's a good default for real-life usage. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 14:55 +0300, Heikki Linnakangas wrote: > I'd like to scaremonger Seems so. recovery_connections was on by default and unanimous agreement until recently and I don't want to change that now, just because a change somewhere else appears to be forcing that but need not be so. It was sensible to add a switch to turn HS off, but it should not be the default, especially not one that requires a restart to enable a high availability feature. That is important in a feature that takes a while to "kick-in" and so the user may patiently wait for it to come up and it never does. I have no wish to repeat the situation that PostgreSQL requires a restart to enable a feature, while other forks retain the ability to enable the parameter without restart, as occurs with archive_mode, IIRC. Perhaps we should not re-think wal_level if we've just moved the parameter problem somewhere else? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 13:46 +0300, Heikki Linnakangas wrote: >> warm standby for high availability - no >> offloading taking filesystem-level backups from master - no > > These can be explicitly turned off. > You're presuming there is benefit in turning recovery_connections = off, > though it is perfectly valid to do those use cases with it on. There are > many ways to control connections, not just that switch. It will > certainly be easier to monitor the HA system by running queries against > it than not. Do you have any evidence there is benefit in the *typical* > case for turning the setting off? It depends on your exact configuration, but one typical one is that you have a work-balancing router or pgbouncer sitting in front of the servers, directing traffic to the server that's up and running. If the standby starts accepting connections prematurely, the clients will be incorrectly routed to the standby server and update operations will fail (and SELECTs will return slightly delayed data). >> All of those either want hot standby, or don't. What use case is there >> for "enabled, if the required information is in the WAL"? If there is >> one, it sure doesn't seem like the most common one. > > I think "I want it to just work" is fairly common. You need quite a bit of set up anyway, flipping one more GUC hardly makes a difference. There is less risk of oversight and accidental misconfiguration if the admin makes a conscious decision to turn it on. I'd like to scaremonger with the following fictional story: An administrator sets up two PostgreSQL servers in a high availability warm standby set up. Clients are set up to try to connect to both servers, so that when failover happens, they will automatically reconnect to the remaining server. wal_level is set to 'archive' in the master, and all is well. After running successfully for six months in production, a reporting server is introduced to offload heavy queries from the mission-critical OLTP server. wal_level is set to 'hot_standby' in the master to allow read-only queries to be run against the reporting server, and the reporting server is set up using the same WAL archive used for the warm standby server. All seems to be running well, the admin logs in to the application and clicks through a few screens to test it. A few hours later a user rings and complains that he's getting a "cannot execute INSERT in a read-only transaction" error. What happened, and why does it work just fine when the admin tries the same? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 7:19 AM, Heikki Linnakangas wrote: > Robert Haas wrote: >> I kind of agree with Simon on this one, except I would probably choose >> to have just on and off and make on work like his auto. >> >> In other words, recovery_connections=on means, give me recovery >> connections if possible, otherwise don't worry about it. > > If you're setting up a reporting server, and hot standby can't start, > the server is not functioning properly. I would like to get an error in > that case. Presumably you will actually try connecting to it, no? And what happens when someone changes the setting on the master from hot_standby back to archive? I'd rather have the reporting server continue recovery without being able to accept connections rather than die in its tracks. I think this is a problem that should be solved by monitoring, rather than by automatically taking the server down. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Robert Haas wrote: > I kind of agree with Simon on this one, except I would probably choose > to have just on and off and make on work like his auto. > > In other words, recovery_connections=on means, give me recovery > connections if possible, otherwise don't worry about it. If you're setting up a reporting server, and hot standby can't start, the server is not functioning properly. I would like to get an error in that case. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 13:46 +0300, Heikki Linnakangas wrote: > purpose of the standby - do you want hot standby or not? > reporting - yes > offloading queries from master - yes > offloading pg_dump from master - yes These would work using the proposed default. > warm standby for high availability - no > offloading taking filesystem-level backups from master - no These can be explicitly turned off. You're presuming there is benefit in turning recovery_connections = off, though it is perfectly valid to do those use cases with it on. There are many ways to control connections, not just that switch. It will certainly be easier to monitor the HA system by running queries against it than not. Do you have any evidence there is benefit in the *typical* case for turning the setting off? > All of those either want hot standby, or don't. What use case is there > for "enabled, if the required information is in the WAL"? If there is > one, it sure doesn't seem like the most common one. I think "I want it to just work" is fairly common. > When you just want > hot standby to be on or off, it's weird to control that from the master. > Action at a distance. That's the proposed default, not the only control. The standby can override what the master says if it definitely doesn't want it, but is smart enough to avoid silly configuration errors. Turning off wal_level on the master *does* affect the standby, so pretending we have a completely separate configuration option makes no sense for me. Robert's point when he raised the wal_level discussion was about reducing the number of parameter settings required to make this work. An intelligent default will reduce level of configuration and allow us to make Hot Standby work on the standby, out of the box, and that is worth having. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 5:55 AM, Heikki Linnakangas wrote: >> The most sensible way is to make the default act intelligently depending >> upon what it finds in the control file. If WAL contains hot_standby >> info, then recovery_connections should be on by default, though can be >> turned off explicitly if required. If WAL does not contain hot_standby >> info we then we throw a WARNING and continue as if recovery_connections >> = off, or if recovery_connections is specified explicitly then we should >> throw an ERROR so that the user knows it won't be possible to use this. >> I think that means we'd need to change recovery_connections to have 2 or >> 3 values, but non-boolean: >> recovery_connections = auto (default) | off >> or >> recovery_connections = auto (default) | on | off >> > > Seems overly complicated. It would be simpler to just set it 'off' by > default. I kind of agree with Simon on this one, except I would probably choose to have just on and off and make on work like his auto. In other words, recovery_connections=on means, give me recovery connections if possible, otherwise don't worry about it. I'd rather not have to change the default to recovery_connections=off - that's one more parameter someone has to set properly. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 5:38 AM, Heikki Linnakangas wrote: > NOTICE: WAL archiving is not enabled, you must ensure that all required > WAL segments are streamed or copied through other means to restore the > backup I might think about dropping the words "through other means" from this sentence. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > The whole point of > wal_level discussion was to avoid needing multiple switches to turn > something on. No, the point of wal_level was to make the behavior easier to understand, by uncoupling the level of WAL-logging from various other switches, controling it directly and explicitly with a new GUC instead. It added a new switch, but made the system as a whole easier to understand and configure. >> > I can't imagine a situation where "allow connections if the WAL allows >> > it" would be a sensible setting. If there is one, we could have an >> > 'auto' mode, but not as the default. > > I can. Simple, obvious behaviour. Turn it on in the master, works on the > standbys. Yes, but when would you want that? Here's the use cases I can think of: purpose of the standby - do you want hot standby or not? reporting - yes offloading queries from master - yes warm standby for high availability - no offloading taking filesystem-level backups from master - no offloading pg_dump from master - yes All of those either want hot standby, or don't. What use case is there for "enabled, if the required information is in the WAL"? If there is one, it sure doesn't seem like the most common one. When you just want hot standby to be on or off, it's weird to control that from the master. Action at a distance. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 12:55 +0300, Heikki Linnakangas wrote: > Seems overly complicated. If you turn wal_level = hot_standby in the master, then the slaves work, unless you say otherwise. What is complicated about that? > It would be simpler to just set it 'off' by > default. As a parameter, yes, it is simpler. The behaviour is not simplified by doing that. I care about the behaviour. > If it's 'off' by default, and you want to use hot standby, you will > notice quickly that the server doesn't accept connections, and you > switch it on. If it's 'on' (or 'auto'), you might not realize that your > standby server is accepting connections, when you only wanted to set it > up as a warm standby server for high availability. That requires we hit the problem, alter the parameter and restart. We don't need to do that at all. It could "Just Work". > The 'auto' mode just makes it more surprising. Connections might be > allowed at first, but when someone switches wal_level in the primary for > some reason, your reporting standby is up, but no longer allows > connections. If you don't do this, what would happen on standby? Does it silently stop applying changes after the point you turned it off, or does it throw an ERROR. > And vice versa, if you set up a warm standby server for > high availability where you don't want to allow connections, and someone > flips the wal_level switch in the master, the standby suddenly starts > accepting connections. Some people would regard that as a good thing. The whole point of wal_level discussion was to avoid needing multiple switches to turn something on. You are proposing exactly that here. Why is this discussion different? Why should we not be able to set in just one place. If that behaviour concerns the DBA then they can turn it off explicitly. > I can't imagine a situation where "allow connections if the WAL allows > it" would be a sensible setting. If there is one, we could have an > 'auto' mode, but not as the default. I can. Simple, obvious behaviour. Turn it on in the master, works on the standbys. We want people to use the feature, not fumble with it. Just think "auto" means "linked to master". If you don't like it you can turn it off. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs wrote: > On Thu, 2010-04-29 at 10:55 +0300, Heikki Linnakangas wrote: > >> Should we change the default to recovery_connections=off ? It is a quite >> big change in default behavior over previous releases that hot standby >> is enabled by default, so maybe that would be the right thing to do anyway. >> >> Or maybe add a hint to the above, "disable hot standby by setting >> recovery_connections=off, or set wal_level='hot_standby' in the primary" > > I've been waiting for this suggestion, a clear knock-on effect from your > earlier changes. If we just have a static default its bad either way. > > The most sensible way is to make the default act intelligently depending > upon what it finds in the control file. If WAL contains hot_standby > info, then recovery_connections should be on by default, though can be > turned off explicitly if required. If WAL does not contain hot_standby > info we then we throw a WARNING and continue as if recovery_connections > = off, or if recovery_connections is specified explicitly then we should > throw an ERROR so that the user knows it won't be possible to use this. > I think that means we'd need to change recovery_connections to have 2 or > 3 values, but non-boolean: > recovery_connections = auto (default) | off > or > recovery_connections = auto (default) | on | off > Seems overly complicated. It would be simpler to just set it 'off' by default. If it's 'off' by default, and you want to use hot standby, you will notice quickly that the server doesn't accept connections, and you switch it on. If it's 'on' (or 'auto'), you might not realize that your standby server is accepting connections, when you only wanted to set it up as a warm standby server for high availability. The 'auto' mode just makes it more surprising. Connections might be allowed at first, but when someone switches wal_level in the primary for some reason, your reporting standby is up, but no longer allows connections. And vice versa, if you set up a warm standby server for high availability where you don't want to allow connections, and someone flips the wal_level switch in the master, the standby suddenly starts accepting connections. I can't imagine a situation where "allow connections if the WAL allows it" would be a sensible setting. If there is one, we could have an 'auto' mode, but not as the default. > and I would suggest removing it from postgresql.conf.sample -1. Why try to hide it? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Tom Lane wrote:
> Yeah. ISTM the real bottom line here is that we have only a weak grasp
> on how these features will end up being used; or for that matter what
> the common error scenarios will be. I think that for the time being
> we should err on the side of being permissive. We can tighten things
> up and add more nanny-ism in the warnings later on, when we have
> more field experience.
Ok, here's a proposed patch. Per discussion, it relaxes the checks in
pg_start/stop_backup() so that they can be used as long as wal_level >=
'archive', even if archiving is disabled.
If archiving is not enabled, it can't wait for the files to be archived.
Instead, it prints a notice:
NOTICE: WAL archiving is not enabled, you must ensure that all required
WAL segments are streamed or copied through other means to restore the
backup
That is instead of the usual notice when archiving is enabled:
NOTICE: pg_stop_backup complete, all required WAL segments have been
archived
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
*** a/src/backend/access/transam/xlog.c
--- b/src/backend/access/transam/xlog.c
***
*** 8200,8217 pg_start_backup(PG_FUNCTION_ARGS)
errmsg("recovery is in progress"),
errhint("WAL control functions cannot be executed during recovery.")));
! if (!XLogArchivingActive())
! ereport(ERROR,
! (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
! errmsg("WAL archiving is not active"),
! errhint("archive_mode must be enabled at server start.")));
!
! if (!XLogArchiveCommandSet())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
! errmsg("WAL archiving is not active"),
! errhint("archive_command must be defined before "
! "online backups can be made safely.")));
backupidstr = text_to_cstring(backupid);
--- 8200,8210
errmsg("recovery is in progress"),
errhint("WAL control functions cannot be executed during recovery.")));
! if (!XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
! errmsg("WAL level not sufficient for making an online backup"),
! errhint("wal_level must be set to 'archive' or 'hot_standby' at server start.")));
backupidstr = text_to_cstring(backupid);
***
*** 8399,8409 pg_stop_backup(PG_FUNCTION_ARGS)
errmsg("recovery is in progress"),
errhint("WAL control functions cannot be executed during recovery.")));
! if (!XLogArchivingActive())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
! errmsg("WAL archiving is not active"),
! errhint("archive_mode must be enabled at server start.")));
/*
* OK to clear forcePageWrites
--- 8392,8402
errmsg("recovery is in progress"),
errhint("WAL control functions cannot be executed during recovery.")));
! if (!XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
! errmsg("WAL level not sufficient for making an online backup"),
! errhint("wal_level must be set to 'archive' or 'hot_standby' at server start.")));
/*
* OK to clear forcePageWrites
***
*** 8511,8526 pg_stop_backup(PG_FUNCTION_ARGS)
CleanupBackupHistory();
/*
! * Wait until both the last WAL file filled during backup and the history
! * file have been archived. We assume that the alphabetic sorting
! * property of the WAL files ensures any earlier WAL files are safely
! * archived as well.
*
* We wait forever, since archive_command is supposed to work and we
* assume the admin wanted his backup to work completely. If you don't
* wish to wait, you can set statement_timeout. Also, some notices are
* issued to clue in anyone who might be doing this interactively.
*/
XLByteToPrevSeg(stoppoint, _logId, _logSeg);
XLogFileName(lastxlogfilename, ThisTimeLineID, _logId, _logSeg);
--- 8504,8530
CleanupBackupHistory();
/*
! * If archiving is enabled, wait for all the required WAL files to be
! * archived before returning. If archiving isn't enabled, the required
! * WAL needs to be transported via streaming replication (hopefully
! * with wal_keep_segments set high enough), or some more exotic
! * mechanism like polling and copying files from pg_xlog with script.
! * We have no control over those mechanisms, so it's up to the user to
! * ensure that he gets all the required WAL.
! *
! * We wait until both the last WAL file filled during backup and the
! * history file have been archived, and assume that the alphabetic
! * sorting property of the WAL files ensures any earlier WAL files are
! * safely archived as well.
*
* We wait forever, since archive_command is supposed to work and we
* assume the admin wanted his backup to work completely. If you don't
* wish to wait, you can set statement_timeout. Also, some notic
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, 2010-04-29 at 10:55 +0300, Heikki Linnakangas wrote: > Should we change the default to recovery_connections=off ? It is a quite > big change in default behavior over previous releases that hot standby > is enabled by default, so maybe that would be the right thing to do anyway. > > Or maybe add a hint to the above, "disable hot standby by setting > recovery_connections=off, or set wal_level='hot_standby' in the primary" I've been waiting for this suggestion, a clear knock-on effect from your earlier changes. If we just have a static default its bad either way. The most sensible way is to make the default act intelligently depending upon what it finds in the control file. If WAL contains hot_standby info, then recovery_connections should be on by default, though can be turned off explicitly if required. If WAL does not contain hot_standby info we then we throw a WARNING and continue as if recovery_connections = off, or if recovery_connections is specified explicitly then we should throw an ERROR so that the user knows it won't be possible to use this. I think that means we'd need to change recovery_connections to have 2 or 3 values, but non-boolean: recovery_connections = auto (default) | off or recovery_connections = auto (default) | on | off and I would suggest removing it from postgresql.conf.sample -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Fujii Masao wrote:
> On Thu, Apr 29, 2010 at 1:14 AM, Heikki Linnakangas
> wrote:
>> Ok, I've finally committed the patch, using wal_level as the name of the
>> GUC.
>
>> !if (InArchiveRecovery && XLogRequestRecoveryConnections)
>> !{
>> !if (ControlFile->wal_level < WAL_LEVEL_HOT_STANDBY)
>> !ereport(ERROR,
>> !(errmsg("recovery connections cannot
>> start because wal_level was not set to 'hot_standby' on the WAL source
>> server")));
>
> I still have the complaint against the above check. Since the default value
> of recovery_connections is TRUE, the users who need only archiving not
> replication (i.e., wal_level is set to 'archive') are likely to often
> see the failure
> of the archive recovery by the above check.
Should we change the default to recovery_connections=off ? It is a quite
big change in default behavior over previous releases that hot standby
is enabled by default, so maybe that would be the right thing to do anyway.
Or maybe add a hint to the above, "disable hot standby by setting
recovery_connections=off, or set wal_level='hot_standby' in the primary"
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Wed, 2010-04-28 at 22:17 +0200, Dimitri Fontaine wrote: > IMO the real fun begins when we talk about multi-slaves support and > their roles (a failover slave wants the master to wait for it to have > applied the WAL before to commit, a reporting slave not so much). So > you'd set the Availability level on each slave and wouldn't commit on > the master until each slave got what it's configured for, or something > like that. Just for the record, I outlined desirable semantics for this on hackers in 2008 and want to keep those ideas on the table. http://archives.postgresql.org/pgsql-hackers/2008-07/msg01001.php My view is that it should be up to the master what happens on master. An additional standby connection should not have the ability to make transactions on the master wait. If we give control to the master rather than the standby, we are then able to allow transactions on the master choose how robust they should be, just as we do with synchronous_commit. IMHO that is extremely important, since we already know that sync rep performs poorly and applications need to mitigate that in some way. Those are the objectives, the parameters to do that are a different story and we might expect much debate. One way of doing this would be to have a parameter called synchronous_replication = N, which would cause the transaction on primary to wait for at least N standbys to reply that they have the data. This would allow settings like synchronous_commit = 0--async synchronous_commit = 1--first reply wins == max performance synchronous_commit = 2--multiple replies needed == max availability ... -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Thu, Apr 29, 2010 at 1:14 AM, Heikki Linnakangas
wrote:
> Ok, I've finally committed the patch, using wal_level as the name of the
> GUC.
> ! if (InArchiveRecovery && XLogRequestRecoveryConnections)
> ! {
> ! if (ControlFile->wal_level < WAL_LEVEL_HOT_STANDBY)
> ! ereport(ERROR,
> ! (errmsg("recovery connections cannot
> start because wal_level was not set to 'hot_standby' on the WAL source
> server")));
I still have the complaint against the above check. Since the default value
of recovery_connections is TRUE, the users who need only archiving not
replication (i.e., wal_level is set to 'archive') are likely to often
see the failure
of the archive recovery by the above check.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Dimitri Fontaine wrote: IMO the real fun begins when we talk about multi-slaves support and their roles (a failover slave wants the master to wait for it to have applied the WAL before to commit, a reporting slave not so much). So you'd set the Availability level on each slave and wouldn't commit on the master until each slave got what it's configured for, or something like that. Ultimately the commit is stuck waiting for the slowest committing sync operation on the list; it's the bottleneck. Let's presume that the commit waits can be done in parallel, after sending the transaction to every slave. Given that and the situation you describe, having per-node sync levels only turns out to be a useful optimization if the reporting slave commits slower than the failover slave does. The master is going to be stuck waiting for the slowest one of the batch regardless of whether you've optimized them individually. There is a related situation that I think a per-node sync option would be more obviously useful for: local failover slave, remote disaster recovery slave over a WAN, where you accept that a serious disaster taking out a whole data center will lose some transactions. In that situation, you'd probably want fsync for the local slave, while going async for the remote datacenter. If the commits are done in a serial fashion, tuning sync per-node would be much more valuable in many use cases. Regardless, I wouldn't want to burden the first sync rep version with this requirement. Let's wait until the current scope is cleared before trying to move the goalposts for the people working on that. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Aidan Van Dyk wrote: I remember a presentation at pgcon a while ago, it was probaly Fujii (from NTT?) about their log streaming, and at that time, they talked about different "sync" options... It's all outlined at http://wiki.postgresql.org/wiki/Streaming_Replication#Synchronization_capability -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
* Kevin Grittner [100428 15:51]: > > I don't personally care about streaming replication replaying WAL > > as it comes, or running queries in recovery... > > I'm with you that far, but I wouldn't want the sender to wait for > remote persistence. I remember a presentation at pgcon a while ago, it was probaly Fujii (from NTT?) about their log streaming, and at that time, they talked about different "sync" options... So I'ld love to be able to have comits be: async (like current option) local wal sync (like current) local wal sync + walsender sent local wal sync + walsender confirmed And ideally, the "walsender sent/confirmed" would even allow making sure it was sent/confirmed to $X connections... I want to be able to guarantee it's on 2 machines, not that if my slave was connected it would be on there, but something happened and my "slave" has disconnected, so it's only got local WAL... And then on whatever "tool" is receiving the log streaming, it can be set to confirm when either: received buffer write buffer to file write buffer to file + sync write buffer to file + sync + replay That should give you all the sync levels they talked about in their presentation... -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
"Kevin Grittner" writes: > Aidan Van Dyk wrote: > >> I'm hoping to be able to build a tool that: >> >> 1) Connects to PG walsender (a la walreceiver) >> 2) Streams WAL from pg master >> 3) Saves WAL into "files" (a la archive)... >> >> i.e. I'm looking to keep a more-up-to-date PITR archive than >> waiting for traditional WAL file archiving... > > I'm interested in that, too. That looks like we have that integrated into walreceiver the day we have cascading support, right? Or maybe we need a special mode of operation where the receiver is (talking to) an archiver. >> I don't personally care about streaming replication replaying WAL >> as it comes, or running queries in recovery... > > I'm with you that far, but I wouldn't want the sender to wait for > remote persistence. That's synchronous replication and its set of synchronicity setting, ranging from sent on the network to the slave, fsync()ed at the slave and applied already on the slave. IMO the real fun begins when we talk about multi-slaves support and their roles (a failover slave wants the master to wait for it to have applied the WAL before to commit, a reporting slave not so much). So you'd set the Availability level on each slave and wouldn't commit on the master until each slave got what it's configured for, or something like that. SyncRep in 9.1 already sounds darn interesting :) Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Aidan Van Dyk wrote: > I'm hoping to be able to build a tool that: > > 1) Connects to PG walsender (a la walreceiver) > 2) Streams WAL from pg master > 3) Saves WAL into "files" (a la archive)... > > i.e. I'm looking to keep a more-up-to-date PITR archive than > waiting for traditional WAL file archiving... I'm interested in that, too. > I don't personally care about streaming replication replaying WAL > as it comes, or running queries in recovery... I'm with you that far, but I wouldn't want the sender to wait for remote persistence. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Heikki Linnakangas writes: > Hmm, you could start streaming the WAL before you start the backup, so > the fact that you've already removed some segments that are needed to > restore from the backup by the time pg_stop_backup() is called doesn't > necessarily mean that the backup is useless. > You'd need a stand-alone tool to do the streaming in that case, and no > such tool exists yet, but I would be surprised if one doesn't appear on > pgfoundry sooner or later :-). Yeah. ISTM the real bottom line here is that we have only a weak grasp on how these features will end up being used; or for that matter what the common error scenarios will be. I think that for the time being we should err on the side of being permissive. We can tighten things up and add more nanny-ism in the warnings later on, when we have more field experience. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
