Re: [HACKERS] Open issues for collations
On Mon, Mar 28, 2011 at 08:02:40PM -0400, Tom Lane wrote: One thing I noticed but didn't push to committing is that the test case has a largely-unnecessary assumption about how the local system's locale names spell utf8. We could eliminate that by having it use the trimmed locale names created by initdb. I would've made more of a push for that if it resulted in a test case that passed on OS X, but it turns out that once you get past the locale name spelling, you find out that Macs still can't sort UTF8 strings correctly :-( Yeah, and I don't think it's likely they're ever going to fix it either. :( On their own website they explain how to do locale based sorting, and they provide their own interfaces for that which, unsurprisingly, uses UTF-16 and ICU underneath. http://developer.apple.com/library/mac/#documentation/CoreFoundation/Conceptual/CFLocales/Articles/CFLocaleConcepts.html I think the rule that we sort the same as the command-line sort program is still true however, so I don't think it's surprising as such. Just unfortunate. Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Patriotism is when love of your own people comes first; nationalism, when hate for people other than your own comes first. - Charles de Gaulle signature.asc Description: Digital signature
Re: [HACKERS] SSI bug?
hi, [no residual SIReadLock] i read it as there are many (7057) SIReadLocks somehow leaked. am i wrong? YAMAMOTO Takashi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
Yeb Havinga yebhavi...@gmail.com writes: The dba interface for recv|fsync|apply seems to be pretty stable, so supporting that for years should be without risk. How it works under the hood - the beta period seems like *the* opportunity to attrach mayor testing from all people waiting to get their hands on syncrep. +1 It would be better to just support it (recv|fsync|apply), or no syncrep at all. Syncrep is incomplete without it. Agreed. More than that, I think we should evaluate this patch on a cost/benefit ratio, rather than trying to apply to it all those procedural fences that we don't have, and that we don't have the size to benefit from. This whole thread only managed to raise the cost of the feature, but compared to its benefits, it's still a wash. Is there any good reason that I missed to ask all our users to wait for at best another year to get the SyncRep waiting behavior that makes sense and has been agreed on for a very long time already? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support PS : we already tweaked the UI in such a way that controling this feature from the standby makes no sense. What we talked about was to setup on the master which durability level you need, and on each standby which one you're able to provide. Then you mixmatch. That won't fly with current way to setup the sync standby list. So current recv|fsync|apply patch is IMO finishing the 9.1 work. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On Mon, Mar 28, 2011 at 05:29:22PM -0500, Kevin Grittner wrote: I have a theory. Can you try it in what would be the failure case, but run an explicit a CHECKPOINT on the master, wait for pg_controldata to show that checkpoint on the slave, and (as soon as you see that) try to trigger the slave to come up in production? =$ ( pg_controldata master/; pg_controldata slave2/ ) | grep Latest checkpoint location: Latest checkpoint location: 0/2D58 Latest checkpoint location: 0/2C58 =$ psql -p 54001 -c checkpoint CHECKPOINT =$ ( pg_controldata master/; pg_controldata slave2/ ) | grep Latest checkpoint location: Latest checkpoint location: 0/2E58 Latest checkpoint location: 0/2C58 ... ~ 1.5 minute later =$ ( pg_controldata master/; pg_controldata slave2/ ) | grep Latest checkpoint location: Latest checkpoint location: 0/2E58 Latest checkpoint location: 0/2E58 =$ touch /home/depesz/slave2/finish.recovery it worked. now the slave2 is working as stand alone. what does it tell us? will any work happening after checkpoint break it anyway? Best regards, depesz -- The best thing about modern society is how easy it is to avoid contact with it. http://depesz.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On Tue, Mar 29, 2011 at 11:20:48AM +0900, Fujii Masao wrote: On Tue, Mar 29, 2011 at 12:11 AM, hubert depesz lubaczewski dep...@depesz.com wrote: On Mon, Mar 28, 2011 at 01:48:13PM +0900, Fujii Masao wrote: In 9.0, recovery doesn't read a backup history file. That FATAL error happens if recovery ends before it reads the WAL record which was generated by pg_stop_backup(). IOW, recovery gets the recovery ending location from WAL record not backup history file. Since you didn't run pg_stop_backup() and there is no WAL record containing the recovery ending location, you got that error. If you want to take hot backup from the standby, you need to do the procedure explained in http://wiki.postgresql.org/wiki/Incrementally_Updated_Backups one more question. how come that I can use this backup to make standalone pg, and it starts without any problem, but when I start it as sr slave, let it run for some time, and then promote to standalone, it breaks? Did you use recovery.conf to start standalone PostgreSQL? If not, recovery doesn't check whether it reaches the recovery ending position or not. So I guess no problem didn't happen. no, i don't use. hmm .. i am nearly 100% certain that previous pgs did in fact check if the end of recovery is reached. Best regards, depesz -- The best thing about modern society is how easy it is to avoid contact with it. http://depesz.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On Tue, Mar 29, 2011 at 11:13:07AM +0900, Fujii Masao wrote: Yes, it's intentional. In streaming replication, at first the master must stream a backup history file to the standby in order to let it know the recovery ending position. But streaming replication doesn't have ability to send a text file, so we changed the code so that the recovery ending position was also written as WAL record which can be streamed. ok, this makes sense. BTW, in my system, I use another trick to take a base backup from the standby: (All of these operations are expected to be performed on the standby) (1) Run CHECKPOINT (2) Copy pg_control to temporary area (3) Take a base backup of $PGDATA (4) Copy back pg_control from temporary area to the backup taken in (2). (5) Calculate the recovery ending position from current pg_control in $PGDATA by using pg_controldata When recovery starts from that backup, it doesn't automatically check whether it has reached the ending position or not. So the user needs to check that manually. Yeah, this trick is very fragile and complicated. I'd like to improve the way in 9.2. I know about it, but I feel very worried about doing stuff like this - i.e. meddling with internal files of pg. Best regards, depesz -- The best thing about modern society is how easy it is to avoid contact with it. http://depesz.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On Tue, Mar 29, 2011 at 6:46 PM, hubert depesz lubaczewski dep...@depesz.com wrote: Did you use recovery.conf to start standalone PostgreSQL? If not, recovery doesn't check whether it reaches the recovery ending position or not. So I guess no problem didn't happen. no, i don't use. hmm .. i am nearly 100% certain that previous pgs did in fact check if the end of recovery is reached. Yes. In 8.4, that was checked only when starting recovery from the backup (i.e., which includes backup_label and backup history file) without recovery.conf. But in 9.0, the behavior was changed so that only archive recovery (i.e., with recovery.conf) checks that. IIRC, we don't have strong opinion about this change. We should revert, in order to make even crash recovery check whether it reaches the ending location? Of course, even if we do that, your problem is not solved at all. So I think that the right direction is to implement the ability to easily take a base backup from the standby, in 9.2. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 3:49 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: It would be better to just support it (recv|fsync|apply), or no syncrep at all. Syncrep is incomplete without it. Agreed. I have trouble viewing the idea that it would be better not to ship sync rep at all than to add more features to it as a serious proposal. Presumably, anyone who is sad that sync rep doesn't have all of the options they might want would be even sadder to hear that we went through a whole development cycle and ended up with nothing at all. Even if we did agree to take this patch, there will certainly be more features that someone might want and not have, such as the ability to sync with multiple standbys at once. More than that, I think we should evaluate this patch on a cost/benefit ratio, rather than trying to apply to it all those procedural fences that we don't have, and that we don't have the size to benefit from. As a community, we've adopted a development plan that proceeds in cycles. For the last several releases, we have had four CommitFests in each release cycle, followed by a feature freeze and eventually by beta and final release. It's certainly a valid question to ask how well that procedure has served us. It does not seem likely to me that we can continue to produce quality releases if we don't at some point cut off the flow of new features into the source tree and work on stabilizing the code we've already got, and I believe the point for that was agreed by a large number of developers who sat in a room at PGCon last year to be on or about February 15th. We ended up extending that by a couple of weeks, to make sure that we had a process that was FAIR: we didn't want patches that had been in the pipeline for a very long time to get postponed to 9.2 because no committer had had a chance to work on them yet. However, we also bumped MANY patches to 9.2 because they weren't in sufficiently good shape soon enough. If we accept this patch now because a bunch of people say they really, really want it, isn't that unfair to the people to whom we've already said sorry, the deadline has passed? Of course, there is always going to be some gray area. I argued for committing the replication_timeout patch because I believe the fact that we haven't got that feature is almost a bug - it interferes significantly with the usability of replication in general, and it will be an even more serious problem with sync rep, where a hung standby connection will not only mean that nothing is replicating but also that no write transactions can be processed at all. However, you could make the opposite argument - that it's really a new feature - and therefore we ought not to commit it. So far no one has taken that position, but it's certainly a reasonable argument. Likewise, there is ongoing discussion on the collations thread about which of those changes are necessary for this release, and which ones are things that ought to be postponed to a future release. I haven't gotten too involved in those discussions because I don't really understand the underlying issues, but I think that's an important discussion. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
On Tue, Mar 29, 2011 at 1:38 AM, Noah Misch n...@leadboat.com wrote: A few years ago, this list had a brief conversation on $SUBJECT: http://archives.postgresql.org/message-id/1215443493.4051.600.ca...@ebony.site What is notable/surprising about the behavior when two backends have different values for deadlock_timeout? After sleeping to acquire a lock, each backend will scan for deadlocks every time its own deadlock_timeout elapses. Some might be surprised that a larger-deadlock_timeout backend can still be the one to give up; consider this timeline: Backend Time Command A N/A SET deadlock_timeout = 1000 B N/A SET deadlock_timeout = 100 A 0 LOCK t B 50 LOCK u A 100 LOCK u B 1050 LOCK t (Backend A gets an ERROR at time 1100) More generally, one cannot choose deadlock_timeout values for two sessions such that a specific session will _always_ get the ERROR. However, one can drive the probability rather high. Compare to our current lack of control. Is some other behavior that only arises when backends have different deadlock_timeout settings more surprising than that one? If we could relax deadlock_timeout to a GucContext below PGC_SIGHUP, it would probably need to stop at PGC_SUSET for now. Otherwise, an unprivileged user could increase deadlock_timeout to hide his lock waits from log_lock_waits. One could remove that limitation by introducing a separate log_lock_waits timeout, but that patch would be significantly meatier. Some might also object to PGC_USERSET on the basis that a user could unfairly preserve his transaction by setting a high deadlock_timeout. However, that user could accomplish a similar denial of service by idly holding locks or trying deadlock-prone lock acquisitions in subtransactions. I'd be inclined to think that PGC_SUSET is plenty. It's actually not clear to me what the user could usefully do other than trying to preserve his transaction by setting a high deadlock_timeout - what is the use case, other than that? Is it worth thinking about having an explicit setting for deadlock priority? That'd be more work, of course, and I'm not sure it it's worth it, but it'd also provide stronger guarantees than you can get with this proposal. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
On Tue, Mar 29, 2011 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: Is it worth thinking about having an explicit setting for deadlock priority? That'd be more work, of course, and I'm not sure it it's worth it, but it'd also provide stronger guarantees than you can get with this proposal. Priority makes better sense, I think. That's what we're trying to control after all. But you would need to change the way the deadlock detector works... -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
Robert Haas robertmh...@gmail.com writes: On Tue, Mar 29, 2011 at 1:38 AM, Noah Misch n...@leadboat.com wrote: What is notable/surprising about the behavior when two backends have different values for deadlock_timeout? I'd be inclined to think that PGC_SUSET is plenty. It's actually not clear to me what the user could usefully do other than trying to preserve his transaction by setting a high deadlock_timeout - what is the use case, other than that? Yeah, that was my reaction too: what is the use case for letting different backends have different settings? It fails to give any real guarantees about who wins a deadlock, and I can't see any other reason for wanting session-specific settings. I don't know how difficult a priority setting would be. IIRC, the current deadlock detector always kills the process that detected the deadlock, but I *think* that's just a random choice and not an essential feature. If so, it'd be pretty easy to instead kill the lowest-priority xact among those involved in the deadlock. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Replication server timeout patch
Fujii Masao masao.fu...@gmail.com writes: On Mon, Mar 28, 2011 at 7:49 PM, Heikki Linnakangas Should we use COMMERROR instead of ERROR if we fail to put the socket in the right mode? Maybe. COMMERROR exists to keep us from trying to send an error report down a failed socket. I would assume (perhaps wrongly) that walsender/walreceiver don't try to push error reports across the socket anyway, only to the postmaster log. If correct, there is no need for COMMERROR, and using it just muddies the code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI bug?
YAMAMOTO Takashi y...@mwd.biglobe.ne.jp wrote: [no residual SIReadLock] i read it as there are many (7057) SIReadLocks somehow leaked. am i wrong? No, I am. Could you send the full SELECT * of pg_locks when this is manifest? (Probably best to do that off-list.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
hubert depesz lubaczewski dep...@depesz.com wrote: it worked. now the slave2 is working as stand alone. what does it tell us? will any work happening after checkpoint break it anyway? I'm less sure about what will put it into a bad state again than I was that an immediate checkpoint would put you into a good state. I have a vague feeling that I've seen or heard something which suggests that doing this during a spread checkpoint might be a problem as things currently stand. I can't be more specific without digging through code, and I'm pretty swamped at the moment. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
On Tue, Mar 29, 2011 at 2:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: IIRC, the current deadlock detector always kills the process that detected the deadlock, but I *think* that's just a random choice and not an essential feature. If so, it'd be pretty easy to instead kill the lowest-priority xact among those involved in the deadlock. I think it was just easier. To kill yourself you just exit with an error. To kill someone else you have to deliver a signal and hope the other process exits cleanly. There are a bunch of things to wonder about too. If you don't kill yourself then you might still be in a deadlock cycle so presumably you have to reset the deadlock timer? What if two backends both decide to kill the same other backend. Does it handle getting a spurious signal late cleanly? How does it know which transaction the signal was for? Alternatively we could have the deadlock timer reset all the time and fire repeatedly. Then we could just have all backends ignore the deadlock if they're not the lowest priority session in the cycle. But this depends on everyone knowing everyone else's priority (and having a consistent view of it). -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
Excerpts from Greg Stark's message of mar mar 29 11:15:50 -0300 2011: Alternatively we could have the deadlock timer reset all the time and fire repeatedly. Then we could just have all backends ignore the deadlock if they're not the lowest priority session in the cycle. But this depends on everyone knowing everyone else's priority (and having a consistent view of it). Presumably it'd be published in MyProc before going to sleep, so it'd be available for everyone and always consistent. Not sure if this requires extra locking, though. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
Joseph Adams joeyadams3.14...@gmail.com writes: Done. The new extension interface isn't exactly compatible with the old, so I dropped support for PostgreSQL 8.4 from the module. I suppose I could maintain a back-ported json module separately. In fact it is, but there's some history hiding the fact. I'm overdue to another doc patch on the matter, but it's quite simple. You don't need to use MODULE_PATHNAME in recent enough versions of PostgreSQL, meaning any version that's not currently EOL'ed. Just use $libdir and the backend code will be happy with it. That means you don't need the .sql.in file either. You don't need to use the control file property module_pathname either in most cases, that's only useful if you are building more than one extension from the same Makefile. So just use $libdir/json in your json.sql file and be done with it. Your extension is now compatible with both pre-9.1 and 9.1. I'm not sure how to spell that in the docs though, any help here would be welcome. Also, do we want to adapt contrib to be better examples, or do we want contrib to remain full of its history? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
Robert Haas robertmh...@gmail.com writes: If we accept this patch now because a bunch of people say they really, really want it, isn't that unfair to the people to whom we've already said sorry, the deadline has passed? No, because each time we're talking procedures we're forgetting about a simple fact. Commiters have the direct responsibility of the code, that is why pushing work from non-commiters takes so much time. Commiting your own code, you don't have a steep learning curve, and you don't have to understand the use case and get convinced. So the rules are not the same for commiter patches and contributor patches, and there's no good in trying to have them the same or pretending they are. In particular, only commiters are able to finish and polish the work between the last commit fest and beta, and then they will be on the hook to get to release candidate and release. But you know all that better than I do. I don't want a release as soon as possible, I want the best we are able to provide, and I think adding in current $subject patch helps reaching this goal. include baring show stoppers QA disclaimer Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On 29.03.2011 14:27, Fujii Masao wrote: On Tue, Mar 29, 2011 at 6:46 PM, hubert depesz lubaczewski dep...@depesz.com wrote: Did you use recovery.conf to start standalone PostgreSQL? If not, recovery doesn't check whether it reaches the recovery ending position or not. So I guess no problem didn't happen. no, i don't use. hmm .. i am nearly 100% certain that previous pgs did in fact check if the end of recovery is reached. Yes. In 8.4, that was checked only when starting recovery from the backup (i.e., which includes backup_label and backup history file) without recovery.conf. But in 9.0, the behavior was changed so that only archive recovery (i.e., with recovery.conf) checks that. IIRC, we don't have strong opinion about this change. We should revert, in order to make even crash recovery check whether it reaches the ending location? Hmm, why did we change that? It seems like a mistake, the database is not consistent until you reach the backup stop location, whether or not you're doing archive recovery. +1 for reverting that, and backpatching it as well. pg_basebackup -x, which includes all the WAL required to restore in the pg_xlog directory of the base backup itself, is also affected. Without the check that you reach the end-of-backup, an aborted base backup will appear to restore fine, even though some WAL segments are missing and the backup is incomplete. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Replication server timeout patch
On Tue, Mar 29, 2011 at 9:24 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Mon, Mar 28, 2011 at 7:49 PM, Heikki Linnakangas Should we use COMMERROR instead of ERROR if we fail to put the socket in the right mode? Maybe. COMMERROR exists to keep us from trying to send an error report down a failed socket. I would assume (perhaps wrongly) that walsender/walreceiver don't try to push error reports across the socket anyway, only to the postmaster log. If correct, there is no need for COMMERROR, and using it just muddies the code. I don't think that's how it works. The error the server sends is copied into some of the messages in the client log, which is really useful for debugging. ERROR: can't connect to the server (server said: you're not authorized) ...or something like that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Alpha 5 is now available for testing
The fifth alpha release for PostgreSQL version 9.1, 9.1alpha5, is now available. There are no new major features in this alpha release as compared to 9.1alpha4, but there are many minor bug fixes and improvements to features added in 9.1alpha4 and earlier alpha releases. It is expected that no major new features will be added before final release; this is likely to be the final alpha release for PostgreSQL 9.1. Please download, install, and test Alpha5. We depend on your bug reports and feedback in order to proceed to 9.1beta and to final release. The more testing you do, the sooner 9.1 will be available. Thank you to the many users who reported bugs in earlier alphas. Most of the 148 changes and fixes between Alpha4 and Alpha5 were around two major features, per-column collations and synchronous replication. Work on per-column collations included some major refactoring, adding support for it in all features of PostgreSQL, and changes to the column collations API and system catalogs. Multiple reported bugs were fixed in synchronous replication including lockups, issues with recovery mode, and replication being very slow with fsync = off. If you tested either of these features, please retest as the code has changed significantly since Alpha4. Other changes included: * add post-creation hook for extensions * numerous additions and corrections to documentation and release notes * allow valid-on-creation foreign keys as column constraints * refactor of min/max aggregate optimization * fix potential race condition with pg_basebackup * fix PL/Python array memory leak * raise maximum value for many timeout configuration settings * fix handling of unknown literals in UNION queries * fix some division-by-zero issues in the code * cleanup some variable handling in ECPG * fix some makefile problems introduced in Alpha4 * make permissions for COMMENT ON ROLE consistent The new features which are expected to be available in PostgreSQL 9.1 are documented in the release notes, at http://developer.postgresql.org/pgdocs/postgres/release-9-1.html. If you are able to help with organized alpha testing, please see the Alpha/Beta testing page: http://wiki.postgresql.org/wiki/HowToBetaTest Alpha releases are not stable and should never be used in production; they are for testing new features only. There is no guarantee that any features or APIs present in the alphas will be present, or the same, in the final release. Alpha release information page: http://www.postgresql.org/developer/alpha Download the alpha release here: http://www.postgresql.org/ftp/source/v9.1alpha5/ Alpha releases are primarily made in source code form only. Binary packages for some operating systems will be prepared in the coming days. -- Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 12:57 PM, Robert Haas robertmh...@gmail.com wrote: However, we also bumped MANY patches to 9.2 because they weren't in sufficiently good shape soon enough. If we accept this patch now because a bunch of people say they really, really want it, isn't that unfair to the people to whom we've already said sorry, the deadline has passed? Of course, there is always going to be some gray area. I argued for committing the replication_timeout patch because I believe the fact that we haven't got that feature is almost a bug - it interferes significantly with the usability of replication in general, and it will be an even more serious problem with sync rep, where a hung standby connection will not only mean that nothing is replicating but also that no write transactions can be processed at all. However, you could make the opposite argument - that it's really a new feature - and therefore we ought not to commit it. So far no one has taken that position, but it's certainly a reasonable argument. The questions and issues you raise are real and I have no idea how to judge them. All I know is that if we spent less time discussing procedural issues, we'd get a lot more done and much more amicably also. I think What makes PostgreSQL better?. I think about a rounded feature set and treat that just as I would a bug. I know that's not the same for everybody. I'm happy there are people looking at replication timeouts also. Regrettably I don't have enough time for everything I would like to see. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 10:48 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So the rules are not the same for commiter patches and contributor patches, and there's no good in trying to have them the same or pretending they are. In particular, only commiters are able to finish and polish the work between the last commit fest and beta, and then they will be on the hook to get to release candidate and release. But you know all that better than I do. Committers can and do get away with slipping things in later than non-committers, and to some extent that's OK for the reasons you mention. But Alvaro was very gracious in conceding that it was a bit too late to push in his key lock patch, as was his employer, JD. They didn't like it, but they accepted that it was necessary to move the community, overall, forward, and to avoid a really long beta period during which, really, nobody gets to do anything at all interesting. We cannot have one standard for features that CommandPrompt really wants committed and a different standard for features that 2ndQuadrant or, say, EnterpriseDB, really want committed. I completely disagree that committers are the only ones who can finish and polish work between the last CommiFest and beta. Fujii Masao, Kevin Grittner, Yeb Havinga, and Yamamoto Takashi all come to mind as people who have been very, very helpful in moving us toward beta through careful testing and code review. I have no fear at all about our ability to maintain SSI even though there is not one committer who fully understands it all, because every bug report that comes in gets a response within hours and a patch within days. The limiting factor there has actually been how long it's taken someone to look and test those patches, not how quickly they've been produced. I think the reality is exactly the other way around: committers are not the people who get the opportunity to fix other people's bugs; they are the people who are *expected* to fix other people's bugs when no one else will. If it's your perception that the (mostly quite minor) changes that I've made to sync rep are somehow for purposes of self-aggrandizement or a desire to micromanage everything that happens in the backend, then I'm sorry for that. I'll readily admit that I have strong opinions on lots of topics, especially but not only PostgreSQL-related topics; but I would be way happier to have spent the last couple of weeks developing new features than swatting bugs. Had I done that, though, I think that not as many bugs would have gotten swatted. So I did it. Whether that makes me a helpful community guy who tries to ensure a quality release or a total jerk who interjects his nose into other people's business is, of course, a matter of opinion. Even today, anyone who would like to write a patch to address more than one of the open items is more than welcome to do so, and I would really appreciate it, even I or someone else ends up having to adjust it a bit before committing. There are at least three issues on the open items list that are obvious candidates for someone to pick up: - fix attinhcount tracking - Typed-tables patch broke pg_upgrade - comments on SQL/MED objects I volunteered to pick up the last one, but I'd be more than happy if the person who reported the problem had already provided the patch. Or if someone else wanted to write the patch. That would be awesome. In my view, the question we should be asking ourselves here is not - why are Tom and Robert getting to make all these commits? - but - where is everybody else who should be helping out? If the answer is well we don't have time to work on this because we all have day jobs we have to do to get paid, then I accept that. But that moves getting to commit changes at a late date from the privilege bucket into the responsibility bucket. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 5:40 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 29, 2011 at 10:48 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So the rules are not the same for commiter patches and contributor patches, and there's no good in trying to have them the same or pretending they are. In particular, only commiters are able to finish and polish the work between the last commit fest and beta, and then they will be on the hook to get to release candidate and release. But you know all that better than I do. Committers can and do get away with slipping things in later than non-committers, and to some extent that's OK for the reasons you mention. But Alvaro was very gracious in conceding that it was a bit too late to push in his key lock patch, as was his employer, JD. They didn't like it, but they accepted that it was necessary to move the community, overall, forward, and to avoid a really long beta period during which, really, nobody gets to do anything at all interesting. We cannot have one standard for features that CommandPrompt really wants committed and a different standard for features that 2ndQuadrant or, say, EnterpriseDB, really want committed. I completely disagree that committers are the only ones who can finish and polish work between the last CommiFest and beta. Fujii Masao, Kevin Grittner, Yeb Havinga, and Yamamoto Takashi all come to mind as people who have been very, very helpful in moving us toward beta through careful testing and code review. I have no fear at all about our ability to maintain SSI even though there is not one committer who fully understands it all, because every bug report that comes in gets a response within hours and a patch within days. The limiting factor there has actually been how long it's taken someone to look and test those patches, not how quickly they've been produced. I think the reality is exactly the other way around: committers are not the people who get the opportunity to fix other people's bugs; they are the people who are *expected* to fix other people's bugs when no one else will. If it's your perception that the (mostly quite minor) changes that I've made to sync rep are somehow for purposes of self-aggrandizement or a desire to micromanage everything that happens in the backend, then I'm sorry for that. I'll readily admit that I have strong opinions on lots of topics, especially but not only PostgreSQL-related topics; but I would be way happier to have spent the last couple of weeks developing new features than swatting bugs. Had I done that, though, I think that not as many bugs would have gotten swatted. So I did it. Whether that makes me a helpful community guy who tries to ensure a quality release or a total jerk who interjects his nose into other people's business is, of course, a matter of opinion. Even today, anyone who would like to write a patch to address more than one of the open items is more than welcome to do so, and I would really appreciate it, even I or someone else ends up having to adjust it a bit before committing. There are at least three issues on the open items list that are obvious candidates for someone to pick up: - fix attinhcount tracking - Typed-tables patch broke pg_upgrade - comments on SQL/MED objects I volunteered to pick up the last one, but I'd be more than happy if the person who reported the problem had already provided the patch. Or if someone else wanted to write the patch. That would be awesome. In my view, the question we should be asking ourselves here is not - why are Tom and Robert getting to make all these commits? - but - where is everybody else who should be helping out? If the answer is well we don't have time to work on this because we all have day jobs we have to do to get paid, then I accept that. But that moves getting to commit changes at a late date from the privilege bucket into the responsibility bucket. Robert, Everybody wants us to be polite and respectful with each other. Writing such long emails seems to be just filibustering to me. I doubt anyone has read and considered every word, there are just too many. A form of disrespect. Main thing I note is that you could have reviewed my patch in the time its taken to discuss these procedural issues. Why are they more important? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] SHMEM_INDEX_SIZE exceeded on startup
I doubt that this is going to matter much, and should only have a trivial impact on shared space calculations and postmaster and connection startup time, but just as a matter of principle we might want to set SHMEM_INDEX_SIZE at least as large as the number of entries in ShmemIndex. At startup that seems to be 40 as of current HEAD. Trivial patch attached. -Kevin *** a/src/include/storage/shmem.h --- b/src/include/storage/shmem.h *** *** 50,56 extern void RequestAddinShmemSpace(Size size); /* max size of data structure string name */ #define SHMEM_INDEX_KEYSIZE(48) /* estimated size of the shmem index table (not a hard limit) */ ! #define SHMEM_INDEX_SIZE (32) /* this is a hash bucket in the shmem index table */ typedef struct --- 50,56 /* max size of data structure string name */ #define SHMEM_INDEX_KEYSIZE(48) /* estimated size of the shmem index table (not a hard limit) */ ! #define SHMEM_INDEX_SIZE (64) /* this is a hash bucket in the shmem index table */ typedef struct -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Triggers on system catalog
Yes, you'd need to hack Postgres to add that capabilty, but that's too darned difficult since Postgres internally does not use SQL to perform the DML operations on system catalog. I'd suggest searching for past discussions on -hackers around this topic. Regards, On Tue, Mar 29, 2011 at 5:37 AM, Shridhar Polas shridharpo...@gmail.comwrote: Thanks Gurjeet. You got it absolutely correct what I am trying to do. It is clear to me that we can not write trigger on system catalog. Can we hack postgres somehow to have triggers create on system catalog? I am using postgres 9.0 version. Thanks, Shridhar On Mon, Mar 28, 2011 at 8:03 PM, Gurjeet Singh singh.gurj...@gmail.comwrote: On Mon, Mar 28, 2011 at 10:19 AM, Shridhar Polas shridharpo...@gmail.com wrote: Hi, Is there a way to create triggers on system catalog tables like pg_class, pg_attribute etc...? No, Postgres does not support triggers on system catalogs; we do not have DDL triggers either, if that's what you were trying to achieve. -- Gurjeet Singh EnterpriseDB Corporation The Enterprise PostgreSQL Company
Re: [HACKERS] Triggers on system catalog
On Mon, Mar 28, 2011 at 10:19 AM, Shridhar Polas shridharpo...@gmail.com wrote: Hi, Is there a way to create triggers on system catalog tables like pg_class, pg_attribute etc...? No, this isn't supported, and, since the normal alterations of the schema involve manipulating these tables, such an addition would be fraught with the risk of breaking system behavior. - Your trigger function needs to be aware of *ALL* functionalities that involve those tables. For instance, a trigger on pg_class needs to be able to cope with changes relating to views, sequences, inherited tables, not just its use for tables. - There are liable to be timing issues, as a series of updates to these tables won't in general be consistent until *all* the updates are done. E.g. - consider that when a table is created, there'd be an insert to pg_class, and then, later, inserts to pg_attribute. A trigger firing against pg_class would capture a partially-completed table creation, which mayn't be what you were expecting. A proposal to adding triggers to system catalog tables won't be terribly popular. -- http://linuxfinances.info/info/postgresql.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] deadlock_timeout at PGC_SIGHUP?
On Tue, Mar 29, 2011 at 08:26:44AM -0400, Robert Haas wrote: I'd be inclined to think that PGC_SUSET is plenty. Agreed. A superuser who would have liked PGC_USERSET can always provide a SECURITY DEFINER function. It's actually not clear to me what the user could usefully do other than trying to preserve his transaction by setting a high deadlock_timeout - what is the use case, other than that? The other major use case is reducing latency in deadlock-prone transactions. By reducing deadlock_timeout for some or all involved transactions, the error will arrive earlier. Is it worth thinking about having an explicit setting for deadlock priority? That'd be more work, of course, and I'm not sure it it's worth it, but it'd also provide stronger guarantees than you can get with this proposal. That is a better UI for the first use case. I have only twice wished to tweak deadlock_timeout: once for the use case you mention, another time for that second use case. Given that, I wouldn't have minded a rough UI. If you'd use this often and assign more than two or three distinct priorities, you'd probably appreciate the richer UI. Not sure how many shops fall in that group. Thanks, nm -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
Joseph Adams joeyadams3.14...@gmail.com writes: It would be nice if I could make a Makefile conditional that skips the relocatable test and loads init-pre9.1.sql if the new extension interface isn't available. Is there a Makefile variable or something I can use to do this? You can use VERSION and MAJORVERSION variables, those are defined in Makefile.global and available as soon as you did include the PGXS Makefile. Also, should uninstall_json.sql be named something else, like json--uninstall--0.1.sql ? You don't need no uninstall script no more, try DROP EXTENSION json; and DROP EXTENSION json CASCADE; Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On 3/29/11 7:48 AM, Dimitri Fontaine wrote: I don't want a release as soon as possible, I want the best we are able to provide, and I think adding in current $subject patch helps reaching this goal. include baring show stoppers QA disclaimer There will *always* be more work we can do on sync rep. If we hold the release until we're done tinkering with it, we'll never release. Our project has had a chronic issue with not being able to progress from feature freeze to release in a timely fashion for years, and the sort of argument expressed above isn't helping. The relevant question is: is sync rep unusable by a large portion of our users because this feature was stripped from what got committed, or is it a value-add feature which makes synch rep nicer? If the former, it's a bug and we should patch it; if the latter, it should wait until 9.2. Writing such long emails seems to be just filibustering to me. I doubt anyone has read and considered every word, there are just too many. A form of disrespect. Simon, Robert has been nothing but respectful to you. You can't accuse him of being rude and hostile just because he disagrees with you. Overselling his case, certainly. But very politely. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
On Tue, Mar 29, 2011 at 2:42 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Also, should uninstall_json.sql be named something else, like json--uninstall--0.1.sql ? You don't need no uninstall script no more, try DROP EXTENSION json; and DROP EXTENSION json CASCADE; It's there for pre-9.1, where DROP EXTENSION is not available. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
On Tue, Mar 29, 2011 at 2:42 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Joseph Adams joeyadams3.14...@gmail.com writes: It would be nice if I could make a Makefile conditional that skips the relocatable test and loads init-pre9.1.sql if the new extension interface isn't available. Is there a Makefile variable or something I can use to do this? You can use VERSION and MAJORVERSION variables, those are defined in Makefile.global and available as soon as you did include the PGXS Makefile. The problem is, I'd have to include the PGXS Makefile before defining a parameter used by the PGXS Makefile. I could include the PGXS Makefile twice, once at the top, and again at the bottom, but that produces a bunch of ugly warnings like: ../../src/Makefile.global:418: warning: overriding commands for target `submake-libpq' Not only that, but MAJORVERSION is formatted as a decimal (like 9.1 or 8.4). Although I could use some hacky string manipulation, or compare MAJORVERSION against all prior supported versions, either approach would be needlessly error-prone. I'm thinking the pg_config utility should either make PG_VERSION_NUM (e.g. 90100) available, or it should define something indicating the presence of the new extension system. Joey Adams -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Problem with pg_upgrade?
I have gotten two reports via IRC that months after using 9.0 pg_upgrade, some of the clog files have been removed while there is still data in the table needing those clog files. These reports came to me through Rhodiumtoad who analyzed the systems. Looking at pg_upgrade, I am concerned that somehow autovaccum is running in frozen mode before I have restored the frozen xids for the table or database. Here is the code I am using: snprintf(cmd, sizeof(cmd), SYSTEMQUOTE \%s/pg_ctl\ -l \%s\ -D \%s\ -o \-p %d -c autovacuum=off -c autovacuum_freeze_max_age=20\ start \%s\ 21 SYSTEMQUOTE, bindir, Does anyone have any other suggestions on how to make sure autovacuum does not run in freeze mode? I know 'autovacuum=off' turns off normal autovacuum. Would increasing autovacuum_naptime help? It looks like the autovacuum code sleeps before processing anything, but I am not certain. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
Joseph Adams joeyadams3.14...@gmail.com writes:
would be needlessly error-prone. I'm thinking the pg_config utility
should either make PG_VERSION_NUM (e.g. 90100) available, or it should
define something indicating the presence of the new extension system.
Here's the ugly trick from ip4r, that's used by more extension:
PREFIX_PGVER = $(shell echo $(VERSION) | awk -F. '{ print $$1*100+$$2 }')
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 2:48 PM, Josh Berkus j...@agliodbs.com wrote: Writing such long emails seems to be just filibustering to me. I doubt anyone has read and considered every word, there are just too many. A form of disrespect. Simon, Robert has been nothing but respectful to you. You can't accuse him of being rude and hostile just because he disagrees with you. Overselling his case, certainly. But very politely. If hostile's wanted, then we can come up with something hostile ;-). Much better is to try to take a different perspective. There are a number of features that were *turned down* for 9.1, deferred for 9.2, because they weren't sufficiently ready at the appointed time. There were 26 patches Returned with Feedback; there are likely several of them which, if given the special handling given to Synchronous Replication, could have been drawn into 9.1. It's only fair to try to get the release out, so that these people that have been waiting patiently have opportunity to get their submissions into 9.2. The longer they wait, the more reason to get hostile for better reason... -- http://linuxfinances.info/info/linuxdistributions.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] gcc 4.6 warnings -Wunused-but-set-variable
As you might have heard, GCC 4.6 was released the other day. It
generates a bunch of new warnings with the PostgreSQL source code, most
of which belong to the new warning scenario -Wunused-but-set-variable,
which is included in -Wall.
Attached is a patch that gets rid of most of these. As you can see,
most of these remove real leftover garbage. The line I marked in
pg_basebackup.c might be an actual problem: It goes through a whole lot
to figure out the timeline and then doesn't do anything with it. In
some other cases, however, one might argue that the changes lose some
clarity, such as when dropping the return value of strtoul() or
va_arg(). How should we proceed? In any case, my patch should be
re-reviewed for any possible side effects that I might have hastily
removed.
diff --git i/contrib/isn/isn.c w/contrib/isn/isn.c
index 46e904b..b698cb0 100644
--- i/contrib/isn/isn.c
+++ w/contrib/isn/isn.c
@@ -341,8 +341,7 @@ ean2isn(ean13 ean, bool errorOK, ean13 *result, enum isn_type accept)
enum isn_type type = INVALID;
char buf[MAXEAN13LEN + 1];
- char *firstdig,
- *aux;
+ char *aux;
unsigned digval;
unsigned search;
ean13 ret = ean;
@@ -354,7 +353,7 @@ ean2isn(ean13 ean, bool errorOK, ean13 *result, enum isn_type accept)
/* convert the number */
search = 0;
- firstdig = aux = buf + 13;
+ aux = buf + 13;
*aux = '\0';/* terminate string; aux points to last digit */
do
{
@@ -528,8 +527,7 @@ ean2string(ean13 ean, bool errorOK, char *result, bool shortType)
const unsigned (*TABLE_index)[2];
enum isn_type type = INVALID;
- char *firstdig,
- *aux;
+ char *aux;
unsigned digval;
unsigned search;
char valid = '\0'; /* was the number initially written with a
@@ -546,7 +544,7 @@ ean2string(ean13 ean, bool errorOK, char *result, bool shortType)
/* convert the number */
search = 0;
- firstdig = aux = result + MAXEAN13LEN;
+ aux = result + MAXEAN13LEN;
*aux = '\0';/* terminate string; aux points to last digit */
*--aux = valid;/* append '!' for numbers with invalid but
* corrected check digit */
diff --git i/contrib/pageinspect/fsmfuncs.c w/contrib/pageinspect/fsmfuncs.c
index eca3230..38c4e23 100644
--- i/contrib/pageinspect/fsmfuncs.c
+++ w/contrib/pageinspect/fsmfuncs.c
@@ -35,7 +35,6 @@ Datum
fsm_page_contents(PG_FUNCTION_ARGS)
{
bytea *raw_page = PG_GETARG_BYTEA_P(0);
- int raw_page_size;
StringInfoData sinfo;
FSMPage fsmpage;
int i;
@@ -45,7 +44,6 @@ fsm_page_contents(PG_FUNCTION_ARGS)
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
(errmsg(must be superuser to use raw page functions;
- raw_page_size = VARSIZE(raw_page) - VARHDRSZ;
fsmpage = (FSMPage) PageGetContents(VARDATA(raw_page));
initStringInfo(sinfo);
diff --git i/contrib/pgcrypto/pgp-s2k.c w/contrib/pgcrypto/pgp-s2k.c
index ef16caf..349234e 100644
--- i/contrib/pgcrypto/pgp-s2k.c
+++ w/contrib/pgcrypto/pgp-s2k.c
@@ -39,14 +39,12 @@ static int
calc_s2k_simple(PGP_S2K *s2k, PX_MD *md, const uint8 *key,
unsigned key_len)
{
- unsigned md_bs,
-md_rlen;
+ unsigned md_rlen;
uint8 buf[PGP_MAX_DIGEST];
unsigned preload;
unsigned remain;
uint8 *dst = s2k-key;
- md_bs = px_md_block_size(md);
md_rlen = px_md_result_size(md);
remain = s2k-key_len;
@@ -83,14 +81,12 @@ calc_s2k_simple(PGP_S2K *s2k, PX_MD *md, const uint8 *key,
static int
calc_s2k_salted(PGP_S2K *s2k, PX_MD *md, const uint8 *key, unsigned key_len)
{
- unsigned md_bs,
-md_rlen;
+ unsigned md_rlen;
uint8 buf[PGP_MAX_DIGEST];
unsigned preload = 0;
uint8 *dst;
unsigned remain;
- md_bs = px_md_block_size(md);
md_rlen = px_md_result_size(md);
dst = s2k-key;
@@ -129,8 +125,7 @@ static int
calc_s2k_iter_salted(PGP_S2K *s2k, PX_MD *md, const uint8 *key,
unsigned key_len)
{
- unsigned md_bs,
-md_rlen;
+ unsigned md_rlen;
uint8 buf[PGP_MAX_DIGEST];
uint8 *dst;
unsigned preload = 0;
@@ -143,7 +138,6 @@ calc_s2k_iter_salted(PGP_S2K *s2k, PX_MD *md, const uint8 *key,
cval = s2k-iter;
count = ((unsigned) 16 + (cval 15)) ((cval 4) + 6);
- md_bs = px_md_block_size(md);
md_rlen = px_md_result_size(md);
remain = s2k-key_len;
diff --git i/contrib/pgcrypto/px-hmac.c w/contrib/pgcrypto/px-hmac.c
index 16abc43..36efabd 100644
--- i/contrib/pgcrypto/px-hmac.c
+++ w/contrib/pgcrypto/px-hmac.c
@@ -52,13 +52,11 @@ static void
hmac_init(PX_HMAC *h, const uint8 *key, unsigned klen)
{
unsigned bs,
-hlen,
i;
uint8 *keybuf;
PX_MD *md = h-md;
bs = px_md_block_size(md);
- hlen = px_md_result_size(md);
keybuf = px_alloc(bs);
memset(keybuf, 0, bs);
diff --git i/contrib/pgcrypto/px.c w/contrib/pgcrypto/px.c
index 768c7c3..e3f5e26 100644
--- i/contrib/pgcrypto/px.c
+++ w/contrib/pgcrypto/px.c
@@ -162,14 +162,12 @@ combo_init(PX_Combo *cx, const uint8 *key, unsigned klen,
const uint8 *iv, unsigned ivlen)
{
int err;
- unsigned bs,
-ks,
+ unsigned ks,
ivs;
[HACKERS] Process local hint bit cache
In a previous thread (http://postgresql.1045698.n5.nabble.com/Set-hint-bits-upon-eviction-from-BufMgr-td4264323.html) I was playing with the idea of granting the bgwriter the ability to due last chance hint bit setting before evicting the page out. I still think might be a good idea, and it might be an especially good idea, especially in scenarios where you get set the PD_ALL_VISIBLE bit when writing out the page, a huge win in some cases. That said, it bugged me that it didn't really fix the general case of large data insertions and the subsequent i/o problems setting out the bits, which is my primary objective in the short term. So I went back to the drawing board, reviewed the archives, and came up with a new proposal. I'd like to see a process local clog page cache of around 1-4 pages (8-32kb typically) that would replace the current TransactionLogFetch last xid cache. It should be small, because I doubt more would really help much (see below) and we want to keep this data in the tight cpu caches since it's going to be constantly scanned. The cache itself will be the clog pages and a small header per page which will contain the minimum information necessary to match an xid against a page to determine a hit, and a count of hits. Additionally we keep a separate small array (say 100) of type xid (int) that we insert write into in a cache miss. So, cache lookup algorithm basically is: scan clog cache headers and check for hit if found (xid in range covered in clog page), header.hit_count++; else miss_array[miss_count++] = xid; A cache hit is defined about getting useful information from the page, that is a transaction being committed or invalid. When the miss count array fills, we sort it and determine the most commonly hit clog page that is not in the cache and use that information to replace pages in the cache if necessary, then reset the counts. Maybe we can add a minimum threshold of hits, say 5-10% if miss_array size for a page to be deemed interesting enough to be loaded into the cache. Interaction w/set hint bits: *) If a clog lookup faults through the cache, we basically keep the current behavior. That is, the hint bits are set and the page is marked BM_DIRTY and the hint bits get written back *) If we get a clog cache hit, that is the hint bits are not set but we pulled the transaction status from the cache, the hint bits are recorded on the page *but the page is not written back*, at least on hint bit basis alone. This behavior branch is more or less the BM_UNTIDY as suggested by haas (see archives), except it's only seen in 'cache hit' scenarios. We are not writing pages back because the cache is suggesting there is little/not benefit to write them back. Thus, if a single backend is scanning a lot of pages with transactions touching a very small number of clog pages, hint bits are generally not written back because they are not needed and in fact not helping. However, if the xid are spread around a large number of clog pages, we get the current behavior more or less (plus the overhead of cache maintenance). With the current code base, hint bits are very beneficial when the xid entropy is high and the number of repeated scan is high, and not so good when the xid entropy is low and the number of repeated scans is low. The process local cache attempts to redress this without disadvantaging the already good cases. Furthermore, if it can be proven that the cache overhead is epsilon, it's pretty unlikely to negatively impact anyone negatively, at lest, that's my hope. Traffic to clog will reduce (although not much, since i'd wager the current 'last xid' cache works pretty well), but i/o should be reduced, in some cases quite significantly for a tiny cpu cost (although that remains to be proven). merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dump --binary-upgrade vs. ALTER TYPE ... DROP ATTRIBUTE
I took a look at the open item concerning typed tables and pg_upgrade:
http://archives.postgresql.org/pgsql-hackers/2011-03/msg00767.php
pg_dump --binary-upgrade emits commands to recreate the dropped-column
situation of the database, and it does not currently understand that it must
alter the parent type when the subject is a typed table. Actually, pg_dump
doesn't handle dropped columns in composite types at all. pg_upgrade runs fine
on a database that received these commands, but the outcome is wrong:
create type t as (x int, y int);
create table has_a (tcol t);
insert into has_a values ('(1,2)');
table has_a; -- (1,2)
alter type t drop attribute y cascade, add attribute z int cascade;
table has_a; -- (1,)
table has_a; -- after pg_upgrade: (1,2)
Fixing that looks clear enough, but the right fix for the typed table issue is
less clear to me. The pg_attribute tuples for a typed table will include any
attributes dropped from the parent type after the table's creation, but not
those attributes dropped before the table's creation. Example:
create type t as (x int, y int);
create table is_a of t;
alter type t drop attribute y cascade;
create table is_a2 of t;
select * from pg_attribute where attrelid = 'is_a'::regclass;
select * from pg_attribute where attrelid = 'is_a2'::regclass;
To reproduce that catalog state, the dump would need to create the type, create
all typed tables predating the DROP ATTRIBUTE, and finally create typed tables
postdating the DROP ATTRIBUTE. That implies an extra dump entry for the DROP
ATTRIBUTE with the appropriate dependencies to compel that order of events. Is
there a better way?
nm
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 7:48 PM, Josh Berkus j...@agliodbs.com wrote: On 3/29/11 7:48 AM, Dimitri Fontaine wrote: I don't want a release as soon as possible, I want the best we are able to provide, and I think adding in current $subject patch helps reaching this goal. include baring show stoppers QA disclaimer There will *always* be more work we can do on sync rep. If we hold the release until we're done tinkering with it, we'll never release. Our project has had a chronic issue with not being able to progress from feature freeze to release in a timely fashion for years, and the sort of argument expressed above isn't helping. The relevant question is: is sync rep unusable by a large portion of our users because this feature was stripped from what got committed, or is it a value-add feature which makes synch rep nicer? If the former, it's a bug and we should patch it; if the latter, it should wait until 9.2. That's not relevant. Can something small be added for large benefit: yes! It's a complete misrepresentation to suggest this would hold up the release in any way and all the actual technical comments have been easily dismissed. We've already used more time writing these emails than it took me to write the patch ( 2 hours). This still can be reviewed and committed easily. This is not a new feature. It's something that's been in the design for months, was submitted in the recent CF and is a minor change with low risk. Given we've all been here before, I've done my homework on what is acceptable at this stage and this passes. There is nothing different between this and the replication timeout patch. I'm not suggesting that should be yanked as well, BTW. And I'm not getting paid a single penny for this. Just me, trying to add value for the PostgreSQL project. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 7:48 PM, Josh Berkus j...@agliodbs.com wrote: Writing such long emails seems to be just filibustering to me. I doubt anyone has read and considered every word, there are just too many. A form of disrespect. Simon, Robert has been nothing but respectful to you. You can't accuse him of being rude and hostile just because he disagrees with you. Overselling his case, certainly. But very politely. I haven't accused Robert of being rude or hostile, when did I say that? He seems perfectly polite to me. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
On Tue, Mar 29, 2011 at 02:56:52PM -0400, Joseph Adams wrote: On Tue, Mar 29, 2011 at 2:42 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Also, should uninstall_json.sql be named something else, like json--uninstall--0.1.sql ? You don't need no uninstall script no more, try DROP EXTENSION json; and DROP EXTENSION json CASCADE; It's there for pre-9.1, where DROP EXTENSION is not available. Anything going into the PostgreSQL code base will be for 9.2, so anything else would be a separate (if somewhat related) project. I suspect the code will be a good deal cleaner if you do just the 9.2+ version and see who wants it back-patched, if anyone does :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Triggers on system catalog
On Mar 29, 2011, at 2:17 PM, Christopher Browne cbbro...@gmail.com wrote: A proposal to adding triggers to system catalog tables won't be terribly popular. Well, I'd support it if I thought it had any chance of actually working, but I don't. I do think we need some kind way of capturing DDL events, though. I wonder if the object-access-hook stuff KaiGai and I did to support SE-PostgreSQL could be extended to meet this need... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with pg_upgrade?
On Tue, 2011-03-29 at 15:52 -0400, Bruce Momjian wrote: Does anyone have any other suggestions on how to make sure autovacuum does not run in freeze mode? Can you run in single user mode? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with pg_upgrade?
Excerpts from Jeff Davis's message of mar mar 29 21:27:34 -0300 2011: On Tue, 2011-03-29 at 15:52 -0400, Bruce Momjian wrote: Does anyone have any other suggestions on how to make sure autovacuum does not run in freeze mode? Can you run in single user mode? I asked the same thing. Apparently the problem is that it would make error handling a lot more difficult. I think it would be better to have some sort of option to disable autovacuum completely which would be used only during pg_upgrade. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] PostgreSQL sourcecode
Hi all i have been trying to compile the sourcecode for postgresql but keep getting an error message when running it in cygwin. it states: dllwrap: gcc exited with status 1 make[3]: *** [cygpq.dll] Error 1 make[3]: Leaving directory `/postgresql-9.0.3/postgresql-9.0.3/src/interfaces/li bpq' make[2]: *** [all] Error 2 make[2]: Leaving directory `/postgresql-9.0.3/postgresql-9.0.3/src/interfaces' make[1]: *** [all] Error 2 make[1]: Leaving directory `/postgresql-9.0.3/postgresql-9.0.3/src' make: *** [all] Error 2 what am trying to do is compile the sc and change the HeapTupleSatisfiesVisibility() to true. From looking at the downloaded sourcecode file within src/include/utils/tqual.h i cannot see anywhere it is set to false, so would appreciate it, if one could explain how i change it to true. thanks -- View this message in context: http://postgresql.1045698.n5.nabble.com/PostgreSQL-sourcecode-tp4270534p4270534.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dumpall
Hi there, I have been having problems using the pg_dumpall function. I am trying to dump all the information on my database onto a file or view it. Actually what am trying to achieve is by using the pg_dumpall i can view dead rows once the db has been dumped into a file. I had a look online and tried to run the pg_dumpall command from dos using this code: pg_dump Test aaron.db but came up with this error: C:\Program Files\PostgreSQL\9.0\binpg_dump Test aaron.db Password: pg_dump: [archiver (db)] connection to database Test failed: FATAL: password authentication failed for user Enabs test being the name of the database and aaron.db being the output file wanted it to view again am sure you might think the password authentication failed due to password, but changed the password, tried again and still no success.. If anyone can link me to a video or example on how the pgdump works or how to use it, i would be very grateful. Thanks -- View this message in context: http://postgresql.1045698.n5.nabble.com/pg-dumpall-tp4270539p4270539.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL sourcecode
On 3/29/11 5:53 PM, aaronenabs wrote: Hi all i have been trying to compile the sourcecode for postgresql but keep getting an error message when running it in cygwin. I don't believe that Cygwin is supported at this point. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dumpall
Aaron, I have been having problems using the pg_dumpall function. I am trying to dump all the information on my database onto a file or view it. Actually what am trying to achieve is by using the pg_dumpall i can view dead rows once the db has been dumped into a file. I had a look online and tried to run the pg_dumpall command from dos using this code: Please take your questions to the pgsql-novice mailing list. This list, pgsql-hackers is for working on the PostgreSQL code. http://archives.postgresql.org/pgsql-novice/ Thanks! -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with pg_upgrade?
On Tue, 2011-03-29 at 21:43 -0300, Alvaro Herrera wrote: I think it would be better to have some sort of option to disable autovacuum completely which would be used only during pg_upgrade. Sounds reasonable to me. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_last_xlog_receive_location()
In my understanding pg_last_xlog_receive_location() returns NULL if applied to non standby server: Get last transaction log location received and synced to disk by streaming replication. While streaming replication is in progress this will increase monotonically. But when streaming replication is restarted this will back off to the replication starting position, typically the beginning of the WAL file containing the current replay location. If recovery has completed this will remain static at the value of the last WAL record received and synced to disk during recovery. If streaming replication is disabled, or if it has not yet started, the function returns NULL. However my primary server returns non NULL. Am I missing something? -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
On Tue, Mar 29, 2011 at 4:02 PM, Dimitri Fontaine
dimi...@2ndquadrant.fr wrote:
Here's the ugly trick from ip4r, that's used by more extension:
PREFIX_PGVER = $(shell echo $(VERSION) | awk -F. '{ print $$1*100+$$2 }')
Thanks. I applied a minor variation of this trick to the JSON module,
so now it builds/installs/tests cleanly on both REL8_4_0 and HEAD
(though it won't work if you copy contrib/json into a pre-9.1
PostgreSQL source directory and type `make` without USE_PGXS=1).
I also went ahead and renamed uninstall_json.sql to
json--uninstall--0.1.sql (again, it's for pre-9.1 users) and removed
unnecessary trailing spaces.
Anything going into the PostgreSQL code base will be for 9.2, so
anything else would be a separate (if somewhat related) project. I
suspect the code will be a good deal cleaner if you do just the 9.2+
version and see who wants it back-patched, if anyone does :)
It's a trivial matter to remove backward compatibility from
contrib/json, if anybody wants me to do it. I can just remove
compat.[ch], */init-pre9.1.* , remove the PREFIX_PGVER trick from the
Makefile, remove a few lines in the source code, and maintain the
backported json module elsewhere. It's just a matter of whether or
not explicit backward-compatibility is desirable in modules shipped
with releases.
Joey Adams
add-json-contrib-module-4.patch.gz
Description: GNU Zip compressed data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL sourcecode
what c compiler would you suggest i use please thanks -- View this message in context: http://postgresql.1045698.n5.nabble.com/PostgreSQL-sourcecode-tp4270534p4270566.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL sourcecode
On 03/29/2011 09:03 PM, Josh Berkus wrote: On 3/29/11 5:53 PM, aaronenabs wrote: Hi all i have been trying to compile the sourcecode for postgresql but keep getting an error message when running it in cygwin. I don't believe that Cygwin is supported at this point. Of course it is. I go to a considerable deal of trouble to keep it that way. See http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=brolgadt=2011-03-29%2011%3A40%3A02 cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL sourcecode
On 03/29/2011 09:21 PM, aaronenabs wrote: what c compiler would you suggest i use please Building on Windows can be difficult. See http://www.postgresql.org/docs/current/static/install-windows.html for alternatives. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Replication server timeout patch
On Wed, Mar 30, 2011 at 1:04 AM, Robert Haas robertmh...@gmail.com wrote: COMMERROR exists to keep us from trying to send an error report down a failed socket. I would assume (perhaps wrongly) that walsender/walreceiver don't try to push error reports across the socket anyway, only to the postmaster log. If correct, there is no need for COMMERROR, and using it just muddies the code. I don't think that's how it works. The error the server sends is copied into some of the messages in the client log, which is really useful for debugging. ERROR: can't connect to the server (server said: you're not authorized) ...or something like that. Yes. Walsender sends its error message to walreceiver, and walreceiver writes it down to the server log. For example; FATAL: could not receive data from WAL stream: FATAL: requested WAL segment 00010016 has already been removed The second FATAL message is sent from walsender. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Triggers on system catalog
* Robert Haas (robertmh...@gmail.com) wrote: I do think we need some kind way of capturing DDL events, though. I wonder if the object-access-hook stuff KaiGai and I did to support SE-PostgreSQL could be extended to meet this need... My inclination would be 'probably', but it's not likely to really be the way we'd want to provide generalized DDL triggers.. Perhaps we might flip it around and have the SE-PG things be called from a DDL-trigger system, but as it's a security-type check, I'm not hugely thrilled by that idea either. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_last_xlog_receive_location()
On Wed, Mar 30, 2011 at 10:13 AM, Tatsuo Ishii is...@postgresql.org wrote: In my understanding pg_last_xlog_receive_location() returns NULL if applied to non standby server: Get last transaction log location received and synced to disk by streaming replication. While streaming replication is in progress this will increase monotonically. But when streaming replication is restarted this will back off to the replication starting position, typically the beginning of the WAL file containing the current replay location. If recovery has completed this will remain static at the value of the last WAL record received and synced to disk during recovery. If streaming replication is disabled, or if it has not yet started, the function returns NULL. However my primary server returns non NULL. Am I missing something? In the primary server which was brought up from the standby, pg_last_xlog_receive_location() returns non-NULL. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xlog_receive_location()
In the primary server which was brought up from the standby, pg_last_xlog_receive_location() returns non-NULL. Thanks. BTW, is there any reliable way to know a standby is promoting to primary? -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with streaming replication, backups, and recovery (9.0.x)
On Wed, Mar 30, 2011 at 12:54 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hmm, why did we change that? I'm not sure, but I guess that's because I missed the case where crash recovery starts from the backup :( It seems like a mistake, the database is not consistent until you reach the backup stop location, whether or not you're doing archive recovery. +1 for reverting that, and backpatching it as well. Agreed. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xlog_receive_location()
On Wed, Mar 30, 2011 at 11:08 AM, Tatsuo Ishii is...@postgresql.org wrote: BTW, is there any reliable way to know a standby is promoting to primary? You can know that from the server log. But there is no way to do that via SQL, I think. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xlog_receive_location()
BTW, is there any reliable way to know a standby is promoting to primary? You can know that from the server log. But there is no way to do that via SQL, I think. Ok. It seems I need to patch PostgreSQL. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Another swing at JSON
Excerpts from Joseph Adams's message of mar mar 29 22:15:11 -0300 2011:
On Tue, Mar 29, 2011 at 4:02 PM, Dimitri Fontaine
dimi...@2ndquadrant.fr wrote:
Here's the ugly trick from ip4r, that's used by more extension:
PREFIX_PGVER = $(shell echo $(VERSION) | awk -F. '{ print $$1*100+$$2 }')
Thanks. I applied a minor variation of this trick to the JSON module,
so now it builds/installs/tests cleanly on both REL8_4_0 and HEAD
(though it won't work if you copy contrib/json into a pre-9.1
PostgreSQL source directory and type `make` without USE_PGXS=1).
Why are you worrying with the non-PGXS build chain anyway? Just assume
that the module is going to be built with PGXS and things should just
work.
We've gone over this a dozen times in the past.
--
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xlog_receive_location()
On Tue, Mar 29, 2011 at 9:08 PM, Tatsuo Ishii is...@postgresql.org wrote: In the primary server which was brought up from the standby, pg_last_xlog_receive_location() returns non-NULL. Thanks. BTW, is there any reliable way to know a standby is promoting to primary? if you know the server was a standby and pg_is_in_recovery() returns false then it was promoted. is that what you want to know? or i misunderstood you? -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] crash-safe visibility map, take four
Hi, Should we do full-page write for visibilitymap all the time? Now, when clear visiblitymap, there is no full-page write for vm since we don't save buffer info in insert/update/delete's log. The full-page write is used to protect pages from disk failure. Without it, 1) set vm: the vm bits that should be set to 1 may still be 0 2) clear vm: the vm bits that should be set to 0 may still be 1 Are these true? Or the page is totally unpredictable? Another question: To address the problem in http://archives.postgresql.org/pgsql-hackers/2010-02/msg02097.php , should we just clear the vm before the log of insert/update/delete? This may reduce the performance, is there another solution? Thanks. -- GaoZengqi pgf...@gmail.com zengqi...@gmail.com On Fri, Mar 25, 2011 at 6:05 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Mar 23, 2011 at 6:16 AM, Jesper Krogh jes...@krogh.cc wrote: On 2011-03-22 21:43, Robert Haas wrote: I took a crack at implementing the first approach described above, which seems to be by far the simplest idea we've come up with to date. Patch attached. It doesn't seem to be that complicated, which could mean either that it's not that complicated or that I'm missing something. Feel free to point and snicker in the latter case. Looks simple, but there is now benefit... Your tests and discussion remind me that I haven't yet seen any tests that show that index-only scans would be useful for performance. Everyone just seems to be assuming that they make a huge difference, and that the difference is practically realisable in a common workload. Perhaps that's already been done and I just didn't notice? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Additional options for Sync Replication
On Tue, Mar 29, 2011 at 8:57 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 29, 2011 at 3:49 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: It would be better to just support it (recv|fsync|apply), or no syncrep at all. Syncrep is incomplete without it. Agreed. I have trouble viewing the idea that it would be better not to ship sync rep at all than to add more features to it as a serious proposal. Presumably, anyone who is sad that sync rep doesn't have all of the options they might want would be even sadder to hear that we went through a whole development cycle and ended up with nothing at all. Even if we did agree to take this patch, there will certainly be more features that someone might want and not have, such as the ability to sync with multiple standbys at once. More than that, I think we should evaluate this patch on a cost/benefit ratio, rather than trying to apply to it all those procedural fences that we don't have, and that we don't have the size to benefit from. As a community, we've adopted a development plan that proceeds in cycles. For the last several releases, we have had four CommitFests in each release cycle, followed by a feature freeze and eventually by beta and final release. It's certainly a valid question to ask how well that procedure has served us. It does not seem likely to me that we can continue to produce quality releases if we don't at some point cut off the flow of new features into the source tree and work on stabilizing the code we've already got, and I believe the point for that was agreed by a large number of developers who sat in a room at PGCon last year to be on or about February 15th. We ended up extending that by a couple of weeks, to make sure that we had a process that was FAIR: we didn't want patches that had been in the pipeline for a very long time to get postponed to 9.2 because no committer had had a chance to work on them yet. However, we also bumped MANY patches to 9.2 because they weren't in sufficiently good shape soon enough. If we accept this patch now because a bunch of people say they really, really want it, isn't that unfair to the people to whom we've already said sorry, the deadline has passed? Of course, there is always going to be some gray area. I argued for committing the replication_timeout patch because I believe the fact that we haven't got that feature is almost a bug - it interferes significantly with the usability of replication in general, and it will be an even more serious problem with sync rep, where a hung standby connection will not only mean that nothing is replicating but also that no write transactions can be processed at all. However, you could make the opposite argument - that it's really a new feature - and therefore we ought not to commit it. So far no one has taken that position, but it's certainly a reasonable argument. Likewise, there is ongoing discussion on the collations thread about which of those changes are necessary for this release, and which ones are things that ought to be postponed to a future release. I haven't gotten too involved in those discussions because I don't really understand the underlying issues, but I think that's an important discussion. I'm very excited about new options, especially recv. But I agree with Robert and Heikki because what the patch provides looks like new feature rather than bug fix. And I think that we still require some discussions of the design; how far transactions must wait for sync rep in recv mode? In the patch, they wait for WAL to be written in the standby, but I think that they should wait until walreceiver has recieved WAL instead. That would increase the performance of sync rep. Anyway, I don't think now is time to discuss about such a design except for bug fix. I like those additional options, but I believe that sync rep which we worked out is still useful without them. Replication timeout looks like a bug fix rather than new feature. Without that, walsender might unexpectedly remain for a while when the standby crashes or the network outage happens. As Robert said, sync rep can get stuck for a long while because of such a remaining walsender. What's the worse is that when hot_standby_feedback is enabled, such a remaining walsender would prevent oldest xmin from advancing and interfere with vacuuming on the master. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Typos
Hi, The attached patch fixes two typos. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center typos_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xlog_receive_location()
On Wed, Mar 30, 2011 at 11:45 AM, Tatsuo Ishii is...@postgresql.org wrote: BTW, is there any reliable way to know a standby is promoting to primary? You can know that from the server log. But there is no way to do that via SQL, I think. Ok. It seems I need to patch PostgreSQL. Why is that feature required? It's for pgpool-II? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
