Re: [HACKERS] Draft release notes up for review
2013-03-29 02:46 keltezéssel, Tom Lane írta: Since there has been some, um, grumbling about the quality of the release notes of late, I've prepared draft notes for next week's releases, covering commits through today. These are now committed into the master branch for review, and should show up at http://www.postgresql.org/docs/devel/static/ after guaibasaurus' next buildfarm run, about three hours from now. Please comment if you find anything that could be improved. The sgml converter seems to choke on the UTF-8 characters that my name contains: Add configuration variable lock_timeout to limit lock wait duration (Zoltán Böszörményi) I don't mind if my name is written without the accented characters like in the other entry: Have pg_basebackup --write-recovery-conf output a minimal recovery.conf (Zoltan Boszormenyi, Magnus Hagander) Alternatively the sgml tool can be taught to emit proper HTML accented characters or my name can be written as Zoltaacute;n Bouml;szouml;rmeacute;nyi Best regards, Zoltán Böszörményi regards, tom lane -- -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes up for review
2013-04-21 08:28 keltezéssel, Boszormenyi Zoltan írta: 2013-03-29 02:46 keltezéssel, Tom Lane írta: Since there has been some, um, grumbling about the quality of the release notes of late, I've prepared draft notes for next week's releases, covering commits through today. These are now committed into the master branch for review, and should show up at http://www.postgresql.org/docs/devel/static/ after guaibasaurus' next buildfarm run, about three hours from now. Please comment if you find anything that could be improved. The sgml converter seems to choke on the UTF-8 characters that my name contains: Add configuration variable lock_timeout to limit lock wait duration (Zoltán Böszörményi) I don't mind if my name is written without the accented characters like in the other entry: Have pg_basebackup --write-recovery-conf output a minimal recovery.conf (Zoltan Boszormenyi, Magnus Hagander) Alternatively the sgml tool can be taught to emit proper HTML accented characters or my name can be written as Zoltaacute;n Bouml;szouml;rmeacute;nyi doc/src/sgml/release.sgml suggest using the last one but when I looked at release-9.3, I saw (AlvaroAacute;lvaro Herrera) in the webpage several times where the sgml contains (Aacute;lvaro Herrera), so it's not bulletproof either. Best regards, Zoltán Böszörményi regards, tom lane -- -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
2013-04-21 07:02 keltezéssel, Bruce Momjian írta: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current plan. I have completed a draft 9.3 release notes, which you can view here: http://momjian.us/pgsql_docs/release-9-3.html I will be working on polishing them for the next ten days, so any feedback, patches, or commits are welcome. I still need to add lots of SGML markup. How comes Álvaro's name comes out right in your page but not at http://www.postgresql.org/docs/devel/static/release-9-3.html ? Anyway, I attached a patch to fix my name in your page using markups. Thanks, Zoltán Böszörményi -- -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/ diff --git a/doc/src/sgml/release-9.3.sgml b/doc/src/sgml/release-9.3.sgml index 76ba128..2927272 100644 --- a/doc/src/sgml/release-9.3.sgml +++ b/doc/src/sgml/release-9.3.sgml @@ -367,7 +367,7 @@ listitem para Add configuration variable lock_timeout to limit lock wait duration -(Zoltán Böszörményi) +(Zoltaacute;n Bouml;szouml;rmeacute;nyi) /para /listitem @@ -488,7 +488,7 @@ listitem para Have pg_basebackup --write-recovery-conf output a minimal -recovery.conf (Zoltan Boszormenyi, Magnus Hagander) +recovery.conf (Zoltaacute;n Bouml;szouml;rmeacute;nyi, Magnus Hagander) /para para @@ -1357,7 +1357,7 @@ listitem para -Create a centralized timeout API (Zoltán Böszörményi) +Create a centralized timeout API (Zoltaacute;n Bouml;szouml;rmeacute;nyi) /para /listitem -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
On Sun, Apr 21, 2013 at 9:02 AM, Bruce Momjian br...@momjian.us wrote: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current plan. I have completed a draft 9.3 release notes, which you can view here: http://momjian.us/pgsql_docs/release-9-3.html I will be working on polishing them for the next ten days, so any feedback, patches, or commits are welcome. I still need to add lots of SGML markup * Collect and use histograms of lower and upper bounds for range types (Alexander Korotkov) Actually, we also collect histogram of range lengths. Probably, we can rephrase it more generally, like Collect and use special statistics for range types. -- With best regards, Alexander Korotkov.
Re: [HACKERS] 9.3 Beta1 status report
On 2013-04-20 22:36:32 -0700, Peter Geoghegan wrote: On Sat, Apr 20, 2013 at 10:02 PM, Bruce Momjian br...@momjian.us wrote: I will be working on polishing them for the next ten days, so any feedback, patches, or commits are welcome. I still need to add lots of SGML markup. I've noticed a few things: * Allow heap-only tuple updates on system tables (Andres Freund) Didn't Andres just fix a bug wherein HOT updates usually wouldn't occur on system tables following the commit of the foreign key locks patch? Yes, that definitely was a bugfix for a HEAD only feature. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Inconsistent DB data in Streaming Replication
On Wed, Apr 17, 2013 at 10:11 PM, Amit Kapila amit.kap...@huawei.com wrote: On Wednesday, April 17, 2013 4:19 PM Florian Pflug wrote: On Apr17, 2013, at 12:22 , Amit Kapila amit.kap...@huawei.com wrote: Do you mean to say that as an error has occurred, so it would not be able to flush received WAL, which could result in loss of WAL? I think even if error occurs, it will call flush in WalRcvDie(), before terminating WALReceiver. Hm, true, but for that to prevent the problem the inner processing loop needs to always read up to EOF before it exits and we attempt to send a reply. Which I don't think it necessarily does. Assume, that the master sends a chunk of data, waits a bit, and finally sends the shutdown record and exits. The slave might then receive the first chunk, and it might trigger sending a reply. At the time the reply is sent, the master has already sent the shutdown record and closed the connection, and we'll thus fail to reply and abort. Since the shutdown record has never been read from the socket, XLogWalRcvFlush won't flush it, and the slave ends up behind the master. Also, since XLogWalRcvProcessMsg responds to keep-alives messages, we might also error out of the inner processing loop if the server closes the socket after sending a keepalive but before we attempt to respond. Fixing this on the receive side alone seems quite messy and fragile. So instead, I think we should let the master send a shutdown message after it has sent everything it wants to send, and wait for the client to acknowledge it before shutting down the socket. If the client fails to respond, we could log a fat WARNING. Your explanation seems to be okay, but I think before discussing the exact solution, If the actual problem can be reproduced, then it might be better to discuss this solution. I got this problem several times when I enabled WAL archiving and shut down the master. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Inconsistent DB data in Streaming Replication
On Wed, Apr 17, 2013 at 7:49 PM, Florian Pflug f...@phlo.org wrote: On Apr17, 2013, at 12:22 , Amit Kapila amit.kap...@huawei.com wrote: Do you mean to say that as an error has occurred, so it would not be able to flush received WAL, which could result in loss of WAL? I think even if error occurs, it will call flush in WalRcvDie(), before terminating WALReceiver. Hm, true, but for that to prevent the problem the inner processing loop needs to always read up to EOF before it exits and we attempt to send a reply. Which I don't think it necessarily does. Assume, that the master sends a chunk of data, waits a bit, and finally sends the shutdown record and exits. The slave might then receive the first chunk, and it might trigger sending a reply. At the time the reply is sent, the master has already sent the shutdown record and closed the connection, and we'll thus fail to reply and abort. Since the shutdown record has never been read from the socket, XLogWalRcvFlush won't flush it, and the slave ends up behind the master. Also, since XLogWalRcvProcessMsg responds to keep-alives messages, we might also error out of the inner processing loop if the server closes the socket after sending a keepalive but before we attempt to respond. Fixing this on the receive side alone seems quite messy and fragile. So instead, I think we should let the master send a shutdown message after it has sent everything it wants to send, and wait for the client to acknowledge it before shutting down the socket. Agreed. I've tried to fix this problem on only the walreceiver side, but that failed. I agree that we should change walsender so that it waits for the replay from the standby before closing the connection. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery target 'immediate'
On Fri, Apr 19, 2013 at 10:30 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Apr 18, 2013 at 2:11 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. If you don't set a recovery target, PostgreSQL will recover all the WAL it finds. You can set recovery target time to a point immediately after the end-of-backup record, but that's tricky. You have to somehow find out the exact time when the backup ended, and set it to that. But if you set it any too early, recovery will abort with requested recovery stop point is before consistent recovery point error. And that's not quite precise anyway; not all record types carry timestamps, so you will always replay a few extra records until the first timestamped record comes along. Setting recovery_target_xid is similarly difficult. If you were well prepared, you created a named recovery point with pg_create_restore_point() immediately after the backup ended, and you can use that, but that requires forethought. It seems that we're missing a setting, something like recovery_target = 'immediate', which would mean stop as soon as consistency is reached. Or am I missing some trick? You know, I've been wondering for years how you're supposed to do this. Huge +1 for adding something like this, if it doesn't exist already. I also don't know good way to do that. +1 Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
On Sat, Apr 20, 2013 at 10:36:32PM -0700, Peter Geoghegan wrote: * Improve grouping of sessions waiting for commit_delay (Peter Geoghegan) I think this should be under General Performance. It's definitely a performance feature. OK, moved. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
On Sun, Apr 21, 2013 at 09:34:10AM +0200, Boszormenyi Zoltan wrote: 2013-04-21 07:02 keltezéssel, Bruce Momjian írta: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current plan. I have completed a draft 9.3 release notes, which you can view here: http://momjian.us/pgsql_docs/release-9-3.html I will be working on polishing them for the next ten days, so any feedback, patches, or commits are welcome. I still need to add lots of SGML markup. How comes Álvaro's name comes out right in your page but not at http://www.postgresql.org/docs/devel/static/release-9-3.html ? Anyway, I attached a patch to fix my name in your page using markups. Thanks, applied. I had not yet gotten to doing the SGML markup for non-ASCII characters. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
On Sun, Apr 21, 2013 at 02:45:42PM +0400, Alexander Korotkov wrote: On Sun, Apr 21, 2013 at 9:02 AM, Bruce Momjian br...@momjian.us wrote: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current plan. I have completed a draft 9.3 release notes, which you can view here: http://momjian.us/pgsql_docs/release-9-3.html I will be working on polishing them for the next ten days, so any feedback, patches, or commits are welcome. I still need to add lots of SGML markup * Collect and use histograms of lower and upper bounds for range types (Alexander Korotkov) Actually, we also collect histogram of range lengths. Probably, we can rephrase it more generally, like Collect and use special statistics for range types. Done, thanks. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes up for review
On Sun, Apr 21, 2013 at 09:25:39AM +0200, Boszormenyi Zoltan wrote: doc/src/sgml/release.sgml suggest using the last one but when I looked at release-9.3, I saw (AlvaroAacute;lvaro Herrera) in the webpage several times where the sgml contains (Aacute;lvaro Herrera), so it's not bulletproof either. Yes, fixed in commit 8 hours ago. Thanks. '' expanded in my regex. :-( -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes up for review
On Sun, Apr 21, 2013 at 08:28:22AM +0200, Boszormenyi Zoltan wrote: 2013-03-29 02:46 keltezéssel, Tom Lane írta: Since there has been some, um, grumbling about the quality of the release notes of late, I've prepared draft notes for next week's releases, covering commits through today. These are now committed into the master branch for review, and should show up at http://www.postgresql.org/docs/devel/static/ after guaibasaurus' next buildfarm run, about three hours from now. Please comment if you find anything that could be improved. The sgml converter seems to choke on the UTF-8 characters that my name contains: Add configuration variable lock_timeout to limit lock wait duration (Zoltán Böszörményi) I don't mind if my name is written without the accented characters like in the other entry: Have pg_basebackup --write-recovery-conf output a minimal recovery.conf (Zoltan Boszormenyi, Magnus Hagander) Alternatively the sgml tool can be taught to emit proper HTML accented characters or my name can be written as Zoltaacute;n Bouml;szouml;rmeacute;nyi I have applied your patch to use entities. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery target 'immediate'
On Fri, Apr 19, 2013 at 8:30 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Apr 18, 2013 at 2:11 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems that we're missing a setting, something like recovery_target = 'immediate', which would mean stop as soon as consistency is reached. Or am I missing some trick? You know, I've been wondering for years how you're supposed to do this. Huge +1 for adding something like this, if it doesn't exist already. Hi, you can use pause_at_recovery_target parameter in recovery.conf and try one recovery_target at a time... or of course create a pause_at_recovery_consistency (name could be different) for that -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación Phone: +593 4 5107566 Cell: +593 987171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checksum failures generate warnings
On Sat, Apr 20, 2013 at 5:54 AM, Bruce Momjian br...@momjian.us wrote: Do we really want to generate just a warning for a checksum failure, and not an error; see PageIsVerified(). Unless you turn on the parameter ignore_checksum_failure, you will get an error. It will be generated by the caller, not by PageIsVerified. Cheers, Jeff
Re: [HACKERS] 9.3 Beta1 status report
Bruce, I don't see parallel pg_dump in the release notes. I thought that got committed? Anyway, see the pgsql-advocacy list for a longish discussion about what we should consider the major fetures for 9.3. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 Beta1 status report
On 2013-04-21 14:50:07 -0700, Josh Berkus wrote: Bruce, I don't see parallel pg_dump in the release notes. I thought that got committed? E.1.3.8.2. pg_dump: Add pg_dump --jobs to dump in parallel when using directory output format (Joachim Wieland) Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery target 'immediate'
On Thu, Apr 18, 2013 at 10:11 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. You can find first WAL file name in backup_label START WAL LOCATION. Last WAL file name location depends on source type, if backup from slave - use pg_control from backup and Minimum recovery ending location, if backup from master - use STOP WAL LOCATION from backup .history file :-) Then I just copy needed WALs from archive into pg_xlog and remove recovery.conf. It seems that we're missing a setting, something like recovery_target = 'immediate', which would mean stop as soon as consistency is reached. Or am I missing some trick? This will be helpful :) -- Sergey Burladyan
Re: [HACKERS] Recovery target 'immediate'
On Fri, Apr 19, 2013 at 3:11 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. If you don't set a recovery target, PostgreSQL will recover all the WAL it finds. You can set recovery target time to a point immediately after the end-of-backup record, but that's tricky. You have to somehow find out the exact time when the backup ended, and set it to that. But if you set it any too early, recovery will abort with requested recovery stop point is before consistent recovery point error. And that's not quite precise anyway; not all record types carry timestamps, so you will always replay a few extra records until the first timestamped record comes along. Setting recovery_target_xid is similarly difficult. If you were well prepared, you created a named recovery point with pg_create_restore_point() immediately after the backup ended, and you can use that, but that requires forethought. It seems that we're missing a setting, something like recovery_target = 'immediate', which would mean stop as soon as consistency is reached. Or am I missing some trick? +1. This will be really helpful. I don't know either of any good way to stop immediately after a consistent point now without tricking a target just after the end of backup. -- Michael
[HACKERS] minimizing the target list for foreign data wrappers
A few years ago I wrote a roll-your-own foreign-data-wrapper system for Postgres because Postgres didn't have one at the time (some details herehttp://unubtainabol.blogspot.com/2013/04/daves-foreign-data-introuction.htmlif anyone is interested). Now I'm being tasked to move it to Postgres 9.2.x and I'd like to use FDW if possible. One of the problems I'm having is that in my application, the foreign tables typically have hundreds of columns while typical queries only access a dozen or so (the foreign server is a columnar SQL database). Furthermore, there is no size optimization for NULL values passed back from the foreign server, so if I return all of the columns from the table --even as NULLs-- the returned data size will be several times the size that it needs to be. My application cannot tolerate this level of inefficiency, so I need to return minimal columns from the foreign table. The documentation doesn't say how to do this, but looking at the code I think it is possible. In GetForeignPlan() you have to pass on the tlist argument, which I presume means that the query plan will use the tlist that I pass in, right? If so, then it should be possible for me to write a function that takes tlist and baserel-reltargetlist and return a version of tlist that knows which foreign-table columns are actually used, and replaces the rest with a NULL constant. For example, suppose the original tlist is this: [VAR(attrno=1), VAR(attrno=2), VAR(attrno=3)] and reltarget list says that I only need args 1 and 3. Then the new tlist would look like this: [VAR(attrno=1), CONST(val=NULL), VAR(attrno=2)] where the attrno of the last VAR has been reduced by one because the 2 column is no longer there. I did something very much like this in my roll-your-own version of FDW so I know basically how to do it, but I did it at the pre-planning stage and I'm not sure how much is already packed into the other plan nodes at this point. Maybe it's too late to change the target list? Can anyone give me some advice or warnings on this? I'd hate to go to the trouble of implementing and testing it only to find that I'm making some bogus assumptions. Thanks, David Gudeman
