Re: [HACKERS] warning message in standby
On 12/06/10 04:19, Bruce Momjian wrote: Robert Haas wrote: If my streaming replication stops working, I want to know about it as soon as possible. WARNING just doesn't cut it. This needs some better thought. If we PANIC, then surely it will PANIC again when we restart unless we do something. So we can't do that. But we need to do something better than WARNING there is a bug that will likely cause major data loss HINT you'll be sacked if you miss this message +1. I was making this same argument (less eloquently) upthread. I particularly like the errhint(). I am wondering what action would be most likely to get the administrator's attention. I've committed the patch to disconnect the SR connection in that case. If the message needs improvement, let's do that separately once we figure out what to do. Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.0beta2 - server crash when using HS + SR
On Mon, Jun 14, 2010 at 9:16 AM, Greg Smith g...@2ndquadrant.com wrote: I wouldn't be adverse to improving the error messages emitted when this happens by the server to make it more obvious what's gone wrong in 9.1. That's the only genuine improvement I'd see value in here, to cut down on other people running into what you did and being as confused by it. What about the attached patch? When we encounter that problem, we get the following hint message: FATAL: directory /path_to/ts does not exist HINT: create /path_to/ts directory for tablespace before restarting the server CONTEXT: xlog redo create ts: 16384 /path_to/ts Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center recovery_ts_hint_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Typo in plperl doc ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hey, While translating the plperl page from the manual, I found the following sentence: The varname%_SHARED/varname variable and other global state within the language is public data. Should it be : The varname%_SHARED/varname variable and other global variables within the language are public data. ? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwV3tYACgkQxWGfaAgowiJPFQCfaROauKGDLu0ODZXCbshb9dVe AfYAnAx+PvopgycLm7KPHdd6l8202vOm =Ky/1 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Fri, Jun 11, 2010 at 11:24 PM, Robert Haas robertmh...@gmail.com wrote: I think the failover case might be OK. But if the master crashes and restarts, the slave might be left thinking its xlog position is ahead of the xlog position on the master. Right. Unless we perform a failover in this case, the standby might go down because of inconsistency of WAL after restarting the master. To avoid this problem, walsender must wait for WAL to be not only written but also *fsynced* on the master before sending it as 9.0 does. Though this would degrade the performance, this might be useful for some cases. We should provide the knob to specify whether to allow the standby to go ahead of the master or not? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Fri, Jun 11, 2010 at 11:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: hmm not sure that is what fujii tried to say - I think his point was that in the original case we would have serialized all the operations (first write+sync on the master, network afterwards and write+sync on the slave) and now we could try parallelizing by sending the wal before we have synced locally. Well, we're already not waiting for fsync, which is the slowest part. No, currently walsender waits for fsync. Walsender tries to send WAL up to xlogctl-LogwrtResult.Write. OTOH, xlogctl-LogwrtResult.Write is updated after XLogWrite() performs fsync. As the result, walsender cannot send WAL not fsynced yet. We should update xlogctl-LogwrtResult.Write before XLogWrite() performs fsync for 9.0? But that change would cause the problem that Robert pointed out. http://archives.postgresql.org/pgsql-hackers/2010-06/msg00670.php If there's a performance problem, it may be because FADVISE_DONTNEED disables kernel buffering so that we're forced to actually read the data back from disk before sending it on down the wire. Currently, if max_wal_senders 0, POSIX_FADV_DONTNEED is not used for WAL files at all. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Sat, Jun 12, 2010 at 12:15 AM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: hmm ok - but assuming sync rep we would end up with something like the following(hypotetically assuming each operation takes 1 time unit): originally: write 1 sync 1 network 1 write 1 sync 1 total: 5 whereas in the new case we would basically have the write+sync compete with network+write+sync in parallel(total 3 units) and we would only have to wait for the slower of those two sets of operations instead of the total time of both or am I missing something. Yeah, this is what I'd like to say. Thanks! Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ExecutorCheckPerms() hook
I attached three patches for the effort. Each patch tries to tackle one theme, so it is not unreasonable. But the ESP security hook patch (quite tiny) depends on the DML permission refactoring patch (relatively larger). So, Robert suggested me to reconsider the dependency of these patches. The attached patch shall be applied on the head of the git repository. It just adds a security hook on ExecCheckRTPerms() as Robert suggested at first. Of course, it does not allow to acquire the control on COPY TO/FROM and RI_Initial_Check(). It will be refactored in the following patches. Thanks, (2010/05/27 12:00), KaiGai Kohei wrote: Stephen, The 'failure' may make an impression of generic errors not only permission denied. How about 'error_on_violation'? Maybe 'ereport_on_violation'? I dunno, guess one isn't really better than the other. You need to go back and fix the comment though- you still say 'abort' there. I have no preference between 'error_on_violation' and 'ereport_on_violation'. OK, I fixed it. BTW, I wonder whether acl.h is a correct place to explain about the hook, although I added comments for the hook. Guess I don't really see a problem putting the comments there. By the way, have we got a place where we actually document the hooks we support somewhere in the official documentation..? If so, that should certainly be updated too.. I could not find Executor hooks from doc/src/sgml using grep. If so, it might be worth to list them on the wikipage. I think we should add a series of explanation about ESP hooks in the internal section of the documentation, when the number of hooks reaches a dozen for example. I believe the goal will be to avoid reaching a dozen hooks for this. Maybe, all we need to hook on DML permissions is only this one. All-in-all, I'm pretty happy with these. Couple minor places which could use some copy editing, but that's about it. Next, we need to get the security label catalog and the grammar to support it implemented and then from that an SELinux module should be pretty easy to implement. Based on the discussions at PGCon, Robert is working on the security label catalog and grammar. The current plan is to have a catalog similar to pg_depend, to minimize impact to the rest of the backend and to those who aren't interested in using security labels. Pg_depend? not pg_description/pg_shdescription? I basically agree with the idea that minimizes damages to the existing schema of system catalogs, but I cannot imagine something like pg_depend well. I'd like to post a new thread to discuss the security label support. OK? Of course, there will also need to be hooks there for an external module to enforce restrictions associated with changing labels on various objects in the system. Yes, the user given has to be validated by ESP. Thanks, -- KaiGai Kohei kai...@ak.jp.nec.com pgsql-v9.1-add-dml-hook.1.patch Description: application/octect-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] [v9.1] add makeRangeTblEntry() into makefuncs.c
The attached patch was a part of DML refactoring and security hook patches. It adds makeRangeTblEntry() into makefuncs.c to keep the code more clean. It shall be also used for the upcoming DML refactor patch. In this refactoring, a common DML permission checker function take a list of RangeTblEntry, so the caller has to set up the object. Thanks, -- KaiGai Kohei kai...@ak.jp.nec.com pgsql-v9.1-add-makeRTE.1.patch Description: application/octect-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Reworks of DML permission checks
The attached patch tries to rework DML permission checks. It was mainly checked at the ExecCheckRTEPerms(), but same logic was implemented in COPY TO/FROM statement and RI_Initial_Check(). This patch tries to consolidate these permission checks into a common function to make access control decision on DML permissions. It enables to eliminate the code duplication, and improve consistency of access controls. Thanks, -- KaiGai Kohei kai...@ak.jp.nec.com pgsql-v9.1-reworks-dml-checks.1.patch Description: application/octect-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC - Materialized Views - is stale or fresh?
2010/6/14 Greg Smith g...@2ndquadrant.com: Pavel Baros wrote: After each INSERT, UPDATE, DELETE statement (transaction) pg_class.rellastxid would be updated. That should not be time- or memory- consuming (not so much) since pg_class is cached, I guess. An update in PostgreSQL is essentially an INSERT followed a later DELETE when VACUUM gets to the dead row no longer visible. The problem with this approach is that it will leave behind so many dead rows in pg_class due to the heavy updates that the whole database could grind to a halt, as so many operations will have to sort through all that garbage. It could potentially double the total write volume on the system, and you'll completely kill people who don't have autovacuum running during some periods of the day. The basic idea of saving the last update time for each relation is not unreasonable, but you can't store the results by updating pg_class. My first thought would be to send this information as a message to the statistics collector. It's already being sent updates at the point you're interested in for the counters of how many INSERT/UPDATE/DELETE statements are executing against the table. You might bundle your last update information into that existing message with minimal overhead. Right. Do remember that the stats collector is designed to be lossy, though, so you're not guaranteed that the information reaches the other end. In reality it tends to do that, but there needs to be some sort of recovery path for the case when it doesn't. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] [v9.1] Add security hook on initialization of instance
The attached patch tries to add one more security hook on the initialization of PostgreSQL instance (InitPostgres()). It gives the external security module a chance to initialize itself, and acquire credential of the client. I assumed the best place to initialize the module is just after the initialize_acl() invocation, if ESP is available. We have not discussed about this hook yet. So, I'd like to see any comments. Thanks, -- KaiGai Kohei kai...@ak.jp.nec.com pgsql-v9.1-add-init-hook.1.patch Description: application/octect-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Heikki Linnakangas wrote: On 12/06/10 04:19, Bruce Momjian wrote: Robert Haas wrote: If my streaming replication stops working, I want to know about it as soon as possible. WARNING just doesn't cut it. This needs some better thought. If we PANIC, then surely it will PANIC again when we restart unless we do something. So we can't do that. But we need to do something better than WARNING there is a bug that will likely cause major data loss HINT you'll be sacked if you miss this message +1. I was making this same argument (less eloquently) upthread. I particularly like the errhint(). I am wondering what action would be most likely to get the administrator's attention. I've committed the patch to disconnect the SR connection in that case. If the message needs improvement, let's do that separately once we figure out what to do. Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. I am afraid they are never going to pick up something tagged WARNING, no matter what the wording is. Crazy idea, but can we force a fatal error line into the logs with something like WARNING ...\nFATAL: -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 12:16, Bruce Momjian br...@momjian.us wrote: Heikki Linnakangas wrote: On 12/06/10 04:19, Bruce Momjian wrote: Robert Haas wrote: If my streaming replication stops working, I want to know about it as soon as possible. WARNING just doesn't cut it. This needs some better thought. If we PANIC, then surely it will PANIC again when we restart unless we do something. So we can't do that. But we need to do something better than WARNING there is a bug that will likely cause major data loss HINT you'll be sacked if you miss this message +1. I was making this same argument (less eloquently) upthread. I particularly like the errhint(). I am wondering what action would be most likely to get the administrator's attention. I've committed the patch to disconnect the SR connection in that case. If the message needs improvement, let's do that separately once we figure out what to do. Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. I am afraid they are never going to pick up something tagged WARNING, no matter If they are properly configured, I imagine they would. And if they're not, well, there's not much for us to do. (What would be more usful then would be to separate user-warnings like warnings about cast from actual system-warnings like this, but that's a whole different story) what the wording is. Crazy idea, but can we force a fatal error line into the logs with something like WARNING ...\nFATAL: That's way too crazy :P And btw, randomly sticking newlines into that will mess up *most* log displayers and I bet a lot of the log monitoring tools as well... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Command to prune archive at restartpoints
Fujii Masao masao.fu...@gmail.com writes: In SR, WAL files in the pg_xlog directory on the standby are recycled by every restartpoints. So your proposed function seems not to be helpful even if hot_standby = on. Then I guess I'm at a loss here: what is the pg_archivecleanup utility good for in a standby? -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_archive_bypass
Hi,
I tend to consider it a bug that there's no known way under windows to
use the same trick as under Unix by using '/usr/bin/true' as your
archive command. And this Unix trick itself does feel like a hack.
Also I'd very much like to be able to recommend (even if not change the
official defaults) to setup wal_level to archive, archive_mode=on and
archive_command=pg_archive_bypass, so that the day you have a HA budget
ain't the day you're going to restart the server to enable the fault
tolerance settings…
So please find attached a very simple let's see about it patch to
implement an internal archive_command that just returns true and is
called pg_archive_bypass. It's missing documentation, which I'll provide
if needed (meaning there's some will to consider applying such a patch).
Regards,
--
dim
*** a/src/backend/postmaster/pgarch.c
--- b/src/backend/postmaster/pgarch.c
***
*** 475,481 pgarch_ArchiverCopyLoop(void)
/*
* pgarch_archiveXlog
*
! * Invokes system(3) to copy one archive file to wherever it should go
*
* Returns true if successful
*/
--- 475,485
/*
* pgarch_archiveXlog
*
! * Invokes system(3) to copy one archive file to wherever it should go, or
! * just return true if set to the internal command pg_archive_bypass. That
! * allows windows users to easily have the /usr/bin/true, and that allows
! * for setting up a archiving only when you need it by changing the command
! * and issuing a reload.
*
* Returns true if successful
*/
***
*** 490,495 pgarch_archiveXlog(char *xlog)
--- 494,509
const char *sp;
int rc;
+ /*
+ * If the command is pg_archive_bypass, just return true
+ */
+ if( strcmp(XLogArchiveCommand, pg_archive_bypass) == 0 )
+ {
+ ereport(DEBUG3,
+ (errmsg_internal(bypassing archive command)));
+ return true;
+ }
+
snprintf(pathname, MAXPGPATH, XLOGDIR /%s, xlog);
/*
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump(all) --quote-all-identifiers
Robert Haas wrote: In response to a complaint from Hartmut Goebel: http://archives.postgresql.org/pgsql-bugs/2010-06/msg00018.php And per a design proposed by Tom Lane: http://archives.postgresql.org/pgsql-bugs/2010-06/msg00211.php PFA a patch to implement $SUBJECT. One interesting aspect of this patch is that types like integer and double precision don't get quoted in the output, whereas types like text do. But it turns out that types like integer and double precision don't *work* if they're quoted, so this is not a bad thing. It might possibly be judged to require documentation somewhere, however. Uh, I thought this was about quoting the identifiers. I am confused about why integer is an issue in this case. Can you show an example? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] Add security hook on initialization of instance
* KaiGai Kohei (kai...@ak.jp.nec.com) wrote: The attached patch tries to add one more security hook on the initialization of PostgreSQL instance (InitPostgres()). It gives the external security module a chance to initialize itself, and acquire credential of the client. I assumed the best place to initialize the module is just after the initialize_acl() invocation, if ESP is available. We have not discussed about this hook yet. So, I'd like to see any comments. Aren't modules given a __PG_Init or something similar that they can define which will be called when the module is loaded..? Stephen signature.asc Description: Digital signature
Re: [HACKERS] SR slaves and .pgpass
On Mon, Jun 14, 2010 at 04:56, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jun 11, 2010 at 7:14 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 09/06/10 08:24, Fujii Masao wrote: On Wed, Jun 9, 2010 at 12:52 PM, Andrew Dunstanand...@dunslane.net wrote: There is precedent for .pgpass being a bit ambiguous. See the way localhost is used. OK. The attached patch allows us to use replication in the database field of the .pgpass file, for the replication connection. Thanks, committed with some rewording of the docs and comments. I hope I made them better, not worse. I could confirm that this has been committed via git log, but not find the related post on the pgsql-committers. Is there a problem in the mailing-list? The cvs server has a delivery of the email to the mailinglist server at least, so it's not there that has the problem. However, there were a lot of unexplained issues across multiple hub.org hosted machines that day (and nearby, iirc), including the mailserver and the dns servers. It was probably eaten by one of those issues. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Mon, Jun 14, 2010 at 4:14 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jun 11, 2010 at 11:24 PM, Robert Haas robertmh...@gmail.com wrote: I think the failover case might be OK. But if the master crashes and restarts, the slave might be left thinking its xlog position is ahead of the xlog position on the master. Right. Unless we perform a failover in this case, the standby might go down because of inconsistency of WAL after restarting the master. To avoid this problem, walsender must wait for WAL to be not only written but also *fsynced* on the master before sending it as 9.0 does. Though this would degrade the performance, this might be useful for some cases. We should provide the knob to specify whether to allow the standby to go ahead of the master or not? Maybe. That sounds like a pretty enormous foot-gun to me, considering that we have no way of recovering from the situation where the standby gets ahead of the master. Right now, I believe we're still in the situation where the standby goes into an infinite CPU-chewing, log-spewing loop, but even after we fix that it's not going to be good enough to really handle that case sensibly, which we probably need to do if we want to make this change. Come to think of it, can this happen already? Can the master stream WAL to the standby after it's written but before it's fsync'd? We should get the open item fixed for 9.0 here before we start worrying about 9.1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Magnus Hagander wrote: Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: ?invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. ?I am afraid they are never going to pick up something tagged WARNING, no matter If they are properly configured, I imagine they would. And if they're not, well, there's not much for us to do. What does that mean? (What would be more usful then would be to separate user-warnings like warnings about cast from actual system-warnings like this, but that's a whole different story) what the wording is. ?Crazy idea, but can we force a fatal error line into the logs with something like WARNING ...\nFATAL: That's way too crazy :P And btw, randomly sticking newlines into that will mess up *most* log displayers and I bet a lot of the log monitoring tools as well... Yeah, it would mess up CSV log output, for example? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: Command Prompt 8.4.4 PRMs compiled with debug/assert enabled
Bruce Momjian wrote: Bruce Momjian wrote: Bruce Momjian wrote: Alvaro Herrera wrote: Excerpts from Tom Lane's message of jue jun 10 11:46:25 -0400 2010: Yes, the folks at commandprompt need to be told about this. Loudly. It's a serious packaging error. Just notified Lacey, the packager (not so loudly, though); she's working on new packages, and apologizes for the inconvenience. [ Thread moved to hackers. 8.4.4 RPMs were built with debug flags. ] Uh, where are we on this? Has it been completed? How are people informed about this? Do we need to post to the announce email list? Does Yum just update them? How did this mistake happen? How many days did it take to detect the problem? Why has no news been posted here? https://public.commandprompt.com/projects/pgcore/news Why have I received no reply to this email? Do people think this is not a serious issue? I know it is a weekend but the problem was identified on Thursday, meaning there was a full workday for someone from CommandPrompt to reply to the issue and report a status: http://archives.postgresql.org/pgsql-performance/2010-06/msg00165.php [ Updated subject line.] I am on IM with Joshua Drake right now and am working to get answers to the questions above. He or I will report in the next few hours. FYI, only Command Prompt-produced RPMs are affected. Devrim's RPMs are not: http://yum.postgresqlrpms.org/ I have still seen no public report about this, 12 hours after talking to Josh Drake on IM about it. :-( -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump(all) --quote-all-identifiers
On Mon, Jun 14, 2010 at 6:57 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: In response to a complaint from Hartmut Goebel: http://archives.postgresql.org/pgsql-bugs/2010-06/msg00018.php And per a design proposed by Tom Lane: http://archives.postgresql.org/pgsql-bugs/2010-06/msg00211.php PFA a patch to implement $SUBJECT. One interesting aspect of this patch is that types like integer and double precision don't get quoted in the output, whereas types like text do. But it turns out that types like integer and double precision don't *work* if they're quoted, so this is not a bad thing. It might possibly be judged to require documentation somewhere, however. Uh, I thought this was about quoting the identifiers. I am confused about why integer is an issue in this case. Can you show an example? Sure. rhaas=# create table bruce (demonstration integer); ERROR: type integer does not exist LINE 1: create table bruce (demonstration integer); ^ rhaas=# create table bruce (demonstration integer); CREATE TABLE See gram.y, around line 8341. Note that if you try the same example with text instead of integer, both variants work. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] Add security hook on initialization of instance
(2010/06/14 20:01), Stephen Frost wrote: * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: The attached patch tries to add one more security hook on the initialization of PostgreSQL instance (InitPostgres()). It gives the external security module a chance to initialize itself, and acquire credential of the client. I assumed the best place to initialize the module is just after the initialize_acl() invocation, if ESP is available. We have not discussed about this hook yet. So, I'd like to see any comments. Aren't modules given a __PG_Init or something similar that they can define which will be called when the module is loaded..? I assume the security module shall be loaded within 'shared_preload_libraries', because we can overwrite 'local_preload_libraries' (PGC_BACKEND) setting using connection string, so it allows users to bypass security features, doesn't it? Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 13:11, Bruce Momjian br...@momjian.us wrote: Magnus Hagander wrote: Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: ?invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. ?I am afraid they are never going to pick up something tagged WARNING, no matter If they are properly configured, I imagine they would. And if they're not, well, there's not much for us to do. What does that mean? It means that we can't prevent people from configuring their tools to ignore important warning. We can't prevent them rom ignoring ERROR or FATAL either... (What would be more usful then would be to separate user-warnings like warnings about cast from actual system-warnings like this, but that's a whole different story) what the wording is. ?Crazy idea, but can we force a fatal error line into the logs with something like WARNING ...\nFATAL: That's way too crazy :P And btw, randomly sticking newlines into that will mess up *most* log displayers and I bet a lot of the log monitoring tools as well... Yeah, it would mess up CSV log output, for example? Possibly. And your general syslog output would become pretty hard to read once this kind of stuff shows up. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_archive_bypass
On Mon, 2010-06-14 at 12:39 +0200, Dimitri Fontaine wrote: I tend to consider it a bug that there's no known way under windows to use the same trick as under Unix by using '/usr/bin/true' as your archive command. And this Unix trick itself does feel like a hack. Also I'd very much like to be able to recommend (even if not change the official defaults) to setup wal_level to archive, archive_mode=on and archive_command=pg_archive_bypass, so that the day you have a HA budget ain't the day you're going to restart the server to enable the fault tolerance settings… So please find attached a very simple let's see about it patch to implement an internal archive_command that just returns true and is called pg_archive_bypass. It's missing documentation, which I'll provide if needed (meaning there's some will to consider applying such a patch). ISTM like a good idea to have a couple of more obvious commands provided purely as internal commands. If we do this on restore_command as well, we can skip pg_archivecleanup completely, for example. So, not for 9.0. But I like concept for 9.1. Would need to be coded as some kind of escape phrase, followed by other command. So not just a quick hack with this one exception. Something like archive_command = 'pg_archive_internal:true' Wouldn't rush to another patch though, needs agreement first. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Command to prune archive at restartpoints
On Mon, 2010-06-14 at 12:21 +0200, Dimitri Fontaine wrote: Fujii Masao masao.fu...@gmail.com writes: In SR, WAL files in the pg_xlog directory on the standby are recycled by every restartpoints. So your proposed function seems not to be helpful even if hot_standby = on. Then I guess I'm at a loss here: what is the pg_archivecleanup utility good for in a standby? Cleaning the archive directory, not the pg_xlog directory. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.0beta2 - server crash when using HS + SR
On Sat, 2010-06-12 at 03:29 +0200, Rafael Martinez wrote: What I didn't expect was such a serious consequence. Postgres crashed in the standby node and it refused to start until the directory needed by the tablespace was created also in the standby. I suppose there is not an easy way of fixing this, but at least it would be a good idea to update the documentation with some information about how to fix this error situation (hot-standby.html#HOT-STANDBY-CAVEATS will be a nice place to have this information) Thanks for testing. This crash has nothing to do with HS or SR. There is already a documented caveat and this has been like this for 6 years http://developer.postgresql.org/pgdocs/postgres/continuous-archiving.html#CONTINUOUS-ARCHIVING-CAVEATS If you said the docs need some work, I would agree with you. There isn't anything to say that the caveats in 24.3.6 still apply to SR, though they still all do. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Magnus Hagander wrote: On Mon, Jun 14, 2010 at 13:11, Bruce Momjian br...@momjian.us wrote: Magnus Hagander wrote: Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: ?invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. ?I am afraid they are never going to pick up something tagged WARNING, no matter If they are properly configured, I imagine they would. And if they're not, well, there's not much for us to do. What does that mean? It means that we can't prevent people from configuring their tools to ignore important warning. We can't prevent them rom ignoring ERROR or FATAL either... My point is that most tools are going to look at the tag first to determine the severity of the message, and might even have log_min_messages set to ignore warnings. Esentially this message would have to be special-cased for log filtering, and considering translation issues, etc, it seems pretty odd. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 7:18 AM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jun 14, 2010 at 13:11, Bruce Momjian br...@momjian.us wrote: Magnus Hagander wrote: Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: ?invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. ?I am afraid they are never going to pick up something tagged WARNING, no matter If they are properly configured, I imagine they would. And if they're not, well, there's not much for us to do. What does that mean? It means that we can't prevent people from configuring their tools to ignore important warning. We can't prevent them rom ignoring ERROR or FATAL either... Right. Certainly, ERROR would be better than WARNING, though, because someone, somewhere out there has a log-fitering tool that extracts ERRORs but ignore WARNINGs. What still bugs me about this situation is that we're essentially trying futilely to recover from what's really a fatal error. There is no manner of proceeding that has any hope of success, yet we just keep hopelessly retrying. Why do we do that here and not elsewhere? By the logic we're using here, we ought to retry when we hit a division by zero error. Maybe the next time we read the second input value it will have some bits set... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_archive_bypass
On 14/06/10 13:39, Dimitri Fontaine wrote: I tend to consider it a bug that there's no known way under windows to use the same trick as under Unix by using '/usr/bin/true' as your archive command. And this Unix trick itself does feel like a hack. Also I'd very much like to be able to recommend (even if not change the official defaults) to setup wal_level to archive, archive_mode=on and archive_command=pg_archive_bypass, so that the day you have a HA budget ain't the day you're going to restart the server to enable the fault tolerance settings… That particular use case is better handled by making archive_mode changeable on-the-fly. Now that we have wal_level as a separate GUC, it shouldn't be too hard. I like internal commands as a solution in general, but it's still in search of a problem.. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC - Materialized Views - is stale or fresh?
On Mon, Jun 14, 2010 at 5:00 AM, Magnus Hagander mag...@hagander.net wrote: 2010/6/14 Greg Smith g...@2ndquadrant.com: Pavel Baros wrote: After each INSERT, UPDATE, DELETE statement (transaction) pg_class.rellastxid would be updated. That should not be time- or memory- consuming (not so much) since pg_class is cached, I guess. An update in PostgreSQL is essentially an INSERT followed a later DELETE when VACUUM gets to the dead row no longer visible. The problem with this approach is that it will leave behind so many dead rows in pg_class due to the heavy updates that the whole database could grind to a halt, as so many operations will have to sort through all that garbage. It could potentially double the total write volume on the system, and you'll completely kill people who don't have autovacuum running during some periods of the day. The basic idea of saving the last update time for each relation is not unreasonable, but you can't store the results by updating pg_class. My first thought would be to send this information as a message to the statistics collector. It's already being sent updates at the point you're interested in for the counters of how many INSERT/UPDATE/DELETE statements are executing against the table. You might bundle your last update information into that existing message with minimal overhead. Right. Do remember that the stats collector is designed to be lossy, though, so you're not guaranteed that the information reaches the other end. In reality it tends to do that, but there needs to be some sort of recovery path for the case when it doesn't. What Pavel's trying to do here is be smart about figuring out when an MV needs to be refreshed. I'm pretty sure this is the wrong way to go about it, but it seems entirely premature considering that we don't have a working implementation of a *manually* refreshed MV. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On 14/06/10 13:16, Bruce Momjian wrote: Heikki Linnakangas wrote: On 12/06/10 04:19, Bruce Momjian wrote: Robert Haas wrote: If my streaming replication stops working, I want to know about it as soon as possible. WARNING just doesn't cut it. This needs some better thought. If we PANIC, then surely it will PANIC again when we restart unless we do something. So we can't do that. But we need to do something better than WARNING there is a bug that will likely cause major data loss HINT you'll be sacked if you miss this message +1. I was making this same argument (less eloquently) upthread. I particularly like the errhint(). I am wondering what action would be most likely to get the administrator's attention. I've committed the patch to disconnect the SR connection in that case. If the message needs improvement, let's do that separately once we figure out what to do. Seems like we need something like WARNING that doesn't cause the process to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a hint to the warning will do. How about WARNING: invalid record length at 0/4005330 HINT: An invalid record was streamed from master. That can be a sign of corruption in the master, or inconsistency between master and standby state. The record will be re-fetched, but that is unlikely to fix the problem. You may have to restore standby from base backup. I am thinking about log monitoring tools like Nagios. I am afraid they are never going to pick up something tagged WARNING, no matter what the wording is. One idea is for the startup process to signal walreceiver process to commit suicide with FATAL, instead of just dying silently like it does now. So you'd get a WARNING explaining how the record was corrupt, followed by a FATAL from the walreceiver process: WARNING: invalid record length at 0/4005330 FATAL: walreceiver killed because of error in WAL stream Crazy idea, but can we force a fatal error line into the logs with something like WARNING ...\nFATAL: Yeah, that's crazy. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ExecutorCheckPerms() hook
2010/6/14 KaiGai Kohei kai...@ak.jp.nec.com: I attached three patches for the effort. Each patch tries to tackle one theme, so it is not unreasonable. But the ESP security hook patch (quite tiny) depends on the DML permission refactoring patch (relatively larger). So, Robert suggested me to reconsider the dependency of these patches. The attached patch shall be applied on the head of the git repository. It just adds a security hook on ExecCheckRTPerms() as Robert suggested at first. Of course, it does not allow to acquire the control on COPY TO/FROM and RI_Initial_Check(). It will be refactored in the following patches. This is essentially the same patch that I wrote and posted several weeks ago, with changes to the comments and renaming of the identifiers. Are you trying to represent it as your own work? With all due respect, I intend to imply my own version. Please make your other proposed patches apply on top of that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Mon, 2010-06-14 at 17:39 +0900, Fujii Masao wrote: On Fri, Jun 11, 2010 at 11:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: hmm not sure that is what fujii tried to say - I think his point was that in the original case we would have serialized all the operations (first write+sync on the master, network afterwards and write+sync on the slave) and now we could try parallelizing by sending the wal before we have synced locally. Well, we're already not waiting for fsync, which is the slowest part. No, currently walsender waits for fsync. Walsender tries to send WAL up to xlogctl-LogwrtResult.Write. OTOH, xlogctl-LogwrtResult.Write is updated after XLogWrite() performs fsync. As the result, walsender cannot send WAL not fsynced yet. We should update xlogctl-LogwrtResult.Write before XLogWrite() performs fsync for 9.0? But that change would cause the problem that Robert pointed out. http://archives.postgresql.org/pgsql-hackers/2010-06/msg00670.php ISTM you just defined some clear objectives for next work. Copying the data from WAL buffers is mostly irrelevant. The majority of time is lost waiting for fsync. The biggest issue is about how to allow WAL write and WALSender to act concurrently and have backend wait for both. Sure, copying data from wal_buffers will be faster still, but it will cause you to address some subtle data structure locking operations that we could solve at a later time. And it still gives the problem of how the master resets itself if the standby really is ahead. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Mon, 2010-06-14 at 17:39 +0900, Fujii Masao wrote: No, currently walsender waits for fsync. ... But that change would cause the problem that Robert pointed out. http://archives.postgresql.org/pgsql-hackers/2010-06/msg00670.php Presumably this means that if synchronous_commit = off on primary that SR in 9.0 will no longer work correctly if the primary crashes? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] add makeRangeTblEntry() into makefuncs.c
2010/6/14 KaiGai Kohei kai...@ak.jp.nec.com: The attached patch was a part of DML refactoring and security hook patches. It adds makeRangeTblEntry() into makefuncs.c to keep the code more clean. It shall be also used for the upcoming DML refactor patch. In this refactoring, a common DML permission checker function take a list of RangeTblEntry, so the caller has to set up the object. I think this is the epitome of pointless. It looks to me like this just makes the code harder to read and very slightly slower without actually accomplishing any useful abstraction. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
On Mon, Jun 14, 2010 at 3:48 AM, Jehan-Guillaume (ioguix) de Rorthais iog...@free.fr wrote: While translating the plperl page from the manual, I found the following sentence: The varname%_SHARED/varname variable and other global state within the language is public data. Should it be : The varname%_SHARED/varname variable and other global variables within the language are public data. ? State is a more general term, which would include any information that is globally accessible from within the interpreter, not just what's stored in variables. Off the top of my head, I'm not sure if there is anything like that, but I wouldn't bet on there not being any... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] Add security hook on initialization of instance
2010/6/14 KaiGai Kohei kai...@kaigai.gr.jp: (2010/06/14 20:01), Stephen Frost wrote: * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: The attached patch tries to add one more security hook on the initialization of PostgreSQL instance (InitPostgres()). It gives the external security module a chance to initialize itself, and acquire credential of the client. I assumed the best place to initialize the module is just after the initialize_acl() invocation, if ESP is available. We have not discussed about this hook yet. So, I'd like to see any comments. Aren't modules given a __PG_Init or something similar that they can define which will be called when the module is loaded..? I assume the security module shall be loaded within 'shared_preload_libraries', because we can overwrite 'local_preload_libraries' (PGC_BACKEND) setting using connection string, so it allows users to bypass security features, doesn't it? Yeah, but so what? Stephen's point is still valid. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ExecutorCheckPerms() hook
* Robert Haas (robertmh...@gmail.com) wrote: This is essentially the same patch that I wrote and posted several weeks ago, with changes to the comments and renaming of the identifiers. Are you trying to represent it as your own work? Ehh, I doubt it. He had included your patch in another patch that he was working, which I then reviewed and asked him to update/change, and I think part of that was me asking that he keep the hook patch split out. He then split it out of his patch rather than just going back to yours. With all due respect, I intend to imply my own version. Please make your other proposed patches apply on top of that. This strikes me as a case of gee, won't git help here?. Perhaps we can use this as an opportunity to show how git can help. Then again, it's not exactly a huge patch. :) Thanks, Stephen (who won't mention the impetus for the hook being put here in the first place.. ;) signature.asc Description: Digital signature
Re: [HACKERS] Command to prune archive at restartpoints
Simon Riggs si...@2ndquadrant.com writes: Cleaning the archive directory, not the pg_xlog directory. Hence the choice of the directory where to act. I was slow on that, sorry guys. I guess my main problem here is that I still picture PostgreSQL has being able to maintain an archive with no external script in the simple case. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] add makeRangeTblEntry() into makefuncs.c
* Robert Haas (robertmh...@gmail.com) wrote: 2010/6/14 KaiGai Kohei kai...@ak.jp.nec.com: It adds makeRangeTblEntry() into makefuncs.c to keep the code more clean. It shall be also used for the upcoming DML refactor patch. In this refactoring, a common DML permission checker function take a list of RangeTblEntry, so the caller has to set up the object. I think this is the epitome of pointless. It looks to me like this just makes the code harder to read and very slightly slower without actually accomplishing any useful abstraction. I had suggested to KaiGai that he check if there was an existing function for creating an RTE rather than just malloc'ing it- he took that to mean he should add one if he couldn't find one. Wasn't my intent, but by the same token I didn't see it as a terribly bad thing either. Perhaps it should be improved or maybe we should just rip it out, but I rather prefer some kind of abstraction for that given it's use in a number of places. Of course, I may just be overly thinking it. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Mon, Jun 14, 2010 at 8:10 PM, Robert Haas robertmh...@gmail.com wrote: Maybe. That sounds like a pretty enormous foot-gun to me, considering that we have no way of recovering from the situation where the standby gets ahead of the master. No, we can do that by reconstructing the standby from the backup. And, that situation is not a problem for users including me who prefer to perform a failover when the master goes down. Of course, we can just restart the master in that case, but it's likely to take longer than a failover because there would be a cause of the crash. For example, if the master goes down because of a media crash, the master would never start up unless PITR is performed. So I'm not sure how many users prefer a restart to a failover. We should get the open item fixed for 9.0 here before we start worrying about 9.1. Yep, so I was submitting some patches in these days :) Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_archive_bypass
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 14/06/10 13:39, Dimitri Fontaine wrote: I tend to consider it a bug that there's no known way under windows to use the same trick as under Unix by using '/usr/bin/true' as your archive command. And this Unix trick itself does feel like a hack. Also I'd very much like to be able to recommend (even if not change the official defaults) to setup wal_level to archive, archive_mode=on and archive_command=pg_archive_bypass, so that the day you have a HA budget ain't the day you're going to restart the server to enable the fault tolerance settings… That particular use case is better handled by making archive_mode changeable on-the-fly. Now that we have wal_level as a separate GUC, it shouldn't be too hard. Ok. I like internal commands as a solution in general, but it's still in search of a problem.. What about /usr/bin/true, or a simple archive where you cp in a given location (which could happen to be a remote server thanks to unix network file systems or windows shares), etc. It seems to me those are existing problem that we solve poorly: let each user face the same pitfalls (error management). I also like Simon's approach to have options to internal commands too. What about this syntax : guc_name = pg_internal function_name arg1 %r ... argn Then function_name is any SQL callable function and you have the %x substitution still happening there for you. So you can mix hard-coded arguments (some path) and dynamic arguments. And you can script your commands in PL/WhateverU. It's more involved a patch, of course, but if we do that and provide some simple commands in -core, then most installations will only use that and stop bothering writing scripts to handle their simple cases. Which is what I'm after. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [v9.1] add makeRangeTblEntry() into makefuncs.c
On Mon, Jun 14, 2010 at 8:46 AM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: 2010/6/14 KaiGai Kohei kai...@ak.jp.nec.com: It adds makeRangeTblEntry() into makefuncs.c to keep the code more clean. It shall be also used for the upcoming DML refactor patch. In this refactoring, a common DML permission checker function take a list of RangeTblEntry, so the caller has to set up the object. I think this is the epitome of pointless. It looks to me like this just makes the code harder to read and very slightly slower without actually accomplishing any useful abstraction. I had suggested to KaiGai that he check if there was an existing function for creating an RTE rather than just malloc'ing it- he took that to mean he should add one if he couldn't find one. Wasn't my intent, but by the same token I didn't see it as a terribly bad thing either. Perhaps it should be improved or maybe we should just rip it out, but I rather prefer some kind of abstraction for that given it's use in a number of places. Of course, I may just be overly thinking it. Well, there's not much point in having a function that initializes ONE member of a 20+ member structure and leaves the initialization of all the rest to the caller. The structure effectively functions like a disjoint union, so maybe there'd be some value in having a function for build a relation RTE, which would set the rtekind to RTE_RELATION and take arguments for the other fields that pertain to that case. But the generic function proposed here doesn't really provide any meaningful abstraction. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Mon, Jun 14, 2010 at 8:41 AM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Jun 14, 2010 at 8:10 PM, Robert Haas robertmh...@gmail.com wrote: Maybe. That sounds like a pretty enormous foot-gun to me, considering that we have no way of recovering from the situation where the standby gets ahead of the master. No, we can do that by reconstructing the standby from the backup. And, that situation is not a problem for users including me who prefer to perform a failover when the master goes down. You don't get to pick - if a backend crashes on the master, it will restart right away and come up, but the slave will now be hosed... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump(all) --quote-all-identifiers
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 6:57 AM, Bruce Momjian br...@momjian.us wrote: Uh, I thought this was about quoting the identifiers. I am confused about why integer is an issue in this case. Can you show an example? Sure. INTEGER is actually a keyword in this context, not an identifier. (Remember the actual name of the type is int4, not integer.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Bruce Momjian br...@momjian.us writes: Magnus Hagander wrote: It means that we can't prevent people from configuring their tools to ignore important warning. We can't prevent them rom ignoring ERROR or FATAL either... My point is that most tools are going to look at the tag first to determine the severity of the message, and might even have log_min_messages set to ignore warnings. Why is this discussion based on the idea that we have to cater to incorrectly written log-filtering apps? The correct log level for this message is LOG. End of discussion. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 10:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Magnus Hagander wrote: It means that we can't prevent people from configuring their tools to ignore important warning. We can't prevent them rom ignoring ERROR or FATAL either... My point is that most tools are going to look at the tag first to determine the severity of the message, and might even have log_min_messages set to ignore warnings. Why is this discussion based on the idea that we have to cater to incorrectly written log-filtering apps? The correct log level for this message is LOG. End of discussion. Why? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14/06/2010 14:08, Robert Haas wrote: On Mon, Jun 14, 2010 at 3:48 AM, Jehan-Guillaume (ioguix) de Rorthais iog...@free.fr wrote: While translating the plperl page from the manual, I found the following sentence: The varname%_SHARED/varname variable and other global state within the language is public data. Should it be : The varname%_SHARED/varname variable and other global variables within the language are public data. ? State is a more general term, which would include any information that is globally accessible from within the interpreter, not just what's stored in variables. Off the top of my head, I'm not sure if there is anything like that, but I wouldn't bet on there not being any... Ok, so shouldn't it be The varname%_SHARED/varname variable and other global state(s?) within the language *are* public data ? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwWOhsACgkQxWGfaAgowiK1WwCfQbFGVIRW4PbJwdrV7fjRL8wt PiEAn3/NuXSGm5CjHtGzHREnl5DB+LBa =4IEb -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 10:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: The correct log level for this message is LOG. End of discussion. Why? Because it's not being issued in a user's session. The only place it can go is to the system log, and if you use a level of WARNING or less, it's likely to get filtered out by log_min_messages. I'm totally unimpressed by the argument that log-filtering applications don't know enough to pay attention to LOG messages. There are already a lot of those that are quite important to notice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 10:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 10:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: The correct log level for this message is LOG. End of discussion. Why? Because it's not being issued in a user's session. The only place it can go is to the system log, and if you use a level of WARNING or less, it's likely to get filtered out by log_min_messages. I'm totally unimpressed by the argument that log-filtering applications don't know enough to pay attention to LOG messages. There are already a lot of those that are quite important to notice. I'm willing to buy the above, but nobody has explained to my satisfaction why it's remotely sane to go into an infinite retry loop on an unrecoverable error. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
On Mon, Jun 14, 2010 at 10:18 AM, Jehan-Guillaume (ioguix) de Rorthais iog...@free.fr wrote: On 14/06/2010 14:08, Robert Haas wrote: On Mon, Jun 14, 2010 at 3:48 AM, Jehan-Guillaume (ioguix) de Rorthais iog...@free.fr wrote: While translating the plperl page from the manual, I found the following sentence: The varname%_SHARED/varname variable and other global state within the language is public data. Should it be : The varname%_SHARED/varname variable and other global variables within the language are public data. ? State is a more general term, which would include any information that is globally accessible from within the interpreter, not just what's stored in variables. Off the top of my head, I'm not sure if there is anything like that, but I wouldn't bet on there not being any... Ok, so shouldn't it be The varname%_SHARED/varname variable and other global state(s?) within the language *are* public data ? It seems correct to me as-is, but I just work here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 10:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: The correct log level for this message is LOG. ?End of discussion. Why? Because it's not being issued in a user's session. The only place it can go is to the system log, and if you use a level of WARNING or less, it's likely to get filtered out by log_min_messages. I'm totally unimpressed by the argument that log-filtering applications don't know enough to pay attention to LOG messages. There are already a lot of those that are quite important to notice. My point was that log filtering applications might ignore WARNING. They don't usually ignore LOG. 8 -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: I'm willing to buy the above, but nobody has explained to my satisfaction why it's remotely sane to go into an infinite retry loop on an unrecoverable error. That's a different question altogether ;-). I assume you're not satisfied by the change Heikki committed a couple hours ago? It will at least try to do something to recover. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 10:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I'm willing to buy the above, but nobody has explained to my satisfaction why it's remotely sane to go into an infinite retry loop on an unrecoverable error. That's a different question altogether ;-). I assume you're not satisfied by the change Heikki committed a couple hours ago? It will at least try to do something to recover. Yeah, I'm not satisfied by that. It's an improvement in the technical sense - it replaces an infinite retry that spins at top speed with a slower retry that won't flog your CPU quite so badly, but the chances that it will actually succeed in correcting the underlying problem seem infinitesimal. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 ... what's stored in variables. Off the top of my head, I'm not sure if there is anything like that, but I wouldn't bet on there not being any... I'm with Robert: I don't see much of a problem here. I might even suggest removing the references to non %_SHARED globals entirely. If there is something else that is implicitly shared, I'd like to know what it is. Tim B? Andrew D? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201006141055 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkwWQuQACgkQvJuQZxSWSsif6QCg0Vsfml3rn5nrIFUXF2oFCWka kPYAoONxmYh8YVUNJnoI8Mtd292PKFUD =YoqE -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, 2010-06-14 at 10:30 -0400, Tom Lane wrote: I'm totally unimpressed by the argument that log-filtering applications don't know enough to pay attention to LOG messages. There are already a lot of those that are quite important to notice. We have a log level where 1 log entry in a million is something serious and all the rest are generally ignorable. That situation leads to the cognitive phenomena is known as normalization of deviance, a term coined in the wake of the Challenger disaster. Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 10:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: That's a different question altogether ;-). I assume you're not satisfied by the change Heikki committed a couple hours ago? It will at least try to do something to recover. Yeah, I'm not satisfied by that. It's an improvement in the technical sense - it replaces an infinite retry that spins at top speed with a slower retry that won't flog your CPU quite so badly, but the chances that it will actually succeed in correcting the underlying problem seem infinitesimal. I'm not sure about that. walreceiver will refetch from the start of the current WAL page, so there's at least some chance of getting a good copy when we didn't have one before. However, I do agree that it's not helpful to loop forever. If we can easily make it retry once and then PANIC, I'd be for that --- otherwise I tend to agree that the best thing is just to PANIC immediately. There are many many situations where a slave resync will be necessary; a transmission error on the WAL data is just one more. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
Robert Haas wrote: Ok, so shouldn't it be The varname%_SHARED/varname variable and other global state(s?) within the language *are* public data ? It seems correct to me as-is, but I just work here. Umm, you don't say Joe and Mary is people. (Or I hope you don't.) So are looks correct here to me. If the subject of the verb is plural then the verb should be in the plural form. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
Fujii Masao masao.fu...@gmail.com writes: On Fri, Jun 11, 2010 at 11:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Well, we're already not waiting for fsync, which is the slowest part. No, currently walsender waits for fsync. No, you're mistaken. Walsender tries to send WAL up to xlogctl-LogwrtResult.Write. OTOH, xlogctl-LogwrtResult.Write is updated after XLogWrite() performs fsync. Wrong. LogwrtResult.Write tracks how far we've written out data, but it is only (known to be) fsync'd as far as LogwrtResult.Flush. But that change would cause the problem that Robert pointed out. http://archives.postgresql.org/pgsql-hackers/2010-06/msg00670.php Yes. Possibly walsender should only be allowed to send as far as LogwrtResult.Flush. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 10:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 10:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: That's a different question altogether ;-). I assume you're not satisfied by the change Heikki committed a couple hours ago? It will at least try to do something to recover. Yeah, I'm not satisfied by that. It's an improvement in the technical sense - it replaces an infinite retry that spins at top speed with a slower retry that won't flog your CPU quite so badly, but the chances that it will actually succeed in correcting the underlying problem seem infinitesimal. I'm not sure about that. walreceiver will refetch from the start of the current WAL page, so there's at least some chance of getting a good copy when we didn't have one before. The testing that I have been doing while we've been discussing this reveals that you are correct. I set up an HS/SR master and slave (running on the same machine), ran pgbench on the master, and then started randomly sending SIGSEGV to one of the master's backends. It seems that complaints about the WAL are possible on both master and slave. Here are a couple from the slave: LOG: unexpected pageaddr 0/89B7A000 in log file 0, segment 152, offset 12034048 WARNING: there is no contrecord flag in log file 0, segment 136, offset 2523136 LOG: invalid magic number in log file 0, segment 136, offset 2531328 The slave reconnects and then things get better. So I think your idea of retrying once and then panicking is probably best. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typo in plperl doc ?
On Mon, Jun 14, 2010 at 10:59 AM, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: Ok, so shouldn't it be The varname%_SHARED/varname variable and other global state(s?) within the language *are* public data ? It seems correct to me as-is, but I just work here. Umm, you don't say Joe and Mary is people. (Or I hope you don't.) So are looks correct here to me. If the subject of the verb is plural then the verb should be in the plural form. Yeah, you might be right. I won't attempt to explain why I thought this case might be different from that one... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC - Materialized Views - is stale or fresh?
Robert Haas robertmh...@gmail.com wrote: What Pavel's trying to do here is be smart about figuring out when an MV needs to be refreshed. I'm pretty sure this is the wrong way to go about it, but it seems entirely premature considering that we don't have a working implementation of a *manually* refreshed MV. Agreed all around. At the risk of sounding obsessed, this is an area where predicate locks might be usefully extended, if and when the serializable patch makes it in. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [PERFORM] Large (almost 50%!) performance drop after upgrading to 8.4.4?
Excerpts from Bruce Momjian's message of dom jun 13 10:00:16 -0400 2010: Why have I received no reply to this email? Do people think this is not a serious issue? I know it is a weekend but the problem was identified on Thursday, meaning there was a full workday for someone from CommandPrompt to reply to the issue and report a status: http://archives.postgresql.org/pgsql-performance/2010-06/msg00165.php The packager did reply to the original inquiry *on the same day*, but the moderator has not approved that email yet, it seems. I do have the reply on my mbox, with CC: pgsql-performance. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, 2010-06-14 at 11:14 -0400, Robert Haas wrote: On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. Yes, *and* how do we decide which this is? Should I downgrade all of my code to throwing LOGs rather than ERRORs, because (following the same argument) doing so would be better for high availability? It's not a facetious question. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. If he wants to throw ERROR, that's fine with me. If your point is that you think WARNING is more severe than LOG, the answer is that you need to readjust your thinking. It's not. See the sort order for log_min_messages values. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 11:34 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-06-14 at 11:14 -0400, Robert Haas wrote: On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. Yes, *and* how do we decide which this is? Should I downgrade all of my code to throwing LOGs rather than ERRORs, because (following the same argument) doing so would be better for high availability? It's not a facetious question. I would say - definitely not. High availability is all well and good, but if things are broken under the hood, it's not really so good after all. We've traditionally put data integrity and system reliability ahead of (for example) giving some answer to a query, even if it's the wrong answer. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_archive_bypass
On Mon, Jun 14, 2010 at 1:55 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: What about /usr/bin/true, or a simple archive where you cp in a given location (which could happen to be a remote server thanks to unix network file systems or windows shares), etc. It seems to me those are existing problem that we solve poorly: let each user face the same pitfalls (error management). I would like to see the case where the archive is just a mounted directory accessible through the filesystem be handled internally. Ideally if I could just specify the archive location on both the master and slave using the same parameter just containing a filesystem path then I could have the same configuration on both machines except for the actual parameter which decides whether they're a master or slave. That would make me much more confident I've configured both machines properly. All the nodes would have the same information and they would be deciding for themselves whether to push or pull archives based on whether they're the master or slave. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC - Materialized Views - is stale or fresh?
Heikki Linnakangas napsal(a): On 12/06/10 17:18, Pavel Baros wrote: I am curious how could I solve the problem: During refreshing I would like to know, if MV is stale or fresh? And I had an idea: In fact, MV need to know if its last refresh (transaction id) is older than any INSERT, UPDATE, DELETE transaction launched against source tables. So if MV has information about last (highest) xmin in source tables, it could simply compare its own xmin to xmins (xmax for deleted rows) from source tables and decide, if is stale or fresh. Whole realization could look like this: 1. Make new column in pg_class (or somewhere in pg_stat* ?): pg_class.rellastxid (of type xid) 2. After each INSERT, UPDATE, DELETE statement (transaction) pg_class.rellastxid would be updated. That should not be time- or memory- consuming (not so much) since pg_class is cached, I guess. rellastxid would have to be updated at every insert/update/delete. It would become a big bottleneck. That's not going to work. Why do you need to know if a MV is stale? Sorry I did not mention it. If we knew MV is fresh, there is no neeed to refresh MV and so it would prevent useless rebuilding of MV. So I thought there is room for saving some work. Anyway, I realized, this idea do not cover all the cases how to find out MV is stale or fresh. For example, when updating a row of source table of MV, that do not participate in MV, in that case refreshing of MV would be useless too. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: Command Prompt 8.4.4 PRMs compiled with debug/assert enabled
Bruce Momjian wrote: Bruce Momjian wrote: Bruce Momjian wrote: Bruce Momjian wrote: Alvaro Herrera wrote: Excerpts from Tom Lane's message of jue jun 10 11:46:25 -0400 2010: Yes, the folks at commandprompt need to be told about this. Loudly. It's a serious packaging error. Just notified Lacey, the packager (not so loudly, though); she's working on new packages, and apologizes for the inconvenience. [ Thread moved to hackers. 8.4.4 RPMs were built with debug flags. ] Uh, where arIf there are further questions, or needs, please let me know, and I will try to get them addressed as soon as I can.e we on this? Has it been completed? How are people informed about this? Do we need to post to the announce email list? Does Yum just update them? How did this mistake happen? How many days did it take to detect the problem? Why has no news been posted here? https://public.commandprompt.com/projects/pgcore/news Why have I received no reply to this email? Do people think this is not a serious issue? I know it is a weekend but the problem was identified on Thursday, meaning there was a full workday for someone from CommandPrompt to reply to the issue and report a status: http://archives.postgresql.org/pgsql-performance/2010-06/msg00165.php [ Updated subject line.] I am on IM with Joshua Drake right now and am working to get answers to the questions above. He or I will report in the next few hours. FYI, only Command Prompt-produced RPMs are affected. Devrim's RPMs are not: http://yum.postgresqlrpms.org/ I have still seen no public report about this, 12 hours after talking to Josh Drake on IM about it. :-( Hello Everyone, I tried to send something out Thursday about this to pgsql-performance, and I tried to send something out last night about this to pgsql-announce. Neither seem to have gotten through, or approved. =( =( =( Thursday to the Performance List: Hello Everyone, New packages for 8.4.4 on CentOS 5.5 and RHEL 5.5 (all arches), have been built, and are available in the PGDG repo. http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-i386/ http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-x86_64/ Output from pg_config --configure --version is below. x86_64: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-rpath' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib64' '--with-openssl' '--with-pam' '--with-krb5' '--with-gssapi' '--with-includes=/usr/include' '--with-libraries=/usr/lib64' '--enable-nls' '--enable-thread-safety' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--datadir=/usr/share/pgsql' '--with-docdir=/usr/share/doc' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et' 'CPPFLAGS= -I/usr/include/et' PostgreSQL 8.4.4 i386: '--build=i686-redhat-linux-gnu' '--host=i686-redhat-linux-gnu' '--target=i386-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-rpath' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib' '--with-openssl' '--with-pam' '--with-krb5' '--with-gssapi' '--with-includes=/usr/include' '--with-libraries=/usr/lib' '--enable-nls' '--enable-thread-safety' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--datadir=/usr/share/pgsql' '--with-docdir=/usr/share/doc' 'build_alias=i686-redhat-linux-gnu' 'host_alias=i686-redhat-linux-gnu' 'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -I/usr/include/et' 'CPPFLAGS= -I/usr/include/et' PostgreSQL 8.4.4 Again, I extend deep apologies for the inconvenience. If there is anything further we can help with, please let us know. Regards, Lacey And last night, for a public announcement: Dear PostgreSQL RPMS users, There was a mistake with the 8.4.4 packages resulting in --enable-debug and --enable-cassert being enabled in the
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. log_min_messages = warning # values in order of decreasing detail: # notice # warning # error # log # fatal # panic I've left out some lines, but the ones I left are in the right order and there's nothing missing in the range. So WARNING ERROR LOG FATAL, right? If that's the case, I guess Tom's right, once more, saying that LOG is fine here. If we want to be more subtle than that, we'd need to revise each and every error message and attribute it the right level, which it probably have already anyway. Regards. -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG 9.1 tentative timeline
On sön, 2010-06-13 at 12:11 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jun 12, 2010 at 1:45 AM, Peter Eisentraut pete...@gmx.net wrote: I wrote it down now: http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan Thanks! Looks good, except I thought our plan was to cut alpha1 before the git switch. Yes, I thought so too. The page lists those events in the right order, but now I see the associated dates don't agree. Yeah, I just wrote down the facts. If the facts don't make sense, we need to discuss changes. I think it would make most sense to moved the Git switch to, say, August 21, which btw. is a Saturday. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Reviewfest 2010-06 Plans and Call for Reviewers
Folks, The PostgreSQL 9.1 Development Plan: http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan calls for a ReviewFest to run from the 15th of June (tomorrow) until the start of the first CommitFest for the 9.1 release. The idea is that those with time available to contribute beyond what they can usefully contribute to getting 9.0 released can help provide feedback on patches submitted so far, to lighten the load of the CF proper when it starts. I have agreed to manage this RF. Of course, we also need reviewers. I do want to emphasize that we *don't* want this process to impact the release of 9.0; it is in the best interest of everyone that 9.0 is tested, stable, and released as soon as practicable. Please think hard about whether there is some testing or review you could do to facilitate the 9.0 release effort, and only participate in this RF to the extent that it doesn't detract from the other effort. Also, in testing these patches, be alert to any problems in the *before* version -- you may find 9.0 issues in the process of attempting to test these patches, and such issues, if found, should take priority. If you find a possible 9.0 issue, please set aside efforts to review the patch until you have pursued the preexisting issue. This is essentially being treated as an early start on the 2010-07 CF, so that is where the process will be managed: https://commitfest.postgresql.org/action/commitfest_view?id=6 Note that we don't expect any commits for these patches to happen until after the 9.0 stable branch is created and committers are done with their 9.0 release efforts, most likely some time after the 2010-07 CF is officially in progress. Also, we probably won't be bumping many patches to returned with feedback status during the RF; the apparent work required would need to be more than could reasonably be expected to be completed for the CF. Before signing up, please review these pages, to get an idea what's involved: http://wiki.postgresql.org/wiki/Reviewing_a_Patch http://wiki.postgresql.org/wiki/RRReviewers On the lighter side: http://wiki.postgresql.org/images/5/58/11_eggyknap-patch-review.pdf Please send me an email (without copying the list) if you are available to review; feel free to include any information that might be helpful in assigning you an appropriate patch. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, 2010-06-14 at 18:11 +0200, Dimitri Fontaine wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jun 14, 2010 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) I think Simon's point here is the same as mine - LOG isn't too high - it's too low. log_min_messages = warning# values in order of decreasing detail: # notice # warning # error # log # fatal # panic I've left out some lines, but the ones I left are in the right order and there's nothing missing in the range. So WARNING ERROR LOG FATAL, right? If that's the case, I guess Tom's right, once more, saying that LOG is fine here. If we want to be more subtle than that, we'd need to revise each and every error message and attribute it the right level, which it probably have already anyway. Nobody is arguing with what Tom has said about log levels. The problem is that LOG already has many things like performance logging which aren't a problem as all. So we need a level between LOG and FATAL to draw anyone's attention. @Robert - I'd point out that the behaviour of archive_cleanup_command and recovery_end_command is broken as a result of this discussion. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 12:31 PM, Simon Riggs si...@2ndquadrant.com wrote: If that's the case, I guess Tom's right, once more, saying that LOG is fine here. If we want to be more subtle than that, we'd need to revise each and every error message and attribute it the right level, which it probably have already anyway. Nobody is arguing with what Tom has said about log levels. Agreed. The problem is that LOG already has many things like performance logging which aren't a problem as all. So we need a level between LOG and FATAL to draw anyone's attention. Not sure I agree with this - what I think the problem is here is we need to make a clear distinction between recoverable errors and unrecoverable errors. @Robert - I'd point out that the behaviour of archive_cleanup_command and recovery_end_command is broken as a result of this discussion. :-( How so? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Robert Haas robertmh...@gmail.com writes: Not sure I agree with this - what I think the problem is here is we need to make a clear distinction between recoverable errors and unrecoverable errors. Um, if it's recoverable, it's not really an error ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] hstore == and deprecate =
On Sat, Jun 12, 2010 at 1:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: David E. Wheeler da...@kineticode.com writes: Which, IIRC, is new in 9.1, so could in theory be removed, especially if there was an hstore(text[], text[]) Oh --- now that I look, both that and the hstore = text[] one are new in 9.0, which means it is not too late to reverse course. So at this point the proposal is: * Leave the text = text operator alone for now, but deprecate it, and document/recommend the underlying hstore(text,text) function instead. * Get rid of the new text[] = text[] operator altogether, and provide/document only the underlying hstore(text[], text[]) function. * Rename the new hstore = text[] operator to something else. (I'm not quite sold on Florian's proposal, but don't have a better idea to offer offhand.) I notice that in 8.4 and before, the function underlying text = text wasn't called hstore() but tconvert(). Which is going to be a serious PITA for anyone who wants to write cross-version-compatible SQL using hstore. Can we do anything about this? I don't think we want to revert to calling it tconvert(). Can we retroactively add the alternate name hstore() to previous versions, and suggest that people do that manually in existing hstore installations? Here's a patch that removes the text[] = text[] operator - as suggested above - and instead documents hstore(text[], text[]). Barring objections, I will commit this and then start looking at the other portions of this proposal. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company hstore_remove_array_constructor_arrow.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 1:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Not sure I agree with this - what I think the problem is here is we need to make a clear distinction between recoverable errors and unrecoverable errors. Um, if it's recoverable, it's not really an error ... Gah. This thread is a terminological mess. I'm using the term error generally, not in the PG-specific sense of ERROR. problems after which there is a sane way to continue and problems after which there isn't -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, 2010-06-14 at 11:09 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: Should I be downgrading Hot Standby breakages to LOG? That will certainly help high availability as well. If a message is being issued in a non-user-connected session, there is basically not a lot of point in WARNING or below. It should either be LOG, or ERROR/FATAL/PANIC (which are probably all about the same thing in the startup process...) This looks like a significant issue to me. Code has been written for 9.0 that assumes WARNING LOG. I've pointed out two places in SR code, I'm pretty sure there are others. There is also lots of pre-existing code where we run the same code in a backend and in a server process, for example autovacuum. In those cases, all the WARNINGs issued in vacuum.c, for example, will all be filtered by the current log-level scheme. LOG is already over-used and so anything said at that level is drowned. In many areas of code we cannot use a higher level without trauma. That is a problem since we have no way to separate the truly important from the barely interesting. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] dblink_build_sql_update versus dropped columns
A recent bug report http://archives.postgresql.org/pgsql-admin/2010-06/msg00101.php shows that dblink_build_sql_update and friends are really not all there when it comes to dealing with dropped columns in the target table. The immediate cause of the reported crash is just an internal matter, but while looking at it I realized that there is also an API issue: are the column numbers in the passed-in primary_key_attnums array to be taken as logical or physical attnums? If the user extracted the array from a pg_index entry then they are physical attnums, but if he just writes the array by hand then they are probably logical numbers, ie, they would not count any dropped columns appearing before the PK columns. I suspect the point has never come up before because PKs are commonly the first columns anyway. The current effective behavior of the code is that the column numbers are physical numbers. Should we document it that way, or change it? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dblink_build_sql_update versus dropped columns
On 06/14/2010 10:58 AM, Tom Lane wrote: A recent bug report http://archives.postgresql.org/pgsql-admin/2010-06/msg00101.php shows that dblink_build_sql_update and friends are really not all there when it comes to dealing with dropped columns in the target table. Yup, was just looking at that... The immediate cause of the reported crash is just an internal matter, but while looking at it I realized that there is also an API issue: are the column numbers in the passed-in primary_key_attnums array to be taken as logical or physical attnums? If the user extracted the array from a pg_index entry then they are physical attnums, but if he just writes the array by hand then they are probably logical numbers, ie, they would not count any dropped columns appearing before the PK columns. Yes, it uses physical attnums, mainly because it was originally written before we even supported dropped columns and never changed/fixed it. I suspect the point has never come up before because PKs are commonly the first columns anyway. The current effective behavior of the code is that the column numbers are physical numbers. Should we document it that way, or change it? Probably it should be changed to deal with dropped columns correctly, but I won't have time to look at this closely until the end of the month -- is that soon enough? Thanks, Joe signature.asc Description: OpenPGP digital signature
Re: [HACKERS] dblink_build_sql_update versus dropped columns
Joe Conway m...@joeconway.com writes: On 06/14/2010 10:58 AM, Tom Lane wrote: The current effective behavior of the code is that the column numbers are physical numbers. Should we document it that way, or change it? Probably it should be changed to deal with dropped columns correctly, but I won't have time to look at this closely until the end of the month -- is that soon enough? Actually, I was working on it myself. On further reflection I think that logical numbers are clearly the right thing --- if we define it as being physical numbers then we will have headaches in the future when/if we support rearranging columns. However, there is some small chance of breaking things in existing DBs if we back-patch that change. Thoughts? It strikes me also that the code is not nearly careful enough about defending itself against garbage input in the primary_key_attnums argument ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC - Materialized Views - is stale or fresh?
At the risk of sounding obsessed, this is an area where predicate locks might be usefully extended, if and when the serializable patch makes it in. Yes, we see your patch in 9.1-first. ;-) -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Simon Riggs si...@2ndquadrant.com writes: LOG is already over-used and so anything said at that level is drowned. This is nonsense. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Simon Riggs si...@2ndquadrant.com wrote: LOG is already over-used and so anything said at that level is drowned. In many areas of code we cannot use a higher level without trauma. That is a problem since we have no way to separate the truly important from the barely interesting. The fact that LOG is categorized the same as INFO has led me to believe that they are morally equivalent -- that the only reason both exist is that one has entries of interest to system administrators and the other has interest to clients. http://www.postgresql.org/docs/8.4/interactive/runtime-config-logging.html#RUNTIME-CONFIG-SEVERITY-LEVELS Our shop chooses to log all connections and disconnections. That's mixed with such as which clients broke their connections without proper handshaking. I am surprised to hear that any time-critical alerts would be logged at this level, versus the sort of information you might want for forensic purposes. Perhaps anything which shouldn't be categorized as INFO to syslog should have some other (new?) level. ALERT, maybe? Mapping to ERR? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
Kevin Grittner kevin.gritt...@wicourts.gov writes: Simon Riggs si...@2ndquadrant.com wrote: LOG is already over-used and so anything said at that level is drowned. In many areas of code we cannot use a higher level without trauma. That is a problem since we have no way to separate the truly important from the barely interesting. The fact that LOG is categorized the same as INFO has led me to believe that they are morally equivalent -- They are not morally equivalent. INFO is for output that the user has explicitly requested appear on his console (eg, via VACUUM VERBOSE). So it's high priority for client output, not so much for log output. LOG is for information that is considered high priority to log, but not so much for client output (indeed maybe there is no client to output it to). If LOG is over-used then the problem is that we have LOG messages that ought to be downgraded to DEBUG. A normally functioning system should not be emitting *any* LOG messages during routine business, other than ones that the user explicitly requested (like log_connections). We have had this discussion in the past, in response to complaints that there was too much log traffic, and have downgraded things that were poorly categorized as LOG. Maybe there are some more of them in the HS/SR code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On 6/14/10 7:57 AM, Tom Lane wrote: However, I do agree that it's not helpful to loop forever. If we can easily make it retry once and then PANIC, I'd be for that --- otherwise I tend to agree that the best thing is just to PANIC immediately. There are many many situations where a slave resync will be necessary; a transmission error on the WAL data is just one more I'd vote for 3 or 5 times. Whatever is the same as we do with trying to get the log files. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning message in standby
On Mon, Jun 14, 2010 at 20:22, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: LOG is already over-used and so anything said at that level is drowned. This is nonsense. Whether it's over-used or not may be, but that doesn't make the general issue nonsense. But the fact is that having LOG at a higher priority than WARNING and ERROR is un-intuitive in the extreme, and I bet there aren't many non-experts who realize this. I don't know of any other software that does this. Heck, *experienced* postgresql users keep getting this wrong. Perhaps we need to re-think the naming there? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dblink_build_sql_update versus dropped columns
On 06/14/2010 11:21 AM, Tom Lane wrote:
Actually, I was working on it myself. On further reflection I think
that logical numbers are clearly the right thing --- if we define it
as being physical numbers then we will have headaches in the future
when/if we support rearranging columns. However, there is some small
chance of breaking things in existing DBs if we back-patch that change.
Thoughts?
I didn't even think people were using those functions for many years
since I never heard any complaints. I'd say better to not backpatch
changes to logical ordering, but FWIW the attached at least fixes the
immediate bug in head and ought to work at least a few branches.
It strikes me also that the code is not nearly careful enough about
defending itself against garbage input in the primary_key_attnums
argument ...
Probably not :-(
Joe
Index: dblink.c
===
RCS file: /opt/src/cvs/pgsql/contrib/dblink/dblink.c,v
retrieving revision 1.94
diff -c -r1.94 dblink.c
*** dblink.c 9 Jun 2010 03:39:26 - 1.94
--- dblink.c 14 Jun 2010 18:37:07 -
***
*** 94,100
static char *quote_literal_cstr(char *rawstr);
static char *quote_ident_cstr(char *rawstr);
static int16 get_attnum_pk_pos(int2vector *pkattnums, int16 pknumatts, int16 key);
! static HeapTuple get_tuple_of_interest(Oid relid, int2vector *pkattnums, int16 pknumatts, char **src_pkattvals);
static Oid get_relid_from_relname(text *relname_text);
static char *generate_relation_name(Oid relid);
static void dblink_connstr_check(const char *connstr);
--- 94,100
static char *quote_literal_cstr(char *rawstr);
static char *quote_ident_cstr(char *rawstr);
static int16 get_attnum_pk_pos(int2vector *pkattnums, int16 pknumatts, int16 key);
! static HeapTuple get_tuple_of_interest(Oid relid, int2vector *pkattnums, int16 pknumatts, char **src_pkattvals, TupleDesc *reltupdesc);
static Oid get_relid_from_relname(text *relname_text);
static char *generate_relation_name(Oid relid);
static void dblink_connstr_check(const char *connstr);
***
*** 1768,1774
static char *
get_sql_insert(Oid relid, int2vector *pkattnums, int16 pknumatts, char **src_pkattvals, char **tgt_pkattvals)
{
- Relation rel;
char *relname;
HeapTuple tuple;
TupleDesc tupdesc;
--- 1768,1773
***
*** 1784,1801
/* get relation name including any needed schema prefix and quoting */
relname = generate_relation_name(relid);
! /*
! * Open relation using relid
! */
! rel = relation_open(relid, AccessShareLock);
! tupdesc = rel-rd_att;
! natts = tupdesc-natts;
!
! tuple = get_tuple_of_interest(relid, pkattnums, pknumatts, src_pkattvals);
if (!tuple)
ereport(ERROR,
(errcode(ERRCODE_CARDINALITY_VIOLATION),
errmsg(source row not found)));
appendStringInfo(buf, INSERT INTO %s(, relname);
--- 1783,1794
/* get relation name including any needed schema prefix and quoting */
relname = generate_relation_name(relid);
! tuple = get_tuple_of_interest(relid, pkattnums, pknumatts, src_pkattvals, tupdesc);
if (!tuple)
ereport(ERROR,
(errcode(ERRCODE_CARDINALITY_VIOLATION),
errmsg(source row not found)));
+ natts = tupdesc-natts;
appendStringInfo(buf, INSERT INTO %s(, relname);
***
*** 1848,1854
}
appendStringInfo(buf, ));
- relation_close(rel, AccessShareLock);
return (buf.data);
}
--- 1841,1846
***
*** 1903,1909
static char *
get_sql_update(Oid relid, int2vector *pkattnums, int16 pknumatts, char **src_pkattvals, char **tgt_pkattvals)
{
- Relation rel;
char *relname;
HeapTuple tuple;
TupleDesc tupdesc;
--- 1895,1900
***
*** 1919,1936
/* get relation name including any needed schema prefix and quoting */
relname = generate_relation_name(relid);
! /*
! * Open relation using relid
! */
! rel = relation_open(relid, AccessShareLock);
! tupdesc = rel-rd_att;
! natts = tupdesc-natts;
!
! tuple = get_tuple_of_interest(relid, pkattnums, pknumatts, src_pkattvals);
if (!tuple)
ereport(ERROR,
(errcode(ERRCODE_CARDINALITY_VIOLATION),
errmsg(source row not found)));
appendStringInfo(buf, UPDATE %s SET , relname);
--- 1910,1921
/* get relation name including any needed schema prefix and quoting */
relname = generate_relation_name(relid);
! tuple = get_tuple_of_interest(relid, pkattnums, pknumatts, src_pkattvals, tupdesc);
if (!tuple)
ereport(ERROR,
(errcode(ERRCODE_CARDINALITY_VIOLATION),
errmsg(source row not found)));
+ natts = tupdesc-natts;
appendStringInfo(buf, UPDATE %s SET , relname);
***
*** 1992,1998
appendStringInfo(buf, IS NULL);
}
- relation_close(rel, AccessShareLock);
return (buf.data);
}
--- 1977,1982
***
*** 2050,2064
}
Re: [HACKERS] Typo in plperl doc ?
Robert Haas wrote: On Mon, Jun 14, 2010 at 10:59 AM, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: Ok, so shouldn't it be The varname%_SHARED/varname variable and other global state(s?) within the language *are* public data ? It seems correct to me as-is, but I just work here. Umm, you don't say Joe and Mary is people. (Or I hope you don't.) So are looks correct here to me. If the subject of the verb is plural then the verb should be in the plural form. Yeah, you might be right. I won't attempt to explain why I thought this case might be different from that one... Agreed, attached patch applied. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + Index: plperl.sgml === RCS file: /cvsroot/pgsql/doc/src/sgml/plperl.sgml,v retrieving revision 2.84 retrieving revision 2.85 diff -c -c -r2.84 -r2.85 *** plperl.sgml 13 May 2010 16:39:43 - 2.84 --- plperl.sgml 14 Jun 2010 18:47:05 - 2.85 *** *** 835,841 para The varname%_SHARED/varname variable and other global state within ! the language is public data, available to all PL/Perl functions within a session. Use with care, especially in situations that involve use of multiple roles or literalSECURITY DEFINER/ functions. /para --- 835,841 para The varname%_SHARED/varname variable and other global state within ! the language are public data, available to all PL/Perl functions within a session. Use with care, especially in situations that involve use of multiple roles or literalSECURITY DEFINER/ functions. /para -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dblink_build_sql_update versus dropped columns
Joe Conway m...@joeconway.com writes: I didn't even think people were using those functions for many years since I never heard any complaints. I'd say better to not backpatch changes to logical ordering, but FWIW the attached at least fixes the immediate bug in head and ought to work at least a few branches. [squint...] This doesn't appear to me to fix the problem. You really need the query-construction loops to track logical and physical numbers separately. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dblink_build_sql_update versus dropped columns
On 06/14/2010 11:54 AM, Tom Lane wrote: Joe Conway m...@joeconway.com writes: I didn't even think people were using those functions for many years since I never heard any complaints. I'd say better to not backpatch changes to logical ordering, but FWIW the attached at least fixes the immediate bug in head and ought to work at least a few branches. [squint...] This doesn't appear to me to fix the problem. You really need the query-construction loops to track logical and physical numbers separately. Hmmm, worked for the provided case, but this is a good example of why I *usually* don't send a patch to the list without spending more quality time thinking about the problem ;-) Joe signature.asc Description: OpenPGP digital signature
Re: [HACKERS] warning message in standby
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: The fact that LOG is categorized the same as INFO has led me to believe that they are morally equivalent -- They are not morally equivalent. INFO is for output that the user has explicitly requested appear on his console (eg, via VACUUM VERBOSE). So it's high priority for client output, not so much for log output. LOG is for information that is considered high priority to log, but not so much for client output (indeed maybe there is no client to output it to). I didn't say I thought they meant the same thing. You omitted the part where I said -- that the only reason both exist is that one has entries of interest to system administrators and the other has interest to clients. Which I think is saying the same thing you just said. If LOG is over-used then the problem is that we have LOG messages that ought to be downgraded to DEBUG. A normally functioning system should not be emitting *any* LOG messages during routine business, other than ones that the user explicitly requested (like log_connections). Current LOG level entries may be something which DBAs find as useful as you find the xmin/xmax breadcrumbs in tuples -- we normally don't want to look at all of these LOG entries, but we want them there to review in case of questions or problems. If anything of a time-critical nature is being written at the LOG level, I view that as a serious issue. We *should* have some logging level available to server processes which maps to something more serious than INFO in syslog and INFORMATION in the Windows event log but which doesn't indicate that something is being terminated. Perhaps we are currently using FATAL for these and I'm misunderstanding the issue. Even so, there seems to be a lot of room between LOG and FATAL, conceptually -- at least in the connotations those words have for me. ALERT keeps coming to mind. Maybe best mapped to NOTICE for syslog and WARNING for Windows event log. That seems to fill a gap in the logging levels. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] hstore == and deprecate =
Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom But actually, there's another issue here: hstore defines not one Tom but three = operators: Tom text = textyields hstore (with 1 element) Tom text[] = text[]yields hstore (with N elements) Tom hstore = text[]yields hstore (subset) Tom It's reasonable to say that the first two are bad design, but Tom I'm a bit less willing to say that the last one is. What shall Tom we do with that? I added the second two primarily by analogy with the first; following the existing pattern seemed to be the way to go at the time. If the first (text = text) operator hadn't already been present when I started looking at it, I'd probably have stuck to hstore() for all construction methods rather than defining an operator. Creating operators that take only existing builtin types is obviously a namespace problem in that multiple independent modules might get into trouble by choosing the same operators. Perhaps this should be formalized as some sort of style guideline for module authors? I'm happy with deprecating the first two = in favour of hstore() if that is in line with general opinion. The hstore = text[] slice could be replaced by another operator name; the existing name comes from the analogy that (hstore - text[]) returns the list of values, whereas (hstore = text[]) returns both the keys and values. -- Andrew (irc:RhodiumToad) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Exposing the Xact commit order to the user
Excerpts from Marko Kreen's message of jue jun 10 18:10:50 -0400 2010: Jan's proposal of storing small struct into segmented files sounds like it could work. Can't say anything more because I can't imagine it as well as Jan. Would need to play with working implementation to say more... We already have such a thing -- see pg_multixact -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: Command Prompt 8.4.4 PRMs compiled with debug/assert enabled
Lacey Powers wrote: I tried to send something out Thursday about this to pgsql-performance, and I tried to send something out last night about this to pgsql-announce. Neither seem to have gotten through, or approved. =( =( =( Yes, I suspected that might have happened. Thursday to the Performance List: Hello Everyone, New packages for 8.4.4 on CentOS 5.5 and RHEL 5.5 (all arches), have been built, and are available in the PGDG repo. http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-i386/ http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-x86_64/ ... Again, I extend deep apologies for the inconvenience. If there is anything further we can help with, please let us know. Do any of the other minor releases made at the same time have this problem, or just 8.4.4? And last night, for a public announcement: Dear PostgreSQL RPMS users, There was a mistake with the 8.4.4 packages resulting in --enable-debug and --enable-cassert being enabled in the packages for CentOS 5.5 x86_64 and i386. This has been corrected in the 8.4.4-2PGDG packages, which are in the PostgreSQL RPMS repository. Please update to these corrected packages as soon as possible. We apologize for any inconvenience. OK, do the Yum folks get these updates automatically? I had this fixed and out in the repo about an hour after I was made aware of it (Alvaro let me know at ~9:30AM PDT ( Thank you *so* much, Alvaro! =) ), and I had things out at ~10:45AM PDT, and tried to reply shortly thereafter. =( ), and tried to let people know as best I could. I know there are a great deal of concerns regarding this, and I am greatly sorry for any trouble that was caused, and will add tests to the build process to ensure that this does not happen again. =( Given the concern, I thought I'd try posting a reply here, to this email, to soothe fears, and to plead for some moderator help, since both of my emails are most likely stuck in moderation. =( =( Again, I'm sorry for the issues I caused, and I will endeavor to make the turnaround and notification time quicker in the future. =( If there are further questions, or needs, please let me know, and I will try to get them addressed as soon as I can. OK, how do we properly get rid of all those buggy 8.4.4 installs? Seems a posting to announce is not enough, and we need to show users how to tell if they are running a de-buggy version. Does the fixed 8.4.4 install have a different visible version number, or do they have to use SHOW debug_assertions? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: Command Prompt 8.4.4 PRMs compiled with debug/assert enabled
Bruce Momjian br...@momjian.us writes: OK, how do we properly get rid of all those buggy 8.4.4 installs? Seems a posting to announce is not enough, and we need to show users how to tell if they are running a de-buggy version. The original thread already covered that in sufficient detail: check debug_assertions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: Command Prompt 8.4.4 PRMs compiled with debug/assert enabled
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: OK, how do we properly get rid of all those buggy 8.4.4 installs? Seems a posting to announce is not enough, and we need to show users how to tell if they are running a de-buggy version. The original thread already covered that in sufficient detail: check debug_assertions. But this was not communicated in the announce email, which was part of my point. We need to tell people how to get the fix, and how to audit their systems to know they have all been upgrade. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + None of us is going to be here forever. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
