Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 2017-06-06 12:53:21 -0700, Andres Freund wrote: > On 2017-06-06 15:48:42 -0400, Robert Haas wrote: > > On Fri, Jun 2, 2017 at 3:24 PM, Andres Freundwrote: > > > Latches work in single user mode, it's just that the new code for some > > > reason uses uninitialized memory as the latch. As I pointed out above, > > > the new code really should just use MyLatch instead of > > > MyProc->procLatch. > > FWIW, I'd misremembered some code here, and we actually reach the > function initializing the shared latch, even in single user mode. > > > > We seem to have accumulated quite a few instance of that. > > > > [rhaas pgsql]$ git grep MyLatch | wc -l > > 116 > > [rhaas pgsql]$ git grep 'MyProc->procLatch' | wc -l > > 33 > > > > Most of the offenders are in src/backend/replication, but there are > > some that are related to parallelism as well (bgworker.c, pqmq.c, > > parallel.c, condition_variable.c). Maybe we (you?) should just go and > > change them all. I don't think using MyLatch instead of > > MyProc->procLatch has become automatic for everyone yet. > > Nevertheless this should be changed. Will do. Here's the patch for that, also addressing some issues I found while updating those callsites (separate thread started, too). - Andres >From 9206ced1dc05d3a9cc99faafa22d5d8b16d998d1 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 6 Jun 2017 16:13:00 -0700 Subject: [PATCH] Clean up latch related code. The larger part of this patch replaces usages of MyProc->procLatch with MyLatch. The latter works even early during backend startup, where MyProc->procLatch doesn't yet. While the affected code shouldn't run in cases where it's not initialized, it might get copied into places where it might. Using MyLatch is simpler and a bit faster to boot, so there's little point to stick with the previous coding. While doing so I noticed some weaknesses around newly introduced uses of latches that could lead to missed events, and an omitted CHECK_FOR_INTERRUPTS() call in worker_spi. As all the actual bugs are in v10 code, there doesn't seem to be sufficient reason to backpatch this. Author: Andres Freund Discussion: https://postgr.es/m/20170606195321.sjmenrfgl2nu6...@alap3.anarazel.de https://postgr.es/m/20170606210405.sim3yl6vpudhm...@alap3.anarazel.de Backpatch: - --- src/backend/access/transam/parallel.c | 4 +-- src/backend/libpq/pqmq.c | 4 +-- src/backend/postmaster/bgworker.c | 4 +-- .../libpqwalreceiver/libpqwalreceiver.c| 13 src/backend/replication/logical/launcher.c | 35 +++--- src/backend/replication/logical/tablesync.c| 12 src/backend/replication/logical/worker.c | 10 +-- src/backend/storage/lmgr/condition_variable.c | 6 ++-- src/test/modules/worker_spi/worker_spi.c | 2 ++ 9 files changed, 56 insertions(+), 34 deletions(-) diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c index cb22174270..16a0bee610 100644 --- a/src/backend/access/transam/parallel.c +++ b/src/backend/access/transam/parallel.c @@ -527,9 +527,9 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt) if (!anyone_alive) break; - WaitLatch(>procLatch, WL_LATCH_SET, -1, + WaitLatch(MyLatch, WL_LATCH_SET, -1, WAIT_EVENT_PARALLEL_FINISH); - ResetLatch(>procLatch); + ResetLatch(MyLatch); } if (pcxt->toc != NULL) diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c index 96939327c3..8fbc03819d 100644 --- a/src/backend/libpq/pqmq.c +++ b/src/backend/libpq/pqmq.c @@ -172,9 +172,9 @@ mq_putmessage(char msgtype, const char *s, size_t len) if (result != SHM_MQ_WOULD_BLOCK) break; - WaitLatch(>procLatch, WL_LATCH_SET, 0, + WaitLatch(MyLatch, WL_LATCH_SET, 0, WAIT_EVENT_MQ_PUT_MESSAGE); - ResetLatch(>procLatch); + ResetLatch(MyLatch); CHECK_FOR_INTERRUPTS(); } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index c3454276bf..712d700481 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -1144,7 +1144,7 @@ WaitForBackgroundWorkerShutdown(BackgroundWorkerHandle *handle) if (status == BGWH_STOPPED) break; - rc = WaitLatch(>procLatch, + rc = WaitLatch(MyLatch, WL_LATCH_SET | WL_POSTMASTER_DEATH, 0, WAIT_EVENT_BGWORKER_SHUTDOWN); @@ -1154,7 +1154,7 @@ WaitForBackgroundWorkerShutdown(BackgroundWorkerHandle *handle) break; } - ResetLatch(>procLatch); + ResetLatch(MyLatch); } return status; diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 726d1b5bd8..89c34b8225 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -176,7 +176,7 @@
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 2017-06-06 15:48:42 -0400, Robert Haas wrote: > On Fri, Jun 2, 2017 at 3:24 PM, Andres Freundwrote: > > Latches work in single user mode, it's just that the new code for some > > reason uses uninitialized memory as the latch. As I pointed out above, > > the new code really should just use MyLatch instead of > > MyProc->procLatch. FWIW, I'd misremembered some code here, and we actually reach the function initializing the shared latch, even in single user mode. > We seem to have accumulated quite a few instance of that. > > [rhaas pgsql]$ git grep MyLatch | wc -l > 116 > [rhaas pgsql]$ git grep 'MyProc->procLatch' | wc -l > 33 > > Most of the offenders are in src/backend/replication, but there are > some that are related to parallelism as well (bgworker.c, pqmq.c, > parallel.c, condition_variable.c). Maybe we (you?) should just go and > change them all. I don't think using MyLatch instead of > MyProc->procLatch has become automatic for everyone yet. Nevertheless this should be changed. Will do. - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Fri, Jun 2, 2017 at 3:24 PM, Andres Freundwrote: > Latches work in single user mode, it's just that the new code for some > reason uses uninitialized memory as the latch. As I pointed out above, > the new code really should just use MyLatch instead of > MyProc->procLatch. We seem to have accumulated quite a few instance of that. [rhaas pgsql]$ git grep MyLatch | wc -l 116 [rhaas pgsql]$ git grep 'MyProc->procLatch' | wc -l 33 Most of the offenders are in src/backend/replication, but there are some that are related to parallelism as well (bgworker.c, pqmq.c, parallel.c, condition_variable.c). Maybe we (you?) should just go and change them all. I don't think using MyLatch instead of MyProc->procLatch has become automatic for everyone yet. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 6/2/17 23:06, Peter Eisentraut wrote: > On 6/2/17 15:41, Tom Lane wrote: >> It's certainly plausible that we could have the latch code just ignore >> WL_POSTMASTER_DEATH if not IsUnderPostmaster. I think that the original >> reasoning for not doing that was that the calling code should know which >> environment it's in, and not pass an unimplementable wait-exit reason; >> so silently ignoring the bit could mask a bug. Perhaps that argument is >> no longer attractive. Alternatively, we could fix the relevant call sites >> to do "(IsUnderPostmaster ? WL_POSTMASTER_DEATH : 0)", and keep the strict >> behavior for the majority of call sites. > > There are a lot of those call sites. (And a lot of duplicate code for > what to do if postmaster death actually happens.) I doubt we want to > check them all. > > The attached patch fixes the reported issue for me. committed -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 6/2/17 15:24, Andres Freund wrote: > but that doesn't really solve the issue that > logical rep pretty essentially requires multiple processes. But it may be sensible to execute certain DDL commands for repair, which is why I'm arguing for a finer-grained approach than just prohibiting everything. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 6/2/17 15:41, Tom Lane wrote: > It's certainly plausible that we could have the latch code just ignore > WL_POSTMASTER_DEATH if not IsUnderPostmaster. I think that the original > reasoning for not doing that was that the calling code should know which > environment it's in, and not pass an unimplementable wait-exit reason; > so silently ignoring the bit could mask a bug. Perhaps that argument is > no longer attractive. Alternatively, we could fix the relevant call sites > to do "(IsUnderPostmaster ? WL_POSTMASTER_DEATH : 0)", and keep the strict > behavior for the majority of call sites. There are a lot of those call sites. (And a lot of duplicate code for what to do if postmaster death actually happens.) I doubt we want to check them all. The attached patch fixes the reported issue for me. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services From 8f0e1c844aabd4a0e3c245180d9019cbf65b1601 Mon Sep 17 00:00:00 2001 From: Peter EisentrautDate: Fri, 2 Jun 2017 23:02:13 -0400 Subject: [PATCH] Ignore WL_POSTMASTER_DEATH latch event in single user mode Otherwise code that uses this will abort with an assertion failure, because postmaster_alive_fds are not initialized. Reported-by: tushar --- src/backend/storage/ipc/latch.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c index 53e6bf2477..55959de91f 100644 --- a/src/backend/storage/ipc/latch.c +++ b/src/backend/storage/ipc/latch.c @@ -370,7 +370,7 @@ WaitLatchOrSocket(volatile Latch *latch, int wakeEvents, pgsocket sock, AddWaitEventToSet(set, WL_LATCH_SET, PGINVALID_SOCKET, (Latch *) latch, NULL); - if (wakeEvents & WL_POSTMASTER_DEATH) + if (wakeEvents & WL_POSTMASTER_DEATH && IsUnderPostmaster) AddWaitEventToSet(set, WL_POSTMASTER_DEATH, PGINVALID_SOCKET, NULL, NULL); -- 2.13.0 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
Peter Eisentrautwrites: > My point is that we shouldn't be putting checks into DDL commands about > single-user mode if the actual cause of the issue is in a lower-level > system. Not all uses of a particular DDL command necessary use a latch, > for example. Also, there could be other things that hit a latch that > are reachable in single-user mode that we haven't found yet. > So I think the check should either go somewhere in the latch code, or > possibly in the libpqwalreceiver code. Or we make the latch code work > so that the check-for-postmaster-death code becomes a noop in > single-user mode. Suggestions? It's certainly plausible that we could have the latch code just ignore WL_POSTMASTER_DEATH if not IsUnderPostmaster. I think that the original reasoning for not doing that was that the calling code should know which environment it's in, and not pass an unimplementable wait-exit reason; so silently ignoring the bit could mask a bug. Perhaps that argument is no longer attractive. Alternatively, we could fix the relevant call sites to do "(IsUnderPostmaster ? WL_POSTMASTER_DEATH : 0)", and keep the strict behavior for the majority of call sites. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 2017-06-02 15:00:21 -0400, Peter Eisentraut wrote: > On 6/1/17 21:55, Andres Freund wrote: > > On 2017-06-01 21:42:41 -0400, Peter Eisentraut wrote: > >> We should look at what the underlying problem is before we prohibit > >> anything at a high level. > > > > I'm not sure there's any underlying issue here, except being in single > > user mode. > > My point is that we shouldn't be putting checks into DDL commands about > single-user mode if the actual cause of the issue is in a lower-level > system. But it's not really. > Not all uses of a particular DDL command necessary use a latch, > for example. Also, there could be other things that hit a latch that > are reachable in single-user mode that we haven't found yet. Latches work in single user mode, it's just that the new code for some reason uses uninitialized memory as the latch. As I pointed out above, the new code really should just use MyLatch instead of MyProc->procLatch. > So I think the check should either go somewhere in the latch code, or > possibly in the libpqwalreceiver code. Or we make the latch code work > so that the check-for-postmaster-death code becomes a noop in > single-user mode. Suggestions? I don't think the postmaster death code is really the issue here. Nor is libpqwalreceiver really the issue. We can put ERRORs in a bunch of unrelated subsystems, sure, but that doesn't really solve the issue that logical rep pretty essentially requires multiple processes. We've prevented parallelism from being used in general (cf. standard_planner), we've not put checks in all the subsystems it uses. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 6/1/17 21:55, Andres Freund wrote: > On 2017-06-01 21:42:41 -0400, Peter Eisentraut wrote: >> We should look at what the underlying problem is before we prohibit >> anything at a high level. > > I'm not sure there's any underlying issue here, except being in single > user mode. My point is that we shouldn't be putting checks into DDL commands about single-user mode if the actual cause of the issue is in a lower-level system. Not all uses of a particular DDL command necessary use a latch, for example. Also, there could be other things that hit a latch that are reachable in single-user mode that we haven't found yet. So I think the check should either go somewhere in the latch code, or possibly in the libpqwalreceiver code. Or we make the latch code work so that the check-for-postmaster-death code becomes a noop in single-user mode. Suggestions? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 2017-06-01 21:42:41 -0400, Peter Eisentraut wrote: > We should look at what the underlying problem is before we prohibit > anything at a high level. I'm not sure there's any underlying issue here, except being in single user mode. > When I try it, I get a > > TRAP: FailedAssertion("!(event->fd != (-1))", File: "latch.c", Line: 861) > > which might indicate that there is a more general problem with latch use > in single-user mode. That just means that the latch isn't initialized. Which makes: > If I remove that assertion, things work fine after that. The originally > reported error "epoll_ctl() failed: Bad file descriptor" might indicate > that there is platform-dependent behavior. quite unsurprising. I'm not sure how this hints at platform dependent behaviour? libpqrcv_connect() uses MyProc->procLatch, which doesn't exist/isn't initialized in single user mode. I'm very unclear why that code uses MyProc->procLatch rather than MyLatch, but that'd not change anything - the tablesync stuff etc would still not work. - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On 6/1/17 04:49, Dilip Kumar wrote: > On Thu, Jun 1, 2017 at 1:02 PM, Michael Paquier >wrote: >> Thanks, this looks correct to me at quick glance. >> >> +if (!IsUnderPostmaster) >> +ereport(FATAL, >> +(errcode(ERRCODE_FEATURE_NOT_SUPPORTED), >> +errmsg("subscription commands are not supported by >> single-user servers"))); >> The messages could be more detailed, like directly the operation of >> CREATE/ALTER/DROP SUBCRIPTION in each error message. But that's a nit. > > Thanks for looking into it. Yeah, I think it's better to give > specific message instead of generic because we still support some of > the subscription commands even in single-user mode i.e ALTER > SUBSCRIPTION OWNER. My patch doesn't block this command and there is > no harm in supporting this in single-user mode but does this make any > sense? We may create some use case like creation subscription in > normal mode and then ALTER OWNER in single user mode but it makes > little sense to me. We should look at what the underlying problem is before we prohibit anything at a high level. When I try it, I get a TRAP: FailedAssertion("!(event->fd != (-1))", File: "latch.c", Line: 861) which might indicate that there is a more general problem with latch use in single-user mode. If I remove that assertion, things work fine after that. The originally reported error "epoll_ctl() failed: Bad file descriptor" might indicate that there is platform-dependent behavior. I think the general problem is that the latch code that checks for postmaster death does not handle single-user mode well. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Thu, Jun 1, 2017 at 1:02 PM, Michael Paquierwrote: > Thanks, this looks correct to me at quick glance. > > +if (!IsUnderPostmaster) > +ereport(FATAL, > +(errcode(ERRCODE_FEATURE_NOT_SUPPORTED), > +errmsg("subscription commands are not supported by > single-user servers"))); > The messages could be more detailed, like directly the operation of > CREATE/ALTER/DROP SUBCRIPTION in each error message. But that's a nit. Thanks for looking into it. Yeah, I think it's better to give specific message instead of generic because we still support some of the subscription commands even in single-user mode i.e ALTER SUBSCRIPTION OWNER. My patch doesn't block this command and there is no harm in supporting this in single-user mode but does this make any sense? We may create some use case like creation subscription in normal mode and then ALTER OWNER in single user mode but it makes little sense to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com subscription_error_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Wed, May 31, 2017 at 10:49 PM, Dilip Kumarwrote: > On Wed, May 31, 2017 at 2:20 PM, Michael Paquier > wrote: >> Yeah, see 0e0f43d6 for example. A simple fix is to look at >> IsUnderPostmaster when creating, altering or dropping a subscription >> in subscriptioncmds.c. > > Yeah, below patch fixes that. Thanks, this looks correct to me at quick glance. +if (!IsUnderPostmaster) +ereport(FATAL, +(errcode(ERRCODE_FEATURE_NOT_SUPPORTED), +errmsg("subscription commands are not supported by single-user servers"))); The messages could be more detailed, like directly the operation of CREATE/ALTER/DROP SUBCRIPTION in each error message. But that's a nit. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Wed, May 31, 2017 at 2:20 PM, Michael Paquierwrote: > Yeah, see 0e0f43d6 for example. A simple fix is to look at > IsUnderPostmaster when creating, altering or dropping a subscription > in subscriptioncmds.c. Yeah, below patch fixes that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com subscription_error.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Wed, May 31, 2017 at 7:01 AM, Dilip Kumarwrote: > On Wed, May 31, 2017 at 7:54 AM, tushar wrote: >> centos@centos-cpula bin]$ ./postgres --single postgres -D m1data >> PostgreSQL stand-alone backend 10beta1 >> backend> create subscription sub connection 'dbname=postgres port=5433 >> user=centos' publication p with (create_slot=0,enabled=off); >> 2017-05-31 12:53:09.318 BST [10469] LOG: statement: create subscription sub >> connection 'dbname=postgres port=5433 user=centos' publication p with >> (create_slot=0,enabled=off); >> >> 2017-05-31 12:53:09.326 BST [10469] ERROR: epoll_ctl() failed: Bad file >> descriptor > > IMHO, In single user mode, it can not support replication (it can not > have background WALReciver task). However, I believe there should be a > proper error if the above statement is correct. Yeah, see 0e0f43d6 for example. A simple fix is to look at IsUnderPostmaster when creating, altering or dropping a subscription in subscriptioncmds.c. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error while creating subscription when server is running in single user mode
On Wed, May 31, 2017 at 7:54 AM, tusharwrote: > centos@centos-cpula bin]$ ./postgres --single postgres -D m1data > PostgreSQL stand-alone backend 10beta1 > backend> create subscription sub connection 'dbname=postgres port=5433 > user=centos' publication p with (create_slot=0,enabled=off); > 2017-05-31 12:53:09.318 BST [10469] LOG: statement: create subscription sub > connection 'dbname=postgres port=5433 user=centos' publication p with > (create_slot=0,enabled=off); > > 2017-05-31 12:53:09.326 BST [10469] ERROR: epoll_ctl() failed: Bad file > descriptor IMHO, In single user mode, it can not support replication (it can not have background WALReciver task). However, I believe there should be a proper error if the above statement is correct. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Error while creating subscription when server is running in single user mode
Hi, There is an error while creating subscription when server is running in single user mode centos@centos-cpula bin]$ ./postgres --single postgres -D m1data PostgreSQL stand-alone backend 10beta1 backend> create subscription sub connection 'dbname=postgres port=5433 user=centos' publication p with (create_slot=0,enabled=off); 2017-05-31 12:53:09.318 BST [10469] LOG: statement: create subscription sub connection 'dbname=postgres port=5433 user=centos' publication p with (create_slot=0,enabled=off); 2017-05-31 12:53:09.326 BST [10469] ERROR: epoll_ctl() failed: Bad file descriptor -- regards,tushar EnterpriseDB https://www.enterprisedb.com/ The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers