Re: [HACKERS] Feature freeze date for 8.1
Oliver Jowett [EMAIL PROTECTED] writes: Tom Lane wrote: I'm not convinced that Postgres ought to provide a way to second-guess the TCP stack ... Would you be ok with a patch that allowed configuration of the TCP_KEEPCNT / TCP_KEEPIDLE / TCP_KEEPINTVL socket options on backend sockets? [ shrug... ] As long as it doesn't fail to build on platforms that don't offer those options, I couldn't complain too hard. But do we really need all that? regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
Russell Smith [EMAIL PROTECTED] writes: I would prefer an idle timeout if it's not costly. Because otherwise estimates need to be made about how long VACUUM and backup could take, and set the timeout longer. Why? No one has suggested that the same timeout must be applied to every connection. Clients that are going to do maintenance stuff like VACUUM could just disable the timeout. This does bring up thoughts of whether the timeout needs to be a protected variable (SUSET or higher). I'd argue not, since a noncooperative client can certainly cause performance issues aplenty no matter what you try to impose with timeouts. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Feature freeze date for 8.1
Tom Lane wrote: #3 Defend against client holding locks unreasonably long, even though not idle I can't get too excited about this case. If the client is malicious, this feature is surely insufficient to stop them from consuming a lot of resources (for example, they could easily drop and then reacquire the locks every (timeout * 0.9) seconds). And how many DBAs are really going to want to abort non-malicious clients doing useful work if they happen to exceed a certain total runtime? Perhaps a motivating example would help... I claim that if you have a problem with #1 you ought to go discuss it with some TCP hackers: you basically want to second-guess the TCP stack's ideas about appropriate timeouts. Well, no -- you might want to set a different timeout for PostgreSQL connections than for other connections. Is there a way to change the socket timeout for some subset of the processes on the machine without hacking the client or server source? You might also want to set this timeout on a more granular basis (e.g. per user, per database, etc.) Implementing this via setting a socket option (provided it can be done portably) would be fine with me. -Neil ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 2 May 2005, Tom Lane wrote: #1 Defend against loss of connectivity to client I claim that if you have a problem with #1 you ought to go discuss it with some TCP hackers: you basically want to second-guess the TCP stack's ideas about appropriate timeouts. Maybe you know what you are doing or maybe not, but it's not a database-level issue. Different applications can have different needs here. For some it's okay to wait a long time, for others it is not. The tcp hackers have provided an api for clients to set these values per socket (setsockopt with TCP_KEEPIDLE and similar (in linux at least)). My problem with the above setting is that some operations can be in progress for a long time on the server without generating any tcp/ip traffic to the client (a non verbose vacuum I guess is such a case). Such an operation would look like it's idle. There is an overlap with session and transaction timeouts, most applications work fine with any of these. -- /Dennis Björklund ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] SPI bug.
Neil Conway wrote: As I said before, we may or may not want to change the executor itself to use a constant-sized type, but as a matter of interface definition, I think using long makes the most sense. One thing that I forgot. If you indeed will do something like that in the future, the implication is yet another change to the SPI interfaces. Why not decide, once and for all and right now, what the size of this integer should be and then *start* with a change of the interface. The change of the underlying implementation can come later. Now you effectively force a second change that will make things incompatible should you decide to change the executor in the future. Regards, Thomas Hallgren ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Feature freeze date for 8.1
Tom Lane wrote: Oliver Jowett [EMAIL PROTECTED] writes: Tom Lane wrote: I'm not convinced that Postgres ought to provide a way to second-guess the TCP stack ... Would you be ok with a patch that allowed configuration of the TCP_KEEPCNT / TCP_KEEPIDLE / TCP_KEEPINTVL socket options on backend sockets? [ shrug... ] As long as it doesn't fail to build on platforms that don't offer those options, I couldn't complain too hard. But do we really need all that? I can't see how you'd aggregate or discard any of those options without losing useful tuning knobs.. if you're going to have one, you might as well have them all. -O ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
Neil Conway wrote: Is there a way to change the socket timeout for some subset of the processes on the machine without hacking the client or server source? The only ways I can see of tuning the TCP idle parameters on Linux are globally via sysfs, or per-socket via setsockopt(). You could LD_PRELOAD something to wrap accept(), I suppose, but that seems needlessly ugly.. -O ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] custom guc vars
[2005-05-01 18:38] Andrew Dunstan said:
|
| Is there a readme somewhere on how modules are supposed to use custom
| GUC variables? If there is I have missed it.
I don't think there is any documentation for this, but here's a
simple overview.
cheers.
Brent
=== postgresql.conf ===
custom_variable_classes = 'mymodule, anothermodule'
mymodule.integer = 10
mymodule.double = 5.5
mymodule.string = 'some string'
mymodule.bool = true
anothermodule.whatever = 'some string'
anothermodule.other = false
=== moduleConfig.c ===
#include utils/guc.h
int anIntVar;
char* aStringVar;
double aDoubleVar;
bool aBoolVar;
void setCustomVars()
{
DefineCustomIntegerVariable(
mymodule.integer,
A custom integer guc var,
NULL,
anIntVar,
PGC_USERSET,
NULL,NULL);
DefineCustomStringVariable(
mymodule.string,
A custom string guc var,
NULL,
aStringVar,
PGC_USERSET,
NULL,NULL);
DefineCustomRealVariable(
mymodule.double,
A custom double guc var,
NULL,
aDoubleVar,
PGC_USERSET,
NULL,NULL);
DefineCustomBoolVariable(
mymodule.bool,
A custom bool guc var,
NULL,
aBoolVar,
PGC_USERSET,
NULL,NULL);
}
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] Feature freeze date for 8.1
Peter Eisentraut wrote: Neil Conway wrote: The specific scenario this feature is intended to resolve is idle-in-transaction backends holding on to resources while the network connection times out; I was under the impression that the specific scenario is busy-in-transaction backends continuing to produce and send data while the client has disappeared. Why does the backend ignore network errors and keep sending data? The scenario I need to deal with is this: There are multiple nodes, network-separated, participating in a cluster. One node is selected to talk to a particular postgresql instance (call this node A). A starts a transaction and grabs some locks in the course of that transaction. Then A falls off the network before committing because of a hardware or network failure. A's connection might be completely idle when this happens. The cluster liveness machinery notices that A is dead and selects a new node to talk to postgresql (call this node B). B resumes the work that A was doing prior to failure. B has to wait for any locks held by A to be released before it can make any progress. Without some sort of tunable timeout, it could take a very long time (2+ hours by default on Linux) before A's connection finally times out and releases the locks. -O ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 12:05:45 +1000 Neil Conway [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: statement_timeout is not a solution if many processes are waiting the resource. Why not? Imagine a process locked some rows to update and process codes like that ; -- Sample Client Codes here : 1. Start a Transaction 2. Set statement_timeout to 1 or any value.. 3. Update the rows * after update is completed the connection was lost and now commit keyword couldnt be sent 4. send commit to postgresql Above, because update is completed the statement_timeout is not effected anymore to cancel query..And others processes that waits same resources / rows are waiting now... I think the only problem with using statement_timeout for this purpose is that the client connection might die during a long-running transaction at a point when no statement is currently executing. Tom's suggested transaction_timeout would be a reasonable way to fix this. Adnan, if you think this is such a significant problem (I can't say that I agree), I'd encourage you to submit a patch. Ok Neil, a transaction_timeout parameters solve this, but this is worst case.. some ppl uses MSADO conneciton component and ADO conneciton has an attributes that send start transaction after a commit or sends start transaction after a rollback so, evertime has a transaction on conneciton / session.. Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Feature freeze date for 8.1
On Sun, 1 May 2005 23:08:39 -0500 Jaime Casanova [EMAIL PROTECTED] wrote: On 5/1/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: On Sun, 1 May 2005 14:35:37 -0500 Bruno Wolff III [EMAIL PROTECTED] wrote: On Sun, May 01, 2005 at 19:57:37 +0300, [EMAIL PROTECTED] wrote: Listen Tom, write a client software that releases the resources / locks that was hold before client power is down or client connection was lost. If Postgres can tell the connection has been lost then it should roll back the connection. Yes, but, Can PostgreSQL know which connection is lost or live or dead ? The problem is that you can't always tell if a connection has been lost. All you can do is timeout, either when TCP times out or some other timeout (such as a statment timeout) that you set. You are right, a timeout parameter must be used for that on the backend. a client application never find the previous instance before it crashed. However more than one connection was able to be established to PostgreSQL backend.. Statement_timeout is just a escape mechanism for active transaction. Imagine; you've started a process to update the rows in a table then your PC power was down but you have not sent commit or rollback yet..What will happen now If you send the update outside a transaction and... Option 1) ...the client crashes then the update will commit, i think. If you don't want that send the update inside a begin/commit block. Option 2) ...the server crashes the update will rollback. If you send the update inside a transaction and... Option 1) ...the client crashes then the update will rollback. Option 2) ...the server crashes the update will rollback. Actually, i can't see what's the problem. :) No, process waits until it is completed.. Adnan DURSUN ASRIN Bili?im Hiz.Ltd. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] Using LDAP for authorization
Hi! I want to pull authorization information from LDAP for my PostgreSQL database. I use kerberos for authentication, so I do *not* want to use LDAP for authentication. I want to pull group membership from LDAP and match it to the kerberos-provisded usernames. Preferrably the users should be auto-created if needed, but the groups are created by the admin (users shuold only be autocreated if they are members of these groups, of course). I see two ways of doing this: 1) Have an external process that syncs database users and groups to the LDAP directory. I have no need for instant updates (if a new user is added, it's Ok if it takes an hour or so before he can log in to the database). Does somebody know of a tool that does this already? 2) Somehow have this functionality in the backend authorization code - native support for LDAP groups. Those who have digged around those parts of the code, is this something that seems reasonable to do? Is it something we'd want in the backend at all? //Magnus ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 00:25:33 -0400 Tom Lane [EMAIL PROTECTED] wrote: Jaime Casanova [EMAIL PROTECTED] writes: Actually, i can't see what's the problem. :) I think the issue is how long does it take for the rollback to happen? so I'll beat up on the database people to override 'em instead. Perhaps the database people don't know the issues and can be browbeaten more easily. Never, How can ppl change TCP timeouts because PostgreSQL can run on different OS. I dont want to browbeat you. All thing i want is to provide a good mechanism to slove this problem on PostgreSQL like others databases etc. Oracle, MSSQL, PervasiveSQL and so... Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] custom guc vars
Andrew Dunstan wrote: Is there a readme somewhere on how modules are supposed to use custom GUC variables? If there is I have missed it. cheers I don't think there is but here's an attempt. The set of variables that PostgreSQL will recognize is well defined and an attempt to extend this set will yield an error, hence the need for a separate namespace. The custom_variable_classes will introduce new namespaces where arbitrary configuration parameters can be added. In addition, this new concept also defines a way to check the consistency of those variables so that misspelled or non-existent variables can be detected by the designated module once it is loaded. Since modules can be loaded on demand long after the parsing of the configuration settings, some additional complexity exists. Initially PostgreSQL will allow all settings and treat everything as PGC_USERSET and placeholder variables. It's not until a module is loaded and define those variables that their placeholder status is promoted to a valid variable. Example: Here's an excerpt from the initialization code of PL/Java DefineCustomIntVariable( pljava.statement_cache_size, Size of the prepared statement MRU cache, NULL, statementCacheSize, 0, 512, PGC_USERSET, NULL, NULL); DefineCustomBoolVariable( pljava.release_lingering_savepoints, If true, lingering savepoints will be released on function exit. If false, the will be rolled back, NULL, pljavaReleaseLingeringSavepoints, PGC_USERSET, NULL, NULL); EmitWarningsOnPlaceholders(pljava); The first call defines pljava.statement_cache_size as an integer variable ranging from 0 to 512 (not possible in 8.0.0 and 8.0.1 since there was a bug in the definition, the signature of the definers for int and real was corrected in 8.0.2). The second defines a boolean variable. The third call, EmitWarningsOnPlaceholders(pljava) deserves a special mention. This call will assert that no pljava.something has been defined that has not been perused by a DefineCustomxxxVariable call. So if someone for instance misspelled and used pljava.release_lingring_savepoints, there would be a warning that this custom variable is not recognized since a placeholder starting with the pljava prefix still has not been promoted. You see 3 parameters passed as NULL in the above. The first is an optional pointer to a long description. The last two are function pointers that will allow you to add a special assign hook and a show hook function. For more info on the details, I'm afraid you need to consult the source code at present. Kind regards, Thomas Hallgren ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] SPI bug.
Neil Conway wrote: My point is that since they are different types, the language itself will need to provide some mechanism for doing this type conversion _anyway_. 'int' and 'long' are used throughout the backend APIs, so I don't see the gain in only converting the SPI functions over to using int32/int64. Mainly because it's easier to do that mapping knowing that the semantics equipped with the involved types never change. There's also a performance issue. I must map a C/C++ long to a 64bit value at all times and thereby get a suboptimal solution. An API should ideally hide the internals of the underlying code so I'm not sure this is a valid reason. Well, the executor allows you to specify a 64-bit count on platforms where long is 64-bit, and a 32-bit count otherwise. Exactly. Why should a user of the SPI API be exposed to or even concerned with this at all? As an application programmer you couldn't care less. You want your app to perform equally well on all platforms without surprises. IMHO, PostgreSQL should make a decision whether the SPI functions support 32-bit or the 64-bit sizes for result sets and the API should reflect that choice. Having the maximum number of rows dependent on platform ports is a bad design. Regards, Thomas Hallgren ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 01:35:14 -0400 Tom Lane [EMAIL PROTECTED] wrote: [ itch... ] This seems to me to be conflating several distinct issues. AFAIR the points that have been raised in the thread are: #1 Defend against loss of connectivity to client #2 Defend against client sitting idle while holding locks (or just holding an open transaction and thereby preventing VACUUM cleanup) #3 Defend against client holding locks unreasonably long, even though not idle (obviously any such constraint will cause clients to fail midstream, but perhaps some DBAs will see this as the lesser of two evils) I claim that if you have a problem with #1 you ought to go discuss it with some TCP hackers: you basically want to second-guess the TCP stack's ideas about appropriate timeouts. Maybe you know what you are doing or maybe not, but it's not a database-level issue. #2 is a fair point if you need to cope with poorly-programmed clients, but I'm not seeing exactly why it has to be measured by idle time rather than total time. The known cases of this involve client code that issues a BEGIN and then just sits, so there's no difference. Ok, the client sent BEGIN and then connection was lost. Does it means that the client sits ? Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 16:07:07 +1000 Neil Conway [EMAIL PROTECTED] wrote: I claim that if you have a problem with #1 you ought to go discuss it with some TCP hackers: you basically want to second-guess the TCP stack's ideas about appropriate timeouts. Well, no -- you might want to set a different timeout for PostgreSQL connections than for other connections. Is there a way to change the socket timeout for some subset of the processes on the machine without hacking the client or server source? You might also want to set this timeout on a more granular basis (e.g. per user, per database, etc.) Implementing this via setting a socket option (provided it can be done portably) would be fine with me. You are right Neil.. A machine can make more than one connection to same server. Some of them are database connection and some only to OS conneciton. For Example : Oracle has an parameter to find and kill dead sessions.. PostgreSQL can run on different OS.I think this is solved database level rather than OS level. Best Regards, Adnan DURSUN ASRIN Bili?im Hiz.Ltd. ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 2 May 2005 10:11:40 +0200 Peter Eisentraut [EMAIL PROTECTED] wrote: I was under the impression that the specific scenario is busy-in-transaction backends continuing to produce and send data while the client has disappeared. Why does the backend ignore network errors and keep sending data? Yes, I think PostgreSQL doesnt know whether client is dead or live ? Or It knows it but, it keeps to send data.. Best Regards, Adnan DURSUN ASRIN Bili?im Hiz.Ltd. ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [pgsql-advocacy] Increased company involvement
On N, 2005-04-28 at 20:13 -0700, Josh Berkus wrote: Hannu, But I too expected the discussion to take place on pgsql-hackers, not some half-hidden mailinglist on pgfoundry. Or at least an announcement of that mailinglist to be made on pgsql-hachers. Yeah, we should announce the mailing list. Actually, I did direct e-mail a bunch of people (including you) about it and invite them to the mailing list. Probably this got lost somewhere on the way. For discussing potential features, though, I'm personally reluctant at this point to discuss major features until I and my collaborators (example, newsysviews) have a concrete proposal together. It's far too easy to get sidetracked into a discussion of minutia and politics on -hackers; by generating a complete draft spec (at least) on a small mailing list, it's a lot easier to focus on specific goals and schedules, and discussions on hackers around detailed proposals tend to be a lot more focused. Maybe :) , still not convinced though. There may be some quite fundamental things we may miss but which could be spotted by wider audience early on. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Feature freeze date for 8.1
On P, 2005-05-01 at 11:37 -0400, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: The problem, as I understand it, is that if you have a long-running query and the client process disappears, the query keeps running and holds whatever resources it may have until it finishes. There is a trivial solution for this: it's called statement_timeout. statement timeout does not solve the fundamental problem of server not seeing when a cleint connection has disappeared. I had another variant of the same problem - i had 300 client connections to the same postmaster, postmaster was configured to handle 500 simultaneous connections. Then there was a network outage for a few minutes, during which clients saw that the postmaster was not there and closed their conections and tried to make new ones. After the network came back, only 200 of them were able to reconnect, as the server never saw them leaving and just kept the 300 zombie connections. It might be interesting to think about a transaction_timeout as well, to bound the time locks can be held. But none of this has anything to do with high availability as I understand the term. It looks more like a forcing function to make your users fix poorly-written client software ;-) Im my case all ttransactions were implicit one command functon calls (select * from dbfunc()), so transaction timeout would not help. probably the only way for server to detect stale connections would be sending/receiving some kind of keepalives. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
On E, 2005-05-02 at 01:35 -0400, Tom Lane wrote: Neil Conway [EMAIL PROTECTED] writes: Tom Lane wrote: We would? Why? Please provide a motivation that justifies the considerably higher cost to make it count that way, as opposed to time-since-BEGIN. The specific scenario this feature is intended to resolve is idle-in-transaction backends holding on to resources while the network connection times out; it isn't intended to implement I never want to run a transaction that takes more than X seconds to execute. [ itch... ] This seems to me to be conflating several distinct issues. AFAIR the points that have been raised in the thread are: #1 Defend against loss of connectivity to client #2 Defend against client sitting idle while holding locks (or just holding an open transaction and thereby preventing VACUUM cleanup) #3 Defend against client holding locks unreasonably long, even though not idle (obviously any such constraint will cause clients to fail midstream, but perhaps some DBAs will see this as the lesser of two evils) I claim that if you have a problem with #1 you ought to go discuss it with some TCP hackers: you basically want to second-guess the TCP stack's ideas about appropriate timeouts. Maybe you know what you are doing or maybe not, but it's not a database-level issue. Well, I've had problems with clients which resolve DB timeouts by closing the current connection and establish a new one. If it is actual DB timeout, then it all is ok, the server soon notices that the client connection is closed and kills itself. Problems happen when the timeout is caused by actual network problems - when i have 300 clients (server's max_connections=500) which try to reconnect after network outage, only 200 of them can do so as the server is holding to 300 old connections. In my case this has nothing to do with locks or transactions. It would be nice if I coud st up some timeut using keepalives (like ssh- s ProtocoKeepalives) and use similar timeouts on client and server. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 13:59:21 +0300 Hannu Krosing [EMAIL PROTECTED] wrote: On E, 2005-05-02 at 01:35 -0400, Tom Lane wrote: Well, I've had problems with clients which resolve DB timeouts by closing the current connection and establish a new one. If it is actual DB timeout, then it all is ok, the server soon notices that the client connection is closed and kills itself. Problems happen when the timeout is caused by actual network problems - when i have 300 clients (server's max_connections=500) which try to reconnect after network outage, only 200 of them can do so as the server is holding to 300 old connections. In my case this has nothing to do with locks or transactions. Yes, but if you would have locks then same problem occours.. It would be nice if I coud st up some timeut using keepalives (like ssh-s ProtocoKeepalives) and use similar timeouts on client and server. Diffrent ppl use diffrent OS. Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
On Sun, 01 May 2005 22:23:19 +0300 Hannu Krosing [EMAIL PROTECTED] wrote: On P, 2005-05-01 at 11:37 -0400, Tom Lane wrote: Im my case all ttransactions were implicit one command functon calls (select * from dbfunc()), so transaction timeout would not help. probably the only way for server to detect stale connections would be sending/receiving some kind of keepalives. ...and then clear stale connection. This would be nice feature for PostgreSQL Best Regards, Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Feature freeze date for 8.1
Hi, -- Dennis Bjorklund [EMAIL PROTECTED] wrote: The tcp hackers have provided an api for clients to set these values per socket (setsockopt with TCP_KEEPIDLE and similar (in linux at least)). you can use SO_KEEPALIVE: [...] SO_KEEPALIVE enables the periodic transmission of messages on a connected socket. Should the connected party fail to respond to these messages, the connection is con- sidered broken and processes using the socket are notified via a SIGPIPE signal when attempting to send data. For me it seems to be a good idea to support this. For the applications (server, client) it is transparent, but notices a cutted or broken network connection faster then before... But even with this you only realise that the connection is gone when sending something, AFAIK. Ciao Alvar -- Alvar C.H. Freude -- http://alvar.a-blast.org/ http://odem.org/ http://www.assoziations-blaster.de/info/Hommingberger-Gepardenforelle.html http://www.assoziations-blaster.de/ pgpYyykmpgrLF.pgp Description: PGP signature
Re: [HACKERS] How to make lazy VACUUM of one table run in several
On T, 2005-04-26 at 17:54 -0400, Tom Lane wrote: Hannu Krosing [EMAIL PROTECTED] writes: Could I avoid having a transaction at all? Not really; too much of the database access infrastructure is tied to transaction stuff ... even facilities as basic as memory management. As VACUUM is not transactional in the sense that it does not change anything visible to users ever, can't be undone by rollback, etc... , could it be possible to create enough transaction-like environment for it to really run outside of transactions. Perhaps just advancing oldestXmin at certain intervals ? I wonder whether you could avoid advertising the VACUUM's XID in PGPROC. Not sure that this can work, but it would be a lot simpler than stopping and starting transactions ... What dreadful things will happed if I just set the xid and xmin of current transaction (proc-xid, -xmin) to some value below FirstNormalTransactionId, so that TransactionIdIsNormal(xid) will make GetOldestXmin ignore my transaction ? Will this for example confuse WAL or cause something else equally grim to happen ? An alternative would be to have some special ignore-me flag in PGPROC. A more general solution to the problem VACUUM does not clean dead tuples fast enough due to an old transaction problem is keeping the OldestXmin for each table separately as a list of table OIDs in each PGPROC. This would be automatically extandable to long COPY, or in fact any single SQL statement running in its implicit transaction by examining the query plan and reserving all tables touched by the query and its dependencies. And this could be done as a user-level command for explicit transactions (LOCK TABLE my_table, othertable, yetanothertable FOR/FROM VACUUM) which would tell postgres that the current query will need these (and only these) tables. If LOCK FOR VACUUM is done inside a transaction, then touching any other table should be an error. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 02 May 2005 13:32:18 +0200 Alvar Freude [EMAIL PROTECTED] wrote: Hi, -- Dennis Bjorklund [EMAIL PROTECTED] wrote: The tcp hackers have provided an api for clients to set these values per socket (setsockopt with TCP_KEEPIDLE and similar (in linux at least)). you can use SO_KEEPALIVE: [...] SO_KEEPALIVE enables the periodic transmission of messages on a connected socket. Should the connected party fail to respond to these messages, the connection is con- sidered broken and processes using the socket are notified via a SIGPIPE signal when attempting to send data. For me it seems to be a good idea to support this. For the applications (server, client) it is transparent, but notices a cutted or broken network connection faster then before... But even with this you only realise that the connection is gone when sending something, AFAIK. So this means, If client does never try to send data the resources would be going to be held. I think it is not a good solution to find zombie / dead connection and clear them.. Best Regards, Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Feature freeze date for 8.1
The world rejoiced as matthew@zeut.net (Matthew T. O'Connor) wrote:
Marc G. Fournier wrote:
On Fri, 29 Apr 2005, Christopher Browne wrote:
Some reasonable approximations might include:
- How much disk I/O was recorded in the last 60 seconds?
- How many application transactions (e.g. - invoices or such) were
issued in the last 60 seconds (monitoring a sequence could be
good enough).
Some way of doing a 'partial vacuum' would be nice ... where a
VACUUM could stop after it processed those '10 elderly tuples' and
on the next pass, resume from that point instead of starting from
the beginning again ...
That is sorta what the vacuum delay settings accomplish.
What they do is orthogonal to that.
Vacuum delay prevents vacuum I/O from taking over the I/O bus.
Unfortunately, if you have a table with a very large number of _live_
tuples, there is no way to skip over those and only concentrate on the
dead ones.
In that scenario vacuum delay leads to the vacuum on the table
running for a Very, Very Long Time, because it sits there delaying a
lot as it walks thru pages it never modifies. The one good news is
that, for any pages where no tuples are touched, the indices are also
left untouched.
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','gmail.com').
http://linuxdatabases.info/info/slony.html
The Board views the endemic use of PowerPoint briefing slides instead
of technical papers as an illustration of the problematic methods of
technical communication at NASA. -- Official report on the Columbia
shuttle disaster.
---(end of broadcast)---
TIP 8: explain analyze is your friend
[HACKERS] Python
Greets, Well, I've been solo'ing it for over a year now, and it's definitely time to make my presence known to the community and invite others to participate as I can't keep putting it off as I have been. There has always been something else that I wanted to finish up before making any announcements, so I kept putting it off and putting it off. Time to stop that. =) For those of you who are not familiar with what I've been doing, I have been working on a Python procedural language extension for PostgreSQL and, furthermore, making attempts to formalize a specification of Python interfaces to PostgreSQL elements(making a draft doc [relatively ;] soon). It is a PgFoundry project, and I do have some information up on the project site[1]. There are other subprojects as well as the PL, such as layout and exceptions, which should be of interest to other developers working on PostgresSQL related Python modules. The backend part--the procedural language--of the project offers a substantial number of improvements over the plpython that is currently in core[2]. The design, in general, has a significant contrast to the one in core, as it focuses on providing 'interfaces to' rather than 'conversions for'. While this appears to cause a larger code base, it provides a level of flexibility that makes adaptations and improvements trivial in many or most cases. For instance, I already have the necessary infrastructure to support OUT parameters in a way that is _very_ natural(tho, support for 8.1dev is not there yet; busy with other stuff). It, currently, is *not* my hope to eventually put this into core; rather, I think I would like to give a try at setting a good example for extension development. PgFoundry is a good resource, and I think it should be exercised, as it makes a great home for projects like this(gborg is pretty good as well! ;).(Yes, I'm +1 for, eventually, pushing things out of core. =) While this is not so much of a formal announcement of the initiative/project(a formal one will eventually come to -announce and c.l.py), it is an invitation to those interested parties on -hackers to join the python-general list[4], which, in addition to up and coming development releases, I hope will lead to the project becoming more active, as opposed to me just silently hacking away. In addition, for those who might want it, I'm open to the idea of giving admin/commit access--especially to well known devs and members of core, so please don't hesitate to ask. If you are concerned about the licensing and the ambiguity of the primary license, don't be. It is--at least meant to be--a gift license, so I have no problems with others using the works under the BSD or MIT licenses; this is explicitly stated/granted here[3], and a reference to the authorization to use those substitute terms will eventually make its way into every instrument-stamped source file. If you want to play with what I've got so far, see the quick start[5]. It gives you some neat code, so you can play around in the interactive console in a single-user backend to get a better feel of things. (It should work without much/any trouble, at least on FreeBSD 5.3. If you do have trouble don't hesitate to contact me by e-mail or on freenode where I go by the nick 'jwp'.) [1]http://python.projects.postgresql.org [2]http://python.projects.postgresql.org/project/be.html [3]http://python.projects.postgresql.org/license.html [4]http://lists.pgfoundry.org/mailman/listinfo/python-general [5]http://python.projects.postgresql.org/quick.html -- Regards, James William Pye ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [pgsql-advocacy] Increased company involvement
Martha Stewart called it a Good Thing when [EMAIL PROTECTED] (Jim C. Nasby) wrote: Anyone interested in pooling funds for features should take a look at http://people.freebsd.org/~phk/funding.html, which is about a FreeBSD developer who offered to work full-time on developing some specific features should enough people donate. Also worthy of mention is http://www.freebsdfoundation.org/. I think that for certain key features there's probably a lot of people who would fork over between $100 and $1000 towards getting a feature completed. Improved replication might be a good example. Table partitioning would absolutely be an example. If there was a means for these people to donate money towards work being done on some feature, it's very likely that large chunks of development time could be paid for just from smaller shops, let alone places that can afford to toss $20k towards the development of something. Note that money isn't necessarily the most useful thing. For Slony-I, I can think of three places where specific contributions of specific efforts could be really valuable, and potentially free up other peoples' time to do heavier lifting. 1. Fully scripted test cases. There are about a dozen scripts that exist now that test for a number of known conditions, either generally checking to see if replication is functioning, or trying to exercise particular bits of functionaly, or verifying that certain bugs are either present or absent. There are some tests I'd like to set up but never get time to script up. Improving the tests that can be _trivially_ run would be a big help, and would have a general positive effect on reliability. 2. Documentation I tried to get someone to write up how to do PG upgrades using Slony-I; wound up essentially writing it myself. There is plenty of room for How I Did Foo With ... with Things Other Haven't Tried yet. For instance, I haven't tried Slony-I on cases involving inheritance; a test script and a document on this would be super. 3. Actually requesting features There is a small queue of outside-requested features for Slony-I, but the queue is pretty small. The vast majority of things queued have come from discussions between about 4 people, all of whom are writing code for the project. I daresay I am being totally myopic here, thinking only of my project. There lie my priorities, tough luck :-)! I'd hazard the guess that would-be contributors might be better off contributing relatively small things like improving documentation or assisting by providing usefully detailed test cases than they would be in contributing small sums of money. It is _really_ not obvious how specks of money can be usefully put together to get bigger features to happen. I think a REALLY valuable thing would be if we could get another person that was pretty expert with the query optimizer. The only way to do that is to get someone to spend a year fighting with it. Throwing a thousand dollars at someone here and there isn't likely to direct them towards that. -- output = (cbbrowne @ gmail.com) http://linuxfinances.info/info/rdbms.html programmer, n: A red eyed, mumbling mammal capable of conversing with inanimate monsters. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Feature freeze date for 8.1
Centuries ago, Nostradamus foresaw when [EMAIL PROTECTED] would write: We sometime discuss here for geographic system datatypes and feature. First, a database must have real database features, not extreme features. Oh, but it would be so much better if we could call the next version PostgreSQL 8.1 Extreme Edition because it was so much more extreme. That way we could get skateboarders to do promotional appearances. And Vin Diesel for a XXX appearance... :-) -- let name=cbbrowne and tld=gmail.com in String.concat @ [name;tld];; http://cbbrowne.com/info/slony.html Signs of a Klingon Programmer - 4. You cannot really appreciate Dilbert unless you've read it in the original Klingon. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Feature freeze date for 8.1
After takin a swig o' Arrakan spice grog, [EMAIL PROTECTED] belched out: On Sun, 1 May 2005 14:35:37 -0500 Bruno Wolff III [EMAIL PROTECTED] wrote: On Sun, May 01, 2005 at 19:57:37 +0300, [EMAIL PROTECTED] wrote: Listen Tom, write a client software that releases the resources / locks that was hold before client power is down or client connection was lost. If Postgres can tell the connection has been lost then it should roll back the connection. Yes, but, Can PostgreSQL know which connection is lost or live or dead ? Certainly, with the magic connection analysis protocol. Or not... In order to that sort of analysis, you need to have some form of heartbeat monitor on the connection, thereby requiring extra connections. It could make sense to attach that kind of extra apparatus to a connection pool manager like pgpool, but probably not directly to PostgreSQL itself. You might want to look into pgpool; that is something that many people interested in enterprise usage of PostgreSQL are looking into... -- (reverse (concatenate 'string moc.liamg @ enworbbc)) http://linuxdatabases.info/info/slony.html A army's effectiveness depends on its size, training, experience and morale, and morale is worth more than all the other factors combined. -- Napoleon Bonaparte ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Feature freeze date for 8.1
Peter Eisentraut [EMAIL PROTECTED] writes: I was under the impression that the specific scenario is busy-in-transaction backends continuing to produce and send data while the client has disappeared. Why does the backend ignore network errors and keep sending data? There are a couple of reasons behind that: 1. In terms of resources held against the rest of the database, a query doing SELECT is hardly likely to be your worst problem. Queries doing UPDATE, VACUUM, etc are holding stronger locks and probably chewing at least as many resources; but they aren't doing any client I/O (until the command tag at the very end) and so are not going to detect client connection loss significantly earlier than they do now anyway. 2. Because of the lack of output, the behavior of non-SELECT queries is that they complete before the backend notices client connection loss. This is nice because it makes it simpler to reason about what will happen. Currently the backend guarantees the same for SELECT queries, which is also nice, since we have plenty of SELECT queries with side-effects (consider user function invocation). Do we really want to give that up? 3. If we error out on send() failure then we have turned a probable failure into certain failure, because we will have lost message-boundary sync with the client --- in other words the error might as well be a FATAL one. This seems like it might be overkill. 4. Erroring out in the low-level routines is trickier than it looks; in particular, after you elog() your complaint, elog.c is going to come right back to you with an error message to send to the client. Not having this turn into an infinite recursion is a bit ticklish. Making sure it stays working is even trickier, considering what a seldom-tested code path it's going to be. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] How to make lazy VACUUM of one table run in several transactions ?
Hannu Krosing [EMAIL PROTECTED] writes: A more general solution to the problem VACUUM does not clean dead tuples fast enough due to an old transaction problem is keeping the OldestXmin for each table separately as a list of table OIDs in each PGPROC. This would be automatically extandable to long COPY, or in fact any single SQL statement running in its implicit transaction by examining the query plan and reserving all tables touched by the query and its dependencies. This is completely unworkable, since it amounts to assuming you know at the start of a serializable transaction which tables it will touch. In point of fact you can't even know that for the current query let alone future ones --- consider user-defined functions. (Not to mention that we can't expect to fit that much info into a fixed amount of shared memory.) regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Using LDAP for authorization
* Magnus Hagander ([EMAIL PROTECTED]) wrote: I want to pull authorization information from LDAP for my PostgreSQL database. I use kerberos for authentication, so I do *not* want to use LDAP for authentication. I want to pull group membership from LDAP and match it to the kerberos-provisded usernames. Preferrably the users should be auto-created if needed, but the groups are created by the admin (users shuold only be autocreated if they are members of these groups, of course). Kind of a neat idea, but... I see two ways of doing this: 1) Have an external process that syncs database users and groups to the LDAP directory. I have no need for instant updates (if a new user is added, it's Ok if it takes an hour or so before he can log in to the database). Does somebody know of a tool that does this already? I don't know of a tool that does it already but it doesn't sound too difficult to do, honestly. 2) Somehow have this functionality in the backend authorization code - native support for LDAP groups. Those who have digged around those parts of the code, is this something that seems reasonable to do? Is it something we'd want in the backend at all? The user/group system is having something of an overhaul at the moment. Basically I'm working on implementing roles. I think the problem at the moment is that Postgres has no system in place for checking user or group information externally/dynamically during operation really. There's some support in the authentication code for checking external sources (Kerberos, PAM, etc) for users. Once you're connected and authenticated though, all of the user information, group information, access controls, etc, are done using Postgres' 'system cache' and the system tables/indexes. Changing that would require quite a bit of effort... Stephen signature.asc Description: Digital signature
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
-Original Message- From: Bruce Momjian [mailto:[EMAIL PROTECTED] Sent: Saturday, April 30, 2005 12:04 PM To: PostgreSQL advocacy Cc: Kris Jurka; Andrew Dunstan; PostgreSQL-development Subject: Re: [pgsql-advocacy] [HACKERS] Increased company involvement [...] The thing that limits centralization is that it is critical that any individual or company feel free to join the community efforts. When centralization happens, there is often an _in_ and and _out_ group that is very bad for encouraging new members. [...] We don't want core to steer development anymore than we want a centralized group to do that, because if we did, the next company that comes along and wants to enhance PostgreSQL or offer technical support services will feel they have to get approval/buy-in from the _in_ group, and that isn't a productive setup. The fact that new companies getting involved can't find a central authority is a _good_ thing, if you think about it. It means that we have succeeded in building a community that allows people to join and feel a part right away, and they don't have to buy-in or play politics to do it. Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? Aren't they the people that you need buy-in from to really contribute to Postgres? The reason I make this point is because I know what a democratic development community really looks like, and the Boost community is one such example. That truly *is* democratic, because decisions are made as a group, and no fixed subset of members has an overriding veto. The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. __ David B. Held Software Engineer/Array Services Group 200 14th Ave. East, Sartell, MN 56377 320.534.3637 320.253.7800 800.752.8129 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pg_locks needs a facelift
In the earlier thread there was talk of separate views for system and user locks, but on reflection I think that's the wrong approach; principally because it will be impossible to get exactly-simultaneous snapshots of the system and user lock states if there are two views involved. And that's something you tend to want when studying lock behavior ;-). So I think we have to maintain the current arrangement of one view, and add enough columns to it to handle all the requirements. This seems perfectly ok...as long as there is 1:1 correspondence between locktag and lock for all present and future types of locks. I'd like to point out though that when querying for user locks it's kind of nice not to wade through transaction locks, etc. One nice things about the generic types (int4) is that they can be easily casted...if a column is displaying an xid that is not really an xid (user lock block offset), this can be annoying if you want to do some post query processing on the field, like bit shift it back into a 64 bit variable...especially since a dump/restore will drop all casts between two system provided columns. What about having a view with all the generic columns and one specialized view (pg_locks) for backwards compatibility? Merlin ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] SPI bug.
Thomas Hallgren [EMAIL PROTECTED] writes: Exactly. Why should a user of the SPI API be exposed to or even concerned with this at all? As an application programmer you couldn't care less. You want your app to perform equally well on all platforms without surprises. IMHO, PostgreSQL should make a decision whether the SPI functions support 32-bit or the 64-bit sizes for result sets and the API should reflect that choice. Having the maximum number of rows dependent on platform ports is a bad design. The fact that 64-bit platforms can tackle bigger problems than 32-bit ones is not a bug to be worked around, and so I don't see any problem with the use of long for tuple counts. Furthermore, we have never promised ABI-level compatibility across versions inside the backend, and we are quite unlikely to make such a promise in the foreseeable future. (Most of the time you are lucky if you get source-level compatibility ;-).) So I can't get excited about avoiding platform dependency in this particular tiny aspect of the API. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Feature freeze date for 8.1
Oliver Jowett [EMAIL PROTECTED] writes: The scenario I need to deal with is this: There are multiple nodes, network-separated, participating in a cluster. One node is selected to talk to a particular postgresql instance (call this node A). A starts a transaction and grabs some locks in the course of that transaction. Then A falls off the network before committing because of a hardware or network failure. A's connection might be completely idle when this happens. The cluster liveness machinery notices that A is dead and selects a new node to talk to postgresql (call this node B). B resumes the work that A was doing prior to failure. B has to wait for any locks held by A to be released before it can make any progress. Without some sort of tunable timeout, it could take a very long time (2+ hours by default on Linux) before A's connection finally times out and releases the locks. Wouldn't it be reasonable to expect the cluster liveness machinery to notify the database server's kernel that connections to A are now dead? I find it really unconvincing to suppose that the above problem should be solved at the database level. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] SPI bug.
Tom Lane wrote: Thomas Hallgren [EMAIL PROTECTED] writes: Exactly. Why should a user of the SPI API be exposed to or even concerned with this at all? As an application programmer you couldn't care less. You want your app to perform equally well on all platforms without surprises. IMHO, PostgreSQL should make a decision whether the SPI functions support 32-bit or the 64-bit sizes for result sets and the API should reflect that choice. Having the maximum number of rows dependent on platform ports is a bad design. The fact that 64-bit platforms can tackle bigger problems than 32-bit ones is not a bug to be worked around, and so I don't see any problem with the use of long for tuple counts. I'm not concerned with the use of 32 or 64 bits. I would be equally happy with both. What I am concerned is that the problem that started this SPI bug was caused by the differences in how platforms handle the int and long types. Instead of rectifying this problem once and for all, the type was just changed to a long. Furthermore, we have never promised ABI-level compatibility across versions inside the backend, and we are quite unlikely to make such a promise in the foreseeable future. I know that no promises has been made but PostgreSQL is improved every day and this would be a very easy promise to make. (Most of the time you are lucky if you get source-level compatibility ;-).) So I can't get excited about avoiding platform dependency in this particular tiny aspect of the API. Maybe I've misunderstood the objectives behind the SPI layer altogether but since it's well documented and seems to be the public interface of the backend that extensions are supposed to use, I think it would be an excellent idea to make that interface as stable and platform independent as possible. I can't really see the disadvantages. The use of int, long, and long long is often a source of bugs (as with this one) and many recommend that you avoid them when possible. The definition of int is meant to be a datatype used for storing integers where size of that datatype equals natural size of processor. The long is defined as 'at least as big as int' and the 'long long' is 'bigger than long'. I wonder what that makes 'long long' on a platform where the int is 64 bits. 128 bits? Also, the interpretation of the definition vary between compiler vendors. On Windows Itanium, the int is 32 bit. On Unix it's 64. It's a mess... The 1998 revision of C declares the following types for a good reason: int8_t , int16_t, int32_t int64_t, uint8_t, uint16_t, uint32_t, uint64_t. Why not use them unless you have very specific requirements? And why not *always* use them in a public interface like the SPI? Regards, Thomas Hallgren ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] pg_locks needs a facelift
Merlin Moncure [EMAIL PROTECTED] writes: So I think we have to maintain the current arrangement of one view, and add enough columns to it to handle all the requirements. This seems perfectly ok...as long as there is 1:1 correspondence between locktag and lock for all present and future types of locks. I'd like to point out though that when querying for user locks it's kind of nice not to wade through transaction locks, etc. Well, sure, but that's what SELECT ... WHERE is for ;-) One nice things about the generic types (int4) is that they can be easily casted...if a column is displaying an xid that is not really an xid (user lock block offset), this can be annoying if you want to do some post query processing on the field, like bit shift it back into a 64 bit variable...especially since a dump/restore will drop all casts between two system provided columns. The proposal I made was to display all fields of a USER lock as either OID or int2, so you can certainly cast the OIDs to int4 if you want to do some kind of arithmetic on them. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 2 May 2005, Hannu Krosing wrote: Well, I've had problems with clients which resolve DB timeouts by closing the current connection and establish a new one. If it is actual DB timeout, then it all is ok, the server soon notices that the client connection is closed and kills itself. Problems happen when the timeout is caused by actual network problems - when i have 300 clients (server's max_connections=500) which try to reconnect after network outage, only 200 of them can do so as the server is holding to 300 old connections. In my case this has nothing to do with locks or transactions. It would be nice if I coud st up some timeut using keepalives (like ssh- s ProtocoKeepalives) and use similar timeouts on client and server. FWIW, I've been bitten by this problem twice with other applications. 1. We had a DB2 database with clients running in other computers in the network. A faulty switch caused random network outages. If the connection timed out and the client was unable to send it's request to the server, the client would notice that the connection was down, and open a new one. But the server never noticed that the connection was dead. Eventually, the maximum number of connections was reached, and the administrator had to kill all the connections manually. 2. We had a custom client-server application using TCP across a network. There was stateful firewall between the server and the clients that dropped the connection at night when there was no activity. After a couple of days, the server reached the maximum number of threads on the platform and stopped accepting new connections. In case 1, the switch was fixed. If another switch fails, the same will happen again. In case 2, we added an application-level heartbeat that sends a dummy message from server to client every 10 minutes. TCP keep-alive with a small interval would have saved the day in both cases. Unfortunately the default interval must be = 2 hours, according to RFC1122. On most platforms, including Windows and Linux, the TCP keep-alive interval can't be set on a per-connection basis. The ideal solution would be to modify the operating system to support it. What we can do in PostgreSQL is to introduce an application-level heartbeat. A simple Hello world message sent from server to client that the client would ignore would do the trick. - Heikki ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Jurka, One thing that definitely would be nice would be to be able to combine funds from various sponsors for various features. Alone a company can't spring for it, but by pooling resources it could get done. This is a lot tougher to coordinate and unless there is a complete spec in place different sponsors will pull in different directions. Other bounty type schemes don't seem to produce results, largely from a lack of cash. (Here's $500 for two weeks of work). Actually, I talked to Opensoucexperts.com about this ages ago and they set up an online bounty system for OSS projects in general. I know that other players in the OSS space have talked about similar things; also companies like SRA and CMD are willing to act as development $$$ funnels for multi-party projects. The problem is organization. Who decides who gets what money? What about features that are paid for and worked on and not accepted into the community codebase? This was something I hoped the PostgreSQL Foundation http://thepostgresqlfoundation.org/ would step in and do, but we seem much more focused on advocacy efforts rather than developemnt ones. Unfortunately, pooling funds for development is not something a non-profit can realistically do in the US without a whole lot of legal/tax help to navigate US law.Here NPOs are strictly defined as non-commercial. It might make sense to set up an NPO in another country, such as Australia, where the regulations on such things are much more liberal. More importantly, the Foundation is *still* waiting on its NPO paperwork from the IRS, and I really don't want to do any major fundraising while there's still the possibility we could be denied. Well the backup should come up in a couple of weeks. I know that the new pgFoundry is being worked on right now. Josh would have a better idea. We ran into a problem installing GForge. I don't know if Tom had time to work on it over the weekend; if not I'll be tackling it tonight. -- Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] How to make lazy VACUUM of one table run in several
On E, 2005-05-02 at 10:38 -0400, Tom Lane wrote: But what about my question about just changing xid in PGPROC ? Hannu Krosing [EMAIL PROTECTED] writes: A more general solution to the problem VACUUM does not clean dead tuples fast enough due to an old transaction problem is keeping the OldestXmin for each table separately as a list of table OIDs in each PGPROC. This would be automatically extandable to long COPY, or in fact any single SQL statement running in its implicit transaction by examining the query plan and reserving all tables touched by the query and its dependencies. This is completely unworkable, since it amounts to assuming you know at the start of a serializable transaction which tables it will touch. I meant it for simple things, like VACUUM and maybe also simple COPY (tables ref'd by foreign keys can be found from pg_depend), but yes we can't automatically know it in general case. That's why I proposed it to be an error for any transaction with such list to touch any tables not in the list, so the assumptions of what tables are tested are simple to test. In point of fact you can't even know that for the current query let alone future ones --- consider user-defined functions. (Not to mention that we can't expect to fit that much info into a fixed amount of shared memory.) what should be in shared memory, is one list of ints per backend (oids of tables for current transaction), mostly just one :) The general case should be not to have such list at all which assumes that any table could be touched. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] ARCHIVE TABLES (was: possible TODO: read-only tables, select from indexes only.)
Third repeat post. Sorry if previous ones suddenly pop up too. ARCHIVE TABLES this is a further development ofthe read-only tables proposal, WHich answers Tom's question - 'how such a tables comes to exists' and inspired by someones (can't find the mail, sorry) suggestion to think of them as archive tables, not R/O tables. So ARCHIVE TABLE is is a table with the following properties 1) The ONLY operations allowed on it are APPEND (i.e. INSERT or COPY) and SELECT. Forbidden ops are UPDATE and DELETE. TRUNCATE should probably allowed too. 2) data in heap is dense, each new APPEND adds a set of one or more tuples with continuous tids, all bigger than existing any existing tid. no system colums are stored in each tuple, WITH OIDS is not supported for archive tables. 3) tuple visibility is determined by tid of last tuple of last successfully committed APPEND . Any tuple with tid LastValidTid is not visible. The LastValidTid is stored for each archive table. 4) 0nly one session can APPEND to an archive table at any one time. 5) if the transaction doing the insert is rolled back, the table is locked for further APPENDS until indexes are cleaned up (by VACUUM or REINDEX) and the end of heap is truncated to its place before the failed transaction. An altervative behaviour is to do the cleanup immediately on rollback, but even this has to lock the table until its done, to be crash-proof. 6) Vacuum on an archive table should - examine only tuples with tid LastValidTid - clean up their index entries - modyfi last valid page and truncate table to its old length corresponding to LastValidTid (4, 5 6 are needed for 3 to work reliably in case of rollbacked transactions.) 7) everything else (constraints, triggers, indexes, rules, statistics) should be the same as for normal tables. Even inheriting an ARCHIVE table from ordinary table should be allowed. This kind of setup allows the following features - index-only scans for cases where all columns needed are in index. visibility can be determined from tid without consulting the heap. - smaller table sizes due to not storing visibility info with each tuple which are often desirable for BusinessIntelligence/DataWarehousing databases and other systems with huge fact tables. I don't think that Tom's concern about pervasiveness of TupleHeader is unsurmountable. I hope that the only stupid hopeful grin thing needing to change is visibility checks when fetching the tuples from heap or index, from that point on it should be possible to handle them as if the data is coming from a view. Also I hope that index structure does not have to change at all, only a new access methods should be added, namely * archive-indexscan (replaces ordinary index-scan) * archive-index-only-scan (new) * archive-seqscan (replaces ordinary seqscan) and planner/executor must be teached to use these. -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Dave Held wrote: Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? Aren't they the people that you need buy-in from to really contribute to Postgres? The reason I make this point is because I know what a democratic development community really looks like, and the Boost community is one such example. That truly *is* democratic, because decisions are made as a group, and no fixed subset of members has an overriding veto. The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. Really? You have a different perspective than I see. I have seen patches be accepted that had no core buy-in. We accept patches based on group feedback, not some closed approval process. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Bruce Momjian wrote: Dave Held wrote: Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? Aren't they the people that you need buy-in from to really contribute to Postgres? The reason I make this point is because I know what a democratic development community really looks like, and the Boost community is one such example. That truly *is* democratic, because decisions are made as a group, and no fixed subset of members has an overriding veto. The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. Really? You have a different perspective than I see. I have seen patches be accepted that had no core buy-in. We accept patches based on group feedback, not some closed approval process. Let me also ask for you to provide an example of the behavior you describe. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_locks needs a facelift
Tom Lane wrote: This seems perfectly ok...as long as there is 1:1 correspondence between locktag and lock for all present and future types of locks. I'd like to point out though that when querying for user locks it's kind of nice not to wade through transaction locks, etc. Well, sure, but that's what SELECT ... WHERE is for ;-) yeah, I misread your earlier post...being able to filter on lock type (or not) is an ideal solution. So, pg_locks it is. I'd also like to make one more comment: This still leaves us with the issue of what to do with user locks. I am inclined to display them as if they were OBJECT locks, ie, fill the database, classid, objid, and objsubid columns. An alternative that would also expose all the info is to display them as if they were TUPLE locks. Or we could add still more columns, but I'm not real enthused about that idea. I don't like the idea of listing user locks with 'tuple' locks for no other reason than this might confuse what user locks are. Even though they will be used as tuple locks 99% of the time, user locks are only loosely coupled with tuples in part because there is no sytem generated column which is persistent and 32 bits. IMO, this is a problem with the current user lock module...it encourages locking over oid which is a bad practice. A properly implemented user lock system would likely maintain a global sequence shared by all lockable objects, tuple or otherwise. Listing them as object locks seems ok. Merlin ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Feature freeze date for 8.1
On Mon, 2 May 2005 18:47:14 +0300 (EEST) Heikki Linnakangas [EMAIL PROTECTED] wrote: FWIW, I've been bitten by this problem twice with other applications. 1. We had a DB2 database with clients running in other computers in the network. A faulty switch caused random network outages. If the connection timed out and the client was unable to send it's request to the server, the client would notice that the connection was down, and open a new one. But the server never noticed that the connection was dead. Eventually, the maximum number of connections was reached, and the administrator had to kill all the connections manually. Are you pleased from this feature on DB2 ? I think you will say no :-) 2. We had a custom client-server application using TCP across a network. There was stateful firewall between the server and the clients that dropped the connection at night when there was no activity. After a couple of days, the server reached the maximum number of threads on the platform and stopped accepting new connections. Yes, because your firewall drops only connectiona between clients and firewall, not database. In case 1, the switch was fixed. If another switch fails, the same will happen again. In case 2, we added an application-level heartbeat that sends a dummy message from server to client every 10 minutes. TCP keep-alive with a small interval would have saved the day in both cases. Unfortunately the default interval must be = 2 hours, according to RFC1122. Yes.. On most platforms, including Windows and Linux, the TCP keep-alive interval can't be set on a per-connection basis. The ideal solution would be to modify the operating system to support it. How will we do this ? What we can do in PostgreSQL is to introduce an application-level heartbeat. A simple Hello world message sent from server to client that the client would ignore would do the trick. This couldnt be not forgetten that a clients can have more than one connection to database and one of them is lost.. Best Regards, Adnan DURSUN ASRIN Biliim Hiz.Ltd. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: Bruce Momjian wrote: Dave Held wrote: Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? Aren't they the people that you need buy-in from to really contribute to Postgres? The reason I make this point is because I know what a democratic development community really looks like, and the Boost community is one such example. That truly *is* democratic, because decisions are made as a group, and no fixed subset of members has an overriding veto. The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. Really? You have a different perspective than I see. I have seen patches be accepted that had no core buy-in. We accept patches based on group feedback, not some closed approval process. Let me also ask for you to provide an example of the behavior you describe. How many patches have you applied thinking they were good/safe, only to have Tom jump on top of you and either require changes, or yank them completely? As far as code submissions are concerned, Tom has pretty much been final arbitrar (not that I'm against that, I think its required to keep the code 'clean') ... those with cvs commit privileges are a bit higher on the totem, but they've already been through the fire with Tom, else they would't have those privileges ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] pg_locks needs a facelift
On Mon, May 02, 2005 at 01:12:06PM -0400, Merlin Moncure wrote: I don't like the idea of listing user locks with 'tuple' locks for no other reason than this might confuse what user locks are. Even though they will be used as tuple locks 99% of the time, user locks are only loosely coupled with tuples in part because there is no sytem generated column which is persistent and 32 bits. IMO, this is a problem with the current user lock module...it encourages locking over oid which is a bad practice. Another way would be to allow user locks to use the four fields of LOCKTAG. So the user would be able to establish more powerful conventions: say the relation's Oid, and a related sequence value if there is one; or a blocknumber/offset (ctid) if there isn't, etc. A properly implemented user lock system would likely maintain a global sequence shared by all lockable objects, tuple or otherwise. That'd just be equivalent to require that user tables are created WITH OIDS, only the counter wouldn't be shared with system tables ... how is that any better? -- Alvaro Herrera ([EMAIL PROTECTED]) Ellos andaban todos desnudos como su madre los parió, y también las mujeres, aunque no vi más que una, harto moza, y todos los que yo vi eran todos mancebos, que ninguno vi de edad de más de XXX años (Cristóbal Colón) ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] pg_locks needs a facelift
Merlin Moncure [EMAIL PROTECTED] writes: I don't like the idea of listing user locks with 'tuple' locks for no other reason than this might confuse what user locks are. Fair enough, although I think that at least one major application of user locks would be equivalent to tuple locks. Somebody was asking for named user locks in the previous thread, and the easiest way to get that is to make a table containing just lock names, and then lock on the CTIDs of that table. Since there would be no reason to allow UPDATE or DELETE in such a table, the putative instability of CTID doesn't really matter. However, displaying them as object locks certainly works, and you'd have to put some intelligence in front of the view anyway about what meaning you were assigning to user locks in your installation. So you can always cast to whatever you need. IMO, this is a problem with the current user lock module...it encourages locking over oid which is a bad practice. A properly implemented user lock system would likely maintain a global sequence shared by all lockable objects, tuple or otherwise. Certainly the current contrib/userlock code could stand a rewrite. Or more likely, addition of new functions --- we should deprecate the old ones, but I see no need to remove 'em right away. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Bruce Momjian wrote: Dave Held wrote: Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? Aren't they the people that you need buy-in from to really contribute to Postgres? The reason I make this point is because I know what a democratic development community really looks like, and the Boost community is one such example. That truly *is* democratic, because decisions are made as a group, and no fixed subset of members has an overriding veto. The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. Really? You have a different perspective than I see. I have seen patches be accepted that had no core buy-in. We accept patches based on group feedback, not some closed approval process. Let me also ask for you to provide an example of the behavior you describe. How many patches have you applied thinking they were good/safe, only to have Tom jump on top of you and either require changes, or yank them completely? As far as code submissions are concerned, Tom has pretty much been final arbitrar (not that I'm against that, I think its required to keep the code 'clean') ... those with cvs commit privileges are a bit higher on the totem, but they've already been through the fire with Tom, else they would't have those privileges ... Tom can bring up issues, but it is up to the group to decide if those are valid. Tom has sway only to the exent the group agrees with Tom's analysis. If someone else made similar observations, they would take similar weight, as soon as we were sure the person was reliable in their observations. Tom gets patches pulled only because his observations are deemed to be right by the group, not because he is Tom. The same holds for pretty much anyone else in the group. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
We don't want core to steer development anymore than we want a centralized group to do that, because if we did, the next company that comes along and wants to enhance PostgreSQL or offer technical support services will feel they have to get approval/buy-in from the _in_ group, and that isn't a productive setup. The fact that new companies getting involved can't find a central authority is a _good_ thing, if you think about it. It means that we have succeeded in building a community that allows people to join and feel a part right away, and they don't have to buy-in or play politics to do it. Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? PostgreSQL is more of Democratic Republic than an actual democracy but they do very well at it. Any person can bring a patch and submit it, any person in the community can argue for it and any person can take the time to fix it to the specifications that core sets forth. Sincerely, Joshua D. Drake Command Prompt,Inc. -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
[HACKERS] Citation for Bad n_distinct estimation; hacks suggested?
Actually, the earliest paper that solves the distinct_n estimation problem in 1 pass is the following: Estimating simple functions on the union of data streams by Gibbons and Tirthapura, SPAA 2001. http://home.eng.iastate.edu/~snt/research/streaming.pdf The above paper addresses a more difficult problem (1 pass _and_ a distributed setting). Gibbon's followup paper in VLDB 2001 limits the problem to a single machine and contains primarily experimental results (for a database audience). The algorithmic breakthrough had already been accomplished in the SPAA paper. Gurmeet -- Gurmeet Singh Manku Google Inc. http://www.cs.stanford.edu/~manku(650) 967 1890 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
-Original Message- From: Bruce Momjian [mailto:[EMAIL PROTECTED] Sent: Monday, May 02, 2005 12:17 PM To: PostgreSQL advocacy Cc: Dave Held; PostgreSQL-development Subject: Re: [pgsql-advocacy] [HACKERS] Increased company involvement [...] Really? You have a different perspective than I see. I have seen patches be accepted that had no core buy-in. We accept patches based on group feedback, not some closed approval process. Let me also ask for you to provide an example of the behavior you describe. Well, I think there's numerous examples where someone suggests some feature or idea, and Tom or one or two other core developers will say: I don't like that idea, and then the proposer will more or less give up on it because it is clear that it won't go anywhere. So whether the process gets stopped at the patch submission level or the feature proposal level isn't really relevant. It seems pretty clear that a handful of people decide the direction of Postgres, and everyone else can either contribute to the features that have been agreed to be acceptable and relevant, or they can fork their own version. Just watching the hackers list suggests to me that this is the norm, rather than the exception. I guess I'm interested to see which patches have been accepted that the core developers opposed. Now don't get me wrong. Sometimes there are good technical reasons why feature A or B can't or shouldn't be added or even developed. And I don't suggest that patches lacking technical merit should not be rejected. But sometimes it seems that ideas with undetermined merit get passed over because of a quick judgement based on intuition, and only if the proposer actively fights for it for a while does it get reconsidered. Of course, it would be quite a bit of work for me to review the list and compile instances where I think this has occurred, but only because of the tedium involved to make a minor point...not because I think I would have difficulty finding evidence. I'm just saying that as an outsider, if I had a lot of resources to devote to contributing to Postgres, I would only consider working on approved TODO items or making sure I more or less had core buy-in before writing any code. I don't think it would be worth my time to work on something that non-core users/developers might like but core hackers don't. Like I said, that's not necessarily a bad thing. Postgres is a piece of software with many interacting components, and there needs to be some coordination to make sure it evolves in a sensible way. But I think that implies that there must be and is some de facto centralization of control, whether that is the published ideology or not. __ David B. Held Software Engineer/Array Services Group 200 14th Ave. East, Sartell, MN 56377 320.534.3637 320.253.7800 800.752.8129 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Well, I think there's numerous examples where someone suggests some feature or idea, and Tom or one or two other core developers will say: I don't like that idea, and then the proposer will more or less give up on it because it is clear that it won't go anywhere. Well I think that is more perception than anything. Sometimes you have to fight for what you believe in. For example plPHP. I believe plPHP belongs in core as do some other people. There are members of core that are for it and against it. Command Prompt as the submitter needs to make a valid argument to sway core. We need to present code they would be happy with. We need to present reasons why. If you don't do that, then yes I can see why it would feel as if the proposer was at a loss once someone like Tom writes his opinion. However Tom isn't the final word, he just happens to have a lot of weight as anyone within the project of good standing who donates as much as he does would. Everything within the community is pretty much done as a vote and there are things that core really has nothing to do with (like pgFoundry). Sincerely, Joshua D. Drake Command Prompt, Inc. -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Joshua D. Drake wrote: We don't want core to steer development anymore than we want a centralized group to do that, because if we did, the next company that comes along and wants to enhance PostgreSQL or offer technical support services will feel they have to get approval/buy-in from the _in_ group, and that isn't a productive setup. The fact that new companies getting involved can't find a central authority is a _good_ thing, if you think about it. It means that we have succeeded in building a community that allows people to join and feel a part right away, and they don't have to buy-in or play politics to do it. Well, you make Postgres sound like a very democratic community, but I'm afraid this is a fairy tale. Aren't the people who approve patches exactly the in group that you claim doesn't exist? PostgreSQL is more of Democratic Republic than an actual democracy but they do very well at it. Any person can bring a patch and submit it, any person in the community can argue for it and any person can take the time to fix it to the specifications that core sets forth. True, but I don't think core sets the specifications. Rather, it is the community that sets them, or agrees to them by not saying anything. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
-Original Message- From: Joshua D. Drake [mailto:[EMAIL PROTECTED] Sent: Monday, May 02, 2005 12:33 PM To: Dave Held Cc: PostgreSQL-development; PostgreSQL advocacy Subject: Re: [pgsql-advocacy] [HACKERS] Increased company involvement [...] PostgreSQL is more of Democratic Republic than an actual democracy but they do very well at it. I buy that. It is probably a fairly accurate description of the Postgres community. Everyone has a voice, but ultimately, the Senate (i.e.: patch approvers) passes the laws. Where it differs is that the Senate is not necessarily democratically elected. ;) Any person can bring a patch and submit it, any person in the community can argue for it and any person can take the time to fix it to the specifications that core sets forth. Which brings up an important point. The core developers define the structure in which change can occur. If people think that Postgres should move in a direction that affects that framework, they have to convince core to redefine that specification. It's like writing new laws vs. amending the Constitution. Even though anyone can draft a bill and submit it to their representative, it's ultimately Congress that makes the laws. And while public opinion can ultimately affect the actions of Congress, it is still a sovereign body. As Bruce himself said, companies that wish to contribute must not assume that their work will be integrated into Postgres. The official stance is that there only needs to be community buy-in, but it seems more realistic that there needs to be core buy-in as well, at the least because of the influence that core thinking has on the community itself. That's not a bad thing per se, but it's definitely something that contributors should consider. __ David B. Held Software Engineer/Array Services Group 200 14th Ave. East, Sartell, MN 56377 320.534.3637 320.253.7800 800.752.8129 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] pg_locks needs a facelift
A properly implemented user lock system would likely maintain a global sequence shared by all lockable objects, tuple or otherwise. That'd just be equivalent to require that user tables are created WITH OIDS, only the counter wouldn't be shared with system tables ... how is that any better? Well, oid is 32 bit and not guaranteed to be unique...therefore useless. However by properly defined, I meant by the application. The server is agnostic about user locks, a.k.a. 'application defined locks'. Merlin ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Any person can bring a patch and submit it, any person in the community can argue for it and any person can take the time to fix it to the specifications that core sets forth. True, but I don't think core sets the specifications. Rather, it is the community that sets them, or agrees to them by not saying anything. Well I can't concur with that. I can concur that may be what core would want but I don't see that as what it is. However that could could also be perception as it isn't really that clear what core does except set the specification. Sincerely, Joshua D. Drake Command Prompt, Inc. -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Joshua D. Drake wrote: Well, I think there's numerous examples where someone suggests some feature or idea, and Tom or one or two other core developers will say: I don't like that idea, and then the proposer will more or less give up on it because it is clear that it won't go anywhere. Well I think that is more perception than anything. Sometimes you have to fight for what you believe in. For example plPHP. I believe plPHP belongs in core as do some other people. There are members of core that are for it and against it. Command Prompt as the submitter needs to make a valid argument to sway core. We need to present code they would be happy with. We need to present reasons why. I think the plan for plphp is to put the source in our CVS, but to require it to be compiled as a separate 'make' step after php is fully installed and using the new libpq. I think we had agreement on that solution. If you don't do that, then yes I can see why it would feel as if the proposer was at a loss once someone like Tom writes his opinion. However Tom isn't the final word, he just happens to have a lot of weight as anyone within the project of good standing who donates as much as he does would. Everything within the community is pretty much done as a vote and there are things that core really has nothing to do with (like pgFoundry). Right, the point is that Tom has weight because he is Tom and people value his opinion, not because he is core. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Dave Held wrote: Just watching the hackers list suggests to me that this is the norm, rather than the exception. I guess I'm interested to see which patches have been accepted that the core developers opposed. Now don't get me wrong. Sometimes there are good technical reasons why feature A or B can't or shouldn't be added or even developed. And I don't suggest that patches lacking technical merit should not be rejected. But sometimes it seems that ideas with undetermined merit get passed over because of a quick judgement based on intuition, and only if the proposer actively fights for it for a while does it get reconsidered. Of course, it would be quite a bit of work for me to review the list and compile instances where I think this has occurred, but only because of the tedium involved to make a minor point...not because I think I would have difficulty finding evidence. I'm just Well, if there was an issue and you had been around for a minimal amount of time, I would think you could come up with at least one example. saying that as an outsider, if I had a lot of resources to devote to contributing to Postgres, I would only consider working on approved TODO items or making sure I more or less had core buy-in before writing any code. I don't think it would be worth my time to work on something that non-core users/developers might like but core hackers don't. Well, our developer's FAQ clearly states you should communicate your ideas to the hackers list before starting work to be sure you have _community_ buy-in, rather than core buy-in. And the TODO list is not a core list, it is accumulated from community suggestions and discussion. Like I said, that's not necessarily a bad thing. Postgres is a piece of software with many interacting components, and there needs to be some coordination to make sure it evolves in a sensible way. But I think that implies that there must be and is some de facto centralization of control, whether that is the published ideology or not. I will give you the example of adding a read-only table as an example. I am betting the idea will not be accepted because the costs outweight the gain, but I will post my opinions on the list and others will as well and we will come to some concensus. If X members feel something is bad, and 5X members think something is good, it almost always gets in --- it doesn't matter if all the core people are in X or not. Another example is the recent patch to check if there are orphaned file system files. That was submitted, Tom had questions, I posted why I thought it was valid, and the patch is going in today. Anyone has the ability to argue their point and try to sway the community, and any member has the right to request a vote on a specific issue. Perhaps we are more a replublic because we do defer our judgement to others who have looked into the area more thoroughly. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
pgman wrote: If you don't do that, then yes I can see why it would feel as if the proposer was at a loss once someone like Tom writes his opinion. However Tom isn't the final word, he just happens to have a lot of weight as anyone within the project of good standing who donates as much as he does would. Everything within the community is pretty much done as a vote and there are things that core really has nothing to do with (like pgFoundry). Right, the point is that Tom has weight because he is Tom and people value his opinion, not because he is core. It seems that though discerning observers agree with the above statement, the perception of new people is that Tom has weight _mostly_ because he is in core. So, how do we fix this? Do we boot Tom out of core so people can see he has the same impact? That seems pointless. (Funny, no one says I have too much power. I will have to look into how to get some someday.) :-) -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] pg_locks needs a facelift
Merlin Moncure [EMAIL PROTECTED] writes: Fair enough, although I think that at least one major application of user locks would be equivalent to tuple locks. Somebody was asking for named user locks in the previous thread, and the easiest way to get that is to make a table containing just lock names, and then lock on the CTIDs of that table. Since there would be no reason to allow UPDATE or DELETE in such a table, the putative instability of CTID doesn't really matter. This is fine, but relying on structures outside of shared memory is a fairly hefty price. User locks are very fast and tight and incur zero maintenance overhead...with a table you have to consider vacuuming strategies + possible reindex for the unique constraint...bleh. If the lock table was not synced and auto-vacuumed, then maybe it could work. I also wonder if there would be a race condition if someone tried to acquire ctid based named lock at the same time a user lock with the same value, unless ctid locks were maintained in a separate hash table. Interesting aside: you can get very similar functionality by abusing pg_listener...not that I'd suggest doing that however ;). Certainly the current contrib/userlock code could stand a rewrite. Or more likely, addition of new functions --- we should deprecate the old ones, but I see no need to remove 'em right away. well, the old ones are GPL. I've made a few attempts to contact the original author...he's MIA. Since 95% of the implementation is in the backend, it seems odd to have a GPL interface. Merlin ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Joshua D. Drake wrote: Well, I think there's numerous examples where someone suggests some feature or idea, and Tom or one or two other core developers will say: I don't like that idea, and then the proposer will more or less give up on it because it is clear that it won't go anywhere. Well I think that is more perception than anything. Sometimes you have to fight for what you believe in. For example plPHP. I believe plPHP belongs in core as do some other people. There are members of core that are for it and against it. Command Prompt as the submitter needs to make a valid argument to sway core. We need to present code they would be happy with. We need to present reasons why. I think the plan for plphp is to put the source in our CVS, but to require it to be compiled as a separate 'make' step after php is fully installed and using the new libpq. I think we had agreement on that solution. Last I read, both Tom and I were against it being in CVS (albeit for different reasons) ... and there hadn't been any discussions past the end of that thread that I've seen ... I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: Joshua D. Drake wrote: Well, I think there's numerous examples where someone suggests some feature or idea, and Tom or one or two other core developers will say: I don't like that idea, and then the proposer will more or less give up on it because it is clear that it won't go anywhere. Well I think that is more perception than anything. Sometimes you have to fight for what you believe in. For example plPHP. I believe plPHP belongs in core as do some other people. There are members of core that are for it and against it. Command Prompt as the submitter needs to make a valid argument to sway core. We need to present code they would be happy with. We need to present reasons why. I think the plan for plphp is to put the source in our CVS, but to require it to be compiled as a separate 'make' step after php is fully installed and using the new libpq. I think we had agreement on that solution. Last I read, both Tom and I were against it being in CVS (albeit for different reasons) ... and there hadn't been any discussions past the end of that thread that I've seen ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
Dave, The group has moderators, but they exist only to moderate discussion on the mailing lists. I'm not saying that it is bad that Postgres is not democratic. Postgres is a totally different kind of beast than Boost, and probably benefits from having a few people ultimately decide its fate. But let's call a spade a spade and not pretend that contributors don't have to get buy-in from core. Hmmm ... why does everyone assume that Core does more than what we do? I think that most people would be surprised by how *little* traffic there is on the pgsql-core mailing list. Core decides on releases, and approves committers. Occasionally we'll handle something which requires confidentiality, like a security issue or a new corporate participant. The committers, who do *not* have exact overlap with Core (for example, Neil is a committer but not on Core, and I am on Core but not a committer) actually commit patches, so the participation of *one* of them is required to get something in to the core code. Materially, what's accepted is decided through open discussion on the pgsql-hackers list; even Tom brings up his patches for discussion before commit, and I'd defy you to point to even one patch which was accepted by consensus on pgsql-hackers and not committed. As you've already observed, if Tom doesn't like something it's very unlikely to get through.But that's true for a lot of major contributors; the consensus process we use provides ample opportunities to veto and slender opportunities to pass. Go back in the archives to 7.4 development, and you will see Peter exercising his veto a lot, rather than Tom -- and Peter was not a Core team member at the time. From my perspective, this is a good thing for a database system which can get easily broken by an ill-considered patch. It's *good* for us to be development-conservative. So there is an insider group, but it's the group of major contributors. Tom has the loudest voice because he writes the most code. The fact that Tom, Bruce or Peter's veto is often as far as a proposal goes is simply because most of the pgsql-hackers subscribers simply don't involve themselves in the process unless it's one of their own pet features. And the important thing about the group of major contributors is that membership is open. This goes beyond new proposals. Just the other day Bruce was lamenting the fact that despite having a number of committers, nobody other than him seems willing to work out the conflicts and get pending patches into acceptable shape for backend integration -- some patches stayed in the queue for months while he was out.This is bad; it bottlenecks us and makes Bruce and Tom the de-facto arbiters of acceptance because they personally have to adjust and commit submissions. If people want the acceptance process to be more democratic, then those people have to be willing to do the work of full participation. This means arguing and doing research on the hackers list, even for proposals that don't personally benefit you; helping debug and/or test patches to get rid of their problems; and ultimately, becoming a major contributor and then a committer yourself so that you can take over part of Bruce's workload. When this system has broken down it's specifically because people on the -hackers list were lazy or distracted and ignored other people's patch proposals, allowing one member's (whether Tom or anyone else) reflexive veto to stand without challenge. And by failing to champion the usefulness of proposals. I know that some of Joe's proposals were unfairly killed simply because nobody on -hackers spoke up for them, leading Tom and others to believe that they weren't popular or needed. Personally, I tend to think that one of the several things fundamentally broken in the US electoral system is that there is no relationship between political participation, voting, and authority. I don't see any reason to replicate those mistakes with our project.So if your definition of democracy is everyone has an equal voice regardless of participation level, then thank the gods we're not a democracy. (P.S. on a complete tangent, call a spade a spade is actually a racist expression originating in the reconstruction-era South. spade does not mean garden tool but is a derogatory slang term for black people. It's an expression I avoid for that reason. I don't expect anyone to have known this, but now you do.) -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Feature freeze date for 8.1
On 2005-05-02, Tom Lane [EMAIL PROTECTED] wrote: While that isn't an unreasonable issue on its face, I think it really boils down to this: the OP is complaining because he thinks the connection-loss timeout mandated by the TCP RFCs is too long. Perhaps the OP knows network engineering far better than the authors of those RFCs, or perhaps not. I'm not convinced that Postgres ought to provide a way to second-guess the TCP stack ... Speaking as someone who _does_ know network engineering, I would say that yes, Postgres absolutely should do so. The TCP keepalive timeout _is not intended_ to do this job; virtually every application-level protocol manages its own timeouts independently of TCP. (The few exceptions, such as telnet, tend to be purely interactive protocols that rely on the user to figure out that something got stuck.) One way to handle this is to have an option, set by the client, that causes the server to send some ignorable message after a given period of time idle while waiting for the client. If the idleness was due to network partitioning or similar failure, then this ensures that the connection breaks within a known time. This is safer than simply having the backend abort after a given idle period. If you want comparisons from other protocols, just look around - SMTP, ssh, IRC, BGP, NNTP, FTP, and many, many more protocols all use timeouts (or in some cases keepalive messages) with intervals much shorter than the TCP keepalive timeout itself. -- Andrew, Supernews http://www.supernews.com - individual and corporate NNTP services ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] pg_locks needs a facelift
Merlin Moncure [EMAIL PROTECTED] writes: Merlin Moncure [EMAIL PROTECTED] writes: Fair enough, although I think that at least one major application of user locks would be equivalent to tuple locks. Somebody was asking for named user locks in the previous thread, and the easiest way to get that is to make a table containing just lock names, and then lock on the CTIDs of that table. Since there would be no reason to allow UPDATE or DELETE in such a table, the putative instability of CTID doesn't really matter. This is fine, but relying on structures outside of shared memory is a fairly hefty price. User locks are very fast and tight and incur zero maintenance overhead...with a table you have to consider vacuuming strategies + possible reindex for the unique constraint...bleh. What vacuuming strategy? It's a constant table, at least in my view of the usage. I see no reason for the table lookups to be part of the performance critical path, either --- if you're grabbing and releasing a particular lock a lot, you could read the needed CTID and cache it on the application side. In any case, we are certainly *not* expanding LOCKTAG to the point where it can hold random user-defined strings ;-) I also wonder if there would be a race condition if someone tried to acquire ctid based named lock at the same time a user lock with the same value, unless ctid locks were maintained in a separate hash table. This would be a matter of making sure you didn't use conflicting LOCKTAG bit patterns for different purposes. In practice the easiest way to do that would be to add more LockTagType enum values, which is trivial enough now. (I'll probably fix pg_locks so that if it doesn't recognize a particular LockTagType value, it prints the locktag as the numeric value of the tag, rather than falling back to something unhelpful like unknown. This way you could make some use of freshly-invented tag values without any changes at all in the core backend.) well, the old ones are GPL. I've made a few attempts to contact the original author...he's MIA. Since 95% of the implementation is in the backend, it seems odd to have a GPL interface. I agree. Wasn't it you that was proposing to rewrite the module from scratch to eliminate the GPL restriction? regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased
Marc G. Fournier wrote: On Mon, 2 May 2005, Josh Berkus wrote: As you've already observed, if Tom doesn't like something it's very unlikely to get through. One thing to note on this one ... I've never seen Tom *not* try and help the submitter to get the code up to spec either ... he's always bent over backwards to try and help set someone on the right path if the patch submission warrants it ... I would second this. Even when he has disagreed he is always willing to say why and even offer guidance to sway him. Sincerely, Joshua D. Drake Command Prompt, Inc. Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased
On Mon, 2 May 2005, Josh Berkus wrote: As you've already observed, if Tom doesn't like something it's very unlikely to get through. One thing to note on this one ... I've never seen Tom *not* try and help the submitter to get the code up to spec either ... he's always bent over backwards to try and help set someone on the right path if the patch submission warrants it ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
-Original Message- From: Josh Berkus [mailto:[EMAIL PROTECTED] Sent: Monday, May 02, 2005 1:21 PM To: Bruce Momjian Cc: Marc G. Fournier; PostgreSQL advocacy; Dave Held; PostgreSQL-development Subject: Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement [...] Hmmm ... why does everyone assume that Core does more than what we do? I think that most people would be surprised by how *little* traffic there is on the pgsql-core mailing list. Well, I never said that core runs around saving the world. I mostly made the point that core developers have special influence, and that should be considered when contributing to Postgres, which is directly relevant to the point of the thread, which was originally called Increased company involvement. Core decides on releases, and approves committers. Occasionally we'll handle something which requires confidentiality, like a security issue or a new corporate participant. Which is also something that new would-be corporate contributors should know about. [...] Materially, what's accepted is decided through open discussion on the pgsql-hackers list; even Tom brings up his patches for discussion before commit, and I'd defy you to point to even one patch which was accepted by consensus on pgsql-hackers and not committed. But this misses the point. The point is that consensus is often an iterative process, and even if a few people support an idea at first, in the end, the weight of a few inner circle people (whether they be core or patch approvers or whatnot) tends to sway the consensus in a certain direction. This isn't always bad, especially if those core people simply know more about the internals of Postgres to have better judgement. It is bad if the person making the proposal doesn't feel he/she has good odds in defending the proposal and gives up without a fight. As you've already observed, if Tom doesn't like something it's very unlikely to get through.But that's true for a lot of major contributors; the consensus process we use provides ample opportunities to veto and slender opportunities to pass. This also misses another point. I'm not saying that the current process is inherently flawed. It's probably about as good as any OSS project. My point is that it's not *democratic*, and that outsiders wishing to contribute should understand the dynamic of the process that is not explicitly and officially spelled out anywhere. [...] From my perspective, this is a good thing for a database system which can get easily broken by an ill-considered patch. It's *good* for us to be development-conservative. Right. I agree. I'm not criticising the process as a whole, and I've more or less made this exact point myself. So there is an insider group, but it's the group of major contributors. That is exactly my point, but you said it better. Tom has the loudest voice because he writes the most code. The fact that Tom, Bruce or Peter's veto is often as far as a proposal goes is simply because most of the pgsql-hackers subscribers simply don't involve themselves in the process unless it's one of their own pet features. Which is perfectly understandable. You can probaby guess that most people who use Postgres haven't tried to implement an RDBMS themselves, and have only a shallow understanding of the details. And the important thing about the group of major contributors is that membership is open. Which may be true philosophically, but in practice, most people who contribute will not have the resources or motivation to become a major contributor. I do not mean to imply that this is necessarily a bad thing; but I think it is the true state of affairs, and part of the dynamic which must be understood by someone considering investing in Postgres as a contributor. [...] If people want the acceptance process to be more democratic, then those people have to be willing to do the work of full participation. That actually doesn't make it more democratic. In a democracy, everyone has an equal vote regardless of their status. The point is that a democracy is not always a priori the best form of organization. What you describe is actually a meritocracy, and for a project like Postgres, it makes a lot of sense. But that merely reinforces my point that contributors need to understand that if their pet feature they create is not in line with core thinking, they will have to earn the credibility to get community buy-in. [...] (P.S. on a complete tangent, call a spade a spade is actually a racist expression originating in the reconstruction-era South. spade does not mean garden tool but is a derogatory slang term for black people. [...] Interesting. Duly noted. __ David B. Held Software Engineer/Array Services Group 200 14th Ave. East, Sartell, MN 56377 320.534.3637 320.253.7800 800.752.8129 ---(end of
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
Bruce, (P.S. on a complete tangent, call a spade a spade is actually a racist expression originating in the reconstruction-era South. spade does You must be from California. :-) Well, yes. Actually, from San Francisco, which is even worse.And I just spent the weekend in Orange County which really gotten my PC dander up. Sorry, Dave! -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
Josh Berkus josh@agliodbs.com writes: As you've already observed, if Tom doesn't like something it's very unlikely to get through. I lose my share of arguments --- in fact, in the twenty minutes since your posting I already notice Bruce committing a patch I had objected to ;-). Our process is not democratic in the sense of any random subscriber to the mailing lists having the same vote as a core member --- and I'll bet Boost doesn't run things that way either. What we have is pretty informal but I think it effectively gives more weight to the opinions of those more involved in the project; which seems a good way to operate. But there isn't anyone here who has an absolute veto, nor contrarily anyone who can force things in unilaterally over strong objections. [ much good commentary snipped ] regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? Well we don't modify the backend or anything but the way plPHP is written it assumes it is part of the tree, which is why we have to patch. However I think part of the argument goes to API. For example the latest release of plPHP will not work with 7.4. Sincerely, Joshua D. Drake Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. And the point is, as Tom has pointed out with tsearch2, that even *in* CVS, it is a fair amount of work to 'whack' other ppls code ... it shouldn't be Tom's responsibility (which is generally what it comes down to) to keep someone else's code up to speed with changes in the server ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. And the point is, as Tom has pointed out with tsearch2, that even *in* CVS, it is a fair amount of work to 'whack' other ppls code ... it shouldn't be Tom's responsibility (which is generally what it comes down to) to keep someone else's code up to speed with changes in the server ... Well we try to keep up :) Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company
Dave Held wrote: [...] (P.S. on a complete tangent, call a spade a spade is actually a racist expression originating in the reconstruction-era South. spade does not mean garden tool but is a derogatory slang term for black people. [...] Interesting. Duly noted. It would be interesting if it were true. Be careful the urban myths you accept. See for more info: http://www.randomhouse.com/wotd/index.pperl?date=19970115 Use of the phrase is unwise these days because of the widespread misconception about its origin. PC bites us all. cheers andrew ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
Dave, Well, I never said that core runs around saving the world. I mostly made the point that core developers have special influence, Yep. Absolutely. I wanted to point out to you that core isn't the only group within PostgreSQL that has special influence. Which is also something that new would-be corporate contributors should know about. Yes. All of this would be worthy of a FAQ somewhere. Up for it? It is bad if the person making the proposal doesn't feel he/she has good odds in defending the proposal and gives up without a fight. Yes.Again, I think a FAQ would help. If people are prepared for the idea of defending their ideas, then they're less likely to quit as soon as someone says no. My point is that it's not *democratic*, and that outsiders wishing to contribute should understand the dynamic of the process that is not explicitly and officially spelled out anywhere. Hmmm. We'll there's two (or more) uses of the word democratic; so I think there's considerable confusion resulting. In the sense of democratic meaning maximizing the participation and authority of all project members, we are democratic. In the sense of one person, one vote, we are not. Classically, our structure could be described as anarchistic -- in the 1890's definition, not the modern one. Right. I agree. I'm not criticising the process as a whole, and I've more or less made this exact point myself. Yes. I'm not responding just to you, btw. I'm responding to a number of comments from other people who erroneously see Core as exercising more authority than we actually do. Which may be true philosophically, but in practice, most people who contribute will not have the resources or motivation to become a major contributor. I do not mean to imply that this is necessarily a bad thing; but I think it is the true state of affairs, and part of the dynamic which must be understood by someone considering investing in Postgres as a contributor. Certainly. Although the decision-making process for acceptance is really of interest primarily for contributors; that is, if you are not submitting, even by proxy, it shouldn't really matter to you how stuff gets accepted. Except to the extent that you *should* jump in and advocate for proposals by others which you like so that the contributors, committers, and core know what people care about. That actually doesn't make it more democratic. In a democracy, everyone has an equal vote regardless of their status. The point is that a democracy is not always a priori the best form of organization. Certainly. See above. What you describe is actually a meritocracy, and for a project like Postgres, it makes a lot of sense. Hmmm ... I dislike the word meritocracy because it is applied equally to corporations, where regardless of merit you're never going to be on the Board. But that merely reinforces my point that contributors need to understand that if their pet feature they create is not in line with core thinking, they will have to earn the credibility to get community buy-in. Substitute major contributors for core, and you have *my* buy-in. -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier [EMAIL PROTECTED] writes: On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? That could be said of *any* of our PLs (at least now that we install all server-side headers by default ;-)). I think the real reason we keep pltcl etc in the core CVS is exactly what Bruce said: it's easier to maintain 'em that way. The problem is that the PLs use all sorts of internal backend APIs that we don't want to freeze, and so they are constantly being affected by changes in the core backend. Just look at the CVS logs for evidence. Personally, I'm willing to fix the PLs whenever I make a change that affects them, but only if they're in core CVS. Dealing with parallel changes in two different code repositories is too much of a pain. So the folks maintaining non-core PLs take a big hit every release when they have to sync with what's happened in the core backend meanwhile. regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Joshua D. Drake wrote: The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. And the point is, as Tom has pointed out with tsearch2, that even *in* CVS, it is a fair amount of work to 'whack' other ppls code ... it shouldn't be Tom's responsibility (which is generally what it comes down to) to keep someone else's code up to speed with changes in the server ... Well we try to keep up :) I'm not pointing fingers at you either :) But, you are one of how many that try and get 'added to core'? How many things do we have in contrib that the only person that does any 'whacking' is Tom? A couple I've seen patches go around for, but for a good portion of them, I imagine that they are 'dead except that Tom keeps fixing them' ... Tom's focus shouldn't be making sure that everyone's third party add on still works during a release cycle, that should be the responsibility of the maintainers of those projects, to follow changes and make sure they are implemented ... That is what pgFoundry was setup for ... to give projects the visibiilty they would get through the core distribution by making sure they are referenced in a central place, but providing the maintainers with direct CVS access to make changes to their code in a timely manner .. *shrug* Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company
Josh Berkus wrote: Dave, Well, I never said that core runs around saving the world. I mostly made the point that core developers have special influence, Yep. Absolutely. I wanted to point out to you that core isn't the only group within PostgreSQL that has special influence. Which is also something that new would-be corporate contributors should know about. Yes. All of this would be worthy of a FAQ somewhere. Up for it? I am working on one based on my original statement that started this thread. Witht this dicussion, it has move to the top of my TODO list. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
I'm not pointing fingers at you either :) But, you are one of how many that try and get 'added to core'? How many things do we have in contrib that the only person that does any 'whacking' is Tom? A couple I've seen patches go around for, but for a good portion of them, I imagine that they are 'dead except that Tom keeps fixing them' ... In contrib I would bet a lot. I have argued for the removal of TSearch (not TSearch2) for example. Also RServ could probably stand to be removed. However we are not talking about contrib (or at least I am not). We were talking about PLs which are a little bit of a different beast. Tom's focus shouldn't be making sure that everyone's third party add on still works during a release cycle, that should be the responsibility of the maintainers of those projects, to follow changes and make sure they are implemented ... I would agree, I suggested test cases for contrib once. I think that would be very good. If the contrib fails the test case for itself say after (this could go for pls to) Beta2 then it gets yanked. That is what pgFoundry was setup for ... to give projects the visibiilty they would get through the core distribution by making sure they are referenced in a central place, but providing the maintainers with direct CVS access to make changes to their code in a timely manner .. *shrug* It was what pgFoundry was setup for but as I have said elsewhere perception is everything. If it isn't in core, it is a second class project. Regardless of how we all want to feel about it. Sincerely, Joshua D. Drake Command Prompt, Inc. -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: I posted this compromise and no one replied so I thought everyone was OK with it. It gets it into CVS, but has a separate compile stage to deal with the recursive dependency problem. Then what is the point of having it in CVS? Other then to make are tar ball bigger? So it can be maintained with other PL languages as the internal API changes. This is the same reason ecpg is in our CVS because it is tied to the grammar. Since when? I thought you didn't need the PostgreSQL sources in order to compile pl/PHP, only the installed headers/libraries ... Joshua, has something changed, or did I mis-understand that requirement? The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. And the point is, as Tom has pointed out with tsearch2, that even *in* CVS, it is a fair amount of work to 'whack' other ppls code ... it shouldn't be Tom's responsibility (which is generally what it comes down to) to keep someone else's code up to speed with changes in the server ... When a PL languages is so tied to the changing API that it will only work for a single major release of PostgreSQL, like ecpg, it is usually kept in our CVS. This isn't true of odbc, jdbc, or other stuff, and not even Slony. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Andrew Dunstan wrote: I've deliberately let the dust settle slightly on this. One thing that might help is a more open sponsorship clearing house. Example (not meant as a bid, but just to illustrate): the JDBC driver needs a scanner overhaul - it breaks on dollar quoting and a bunch of other stuff. I could do that work (as could others, of course) but I don't have time, unless someone buys some of my professional time. Someone might want to do just that, but how would they find me? Regarding the secret code stuff - I predict that it will quickly bite whoever does it, unless they are extremely lucky. I like this idea. There is another issue too. In general, there is a feeling like one needs to produce something that works and not wait for the slower movement of the community's approval. I don't think several open source forks for the project necessarily produce problems if most of these are used as experimental projects. The example which comes to mind is Samba-TNG. So some of this concern may be overblown. This is also the way things work with the SQL Standard: The various vendors (PostgreSQL included) go out and start with the base, extend that feature set, and eventually come back together to try to build the next standard based on everyone's experience. This embrace and extend process is indeed critical for the further development of the standard. At the same time, I agree with Bruce's main point-- that the lack of communication is a threat to this progress. So at least some note of Best practices regarding these extensions or contributions would be a help. Adding a clearing house to this would also add a critical tool and would also have the side effect of increasing the pace of development. Maybe have it divided into two sections: Bids and Bounties. Best Wishes, Chris Travers Metatron Technology Consulting ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Joshua D. Drake wrote: The issue is that we have had to wack around the existing PL languages for almost every release to make them work with server changes, and being outside our CVS, plPHP isn't getting that whacking. And the point is, as Tom has pointed out with tsearch2, that even *in* CVS, it is a fair amount of work to 'whack' other ppls code ... it shouldn't be Tom's responsibility (which is generally what it comes down to) to keep someone else's code up to speed with changes in the server ... Well we try to keep up :) I'm not pointing fingers at you either :) But, you are one of how many that try and get 'added to core'? How many things do we have in contrib that the only person that does any 'whacking' is Tom? A couple I've seen patches go around for, but for a good portion of them, I imagine that they are 'dead except that Tom keeps fixing them' ... Tom's focus shouldn't be making sure that everyone's third party add on still works during a release cycle, that should be the responsibility of the maintainers of those projects, to follow changes and make sure they are implemented ... That is what pgFoundry was setup for ... to give projects the visibiilty they would get through the core distribution by making sure they are referenced in a central place, but providing the maintainers with direct CVS access to make changes to their code in a timely manner .. *shrug* The bottom line is that it is more efficient for one person to adjust all the PL's at once rather than each PL learning about the changes and making them. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company involvement
On Monday 02 May 2005 14:49, Josh Berkus wrote:
Bruce,
(P.S. on a complete tangent, call a spade a spade is actually a
racist expression originating in the reconstruction-era South.
spade does
You must be from California. :-)
Well, yes. Actually, from San Francisco, which is even worse.And I
just spent the weekend in Orange County which really gotten my PC dander
up. Sorry, Dave!
And this is why I hate the PC movement... The phrase calling a spade a spade
pre-dates the U.S. as whole, much less the reconstruction-era South. It is
believed to have been introduced into the english language via John Knox in
the 1500's who wrote I have learned to call wickedness by its own terms: A
fig, a fig, and a spade a spade, borrowing from an earlier latin text, which
is believed to be based on an even earlier Greek writer.
--
Robert Treat
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
As someone who has made a few minor contributions and plenty of suggestions, but who is not on the core team, I would like to offer my observations. Every suggestion I have ever made that had any merit at all has eventually worked its way into PostgreSQL (most -- perhaps all -- were already under consideration). The dumb ideas I coughed up were neatly shot down in flames as they should have been. The PostgreSQL project is very well run, in fact amazingly well considering the loose structure of an open source project. There are a few things that might be accomplished a bit differently, but I do not know how it would be possible to architect it (for instance, I would completely divorce the testing team from the development team since that is how it is usually done with commercial systems). IMO-YMMV. -Original Message- From: [EMAIL PROTECTED] [mailto:pgsql-hackers- [EMAIL PROTECTED] On Behalf Of Bruce Momjian Sent: Monday, May 02, 2005 10:56 AM To: Dave Held Cc: PostgreSQL-development; PostgreSQL advocacy Subject: Re: [pgsql-advocacy] [HACKERS] Increased company involvement Dave Held wrote: Just watching the hackers list suggests to me that this is the norm, rather than the exception. I guess I'm interested to see which patches have been accepted that the core developers opposed. Now don't get me wrong. Sometimes there are good technical reasons why feature A or B can't or shouldn't be added or even developed. And I don't suggest that patches lacking technical merit should not be rejected. But sometimes it seems that ideas with undetermined merit get passed over because of a quick judgement based on intuition, and only if the proposer actively fights for it for a while does it get reconsidered. Of course, it would be quite a bit of work for me to review the list and compile instances where I think this has occurred, but only because of the tedium involved to make a minor point...not because I think I would have difficulty finding evidence. I'm just Well, if there was an issue and you had been around for a minimal amount of time, I would think you could come up with at least one example. saying that as an outsider, if I had a lot of resources to devote to contributing to Postgres, I would only consider working on approved TODO items or making sure I more or less had core buy-in before writing any code. I don't think it would be worth my time to work on something that non-core users/developers might like but core hackers don't. Well, our developer's FAQ clearly states you should communicate your ideas to the hackers list before starting work to be sure you have _community_ buy-in, rather than core buy-in. And the TODO list is not a core list, it is accumulated from community suggestions and discussion. Like I said, that's not necessarily a bad thing. Postgres is a piece of software with many interacting components, and there needs to be some coordination to make sure it evolves in a sensible way. But I think that implies that there must be and is some de facto centralization of control, whether that is the published ideology or not. I will give you the example of adding a read-only table as an example. I am betting the idea will not be accepted because the costs outweight the gain, but I will post my opinions on the list and others will as well and we will come to some concensus. If X members feel something is bad, and 5X members think something is good, it almost always gets in --- it doesn't matter if all the core people are in X or not. Another example is the recent patch to check if there are orphaned file system files. That was submitted, Tom had questions, I posted why I thought it was valid, and the patch is going in today. Anyone has the ability to argue their point and try to sway the community, and any member has the right to request a vote on a specific issue. Perhaps we are more a replublic because we do defer our judgement to others who have looked into the area more thoroughly. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Feature freeze date for 8.1
One way to handle this is to have an option, set by the client, that causes the server to send some ignorable message after a given period of time idle while waiting for the client. If the idleness was due to network partitioning or similar failure, then this ensures that the connection breaks within a known time. This is safer than simply having the backend abort after a given idle period. Another option is to have the client driver send some ignorable message instead of the server. If the server doesn't get a message every timeout minutes/seconds + slop factor, then it drops the connection. So libpq, JDBC, .net etc would all have to have this implemented, but the changes to the server would probably be simpler this way, wouldn't they? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Joshua D. Drake wrote: I'm not pointing fingers at you either :) But, you are one of how many that try and get 'added to core'? How many things do we have in contrib that the only person that does any 'whacking' is Tom? A couple I've seen patches go around for, but for a good portion of them, I imagine that they are 'dead except that Tom keeps fixing them' ... In contrib I would bet a lot. I have argued for the removal of TSearch (not TSearch2) for example. Also RServ could probably stand to be removed. rserv was removed long ago ... not sure why tsearch is in there stil though ... If it isn't in core, it is a second class project. Regardless of how we all want to feel about it. As you've seen in the note that I sent to you ... my biggest 'beef' against continually adding things is the download size just keeps growing, and when someone already has the core installed, having to redownload it because they've decided to add something like pl/PHP when pl/PHP doesn't need anything but the headers/libraries that are already installed seems idiotic at best ... if some method of extending 'make dist' for the release cycle can be devised so that pl/PHP (and the other pl/'s would be nice eventually) tar *with* the core .tar.gz *and* a seperate plphp-release.tar.gz file that can be downloaded seperately, my arguments against will have been negated ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Feature freeze date for 8.1
On Mon, May 02, 2005 at 12:29:33 -0700, Rob Butler [EMAIL PROTECTED] wrote: One way to handle this is to have an option, set by the client, that causes the server to send some ignorable message after a given period of time idle while waiting for the client. If the idleness was due to network partitioning or similar failure, then this ensures that the connection breaks within a known time. This is safer than simply having the backend abort after a given idle period. Another option is to have the client driver send some ignorable message instead of the server. If the server doesn't get a message every timeout minutes/seconds + slop factor, then it drops the connection. So libpq, JDBC, .net etc would all have to have this implemented, but the changes to the server would probably be simpler this way, wouldn't they? Except it won't work, because the server is who needs to know about the problem. If the network is down, you can't send a TCP RST packet to close the connection on the server side. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Joshua D. Drake wrote: I'm not pointing fingers at you either :) But, you are one of how many that try and get 'added to core'? How many things do we have in contrib that the only person that does any 'whacking' is Tom? A couple I've seen patches go around for, but for a good portion of them, I imagine that they are 'dead except that Tom keeps fixing them' ... In contrib I would bet a lot. I have argued for the removal of TSearch (not TSearch2) for example. Also RServ could probably stand to be removed. rserv was removed long ago ... not sure why tsearch is in there stil though ... Why is dbmirror still there? Can't it be moved to pgfoundry? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Feature freeze date for 8.1
On 2005-05-02, Rob Butler [EMAIL PROTECTED] wrote: Another option is to have the client driver send some ignorable message instead of the server. If the server doesn't get a message every timeout minutes/seconds + slop factor, then it drops the connection. So libpq, JDBC, .net etc would all have to have this implemented, but the changes to the server would probably be simpler this way, wouldn't they? Then the client has to guarantee that it can stop whatever it was doing (which might have nothing to do with the database) every so often in order to send a message; this isn't feasible for most clients. The server-based method is actually no more complex to implement on the server end and does not impose any such restrictions on the client (even if the client sets the option and then ignores the database connection for a long time, all that happens is that the TCP window fills up). -- Andrew, Supernews http://www.supernews.com - individual and corporate NNTP services ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-advocacy] [HACKERS] Decision Process WAS: Increased company
Tom Lane wrote: Josh Berkus josh@agliodbs.com writes: As you've already observed, if Tom doesn't like something it's very unlikely to get through. I lose my share of arguments --- in fact, in the twenty minutes since your posting I already notice Bruce committing a patch I had objected to ;-). I replied to your concerns on Saturday night, and when I saw no community response by noon today, I assume everyone as OK and applied it. I suppose this is as good an illustration of the process as we are going to get. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
On Mon, 2 May 2005, Bruce Momjian wrote: Why is dbmirror still there? Can't it be moved to pgfoundry? Anyone willing to take ownership of it to setup the project itself on pgfoundry? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [pgsql-advocacy] [HACKERS] Increased company involvement
Marc G. Fournier wrote: On Mon, 2 May 2005, Bruce Momjian wrote: Why is dbmirror still there? Can't it be moved to pgfoundry? Anyone willing to take ownership of it to setup the project itself on pgfoundry? I thought it was still maintained? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 8: explain analyze is your friend -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedication Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
