[HACKERS] gin fast insert performance
Here is a test of the fast insert patch. The patch has gone through some changes, so this set of tests is to see the current performance impact compared with HEAD. The test is simple: inserting a bunch of integer arrays into a table with a GIN index on the array column. I'm testing with small work_mem and large work_mem because the smaller work mem should be periodically flushing the pending list throughout a large insert, while large work_mem should allow larger batches to build up before flushing the pending list. For HEAD, larger work_mem should have no effect. HEAD (work_mem = 1MB): 1 tuples, 1000 array elements each insert: run1: 127859.012 ms; run2: 124092.098 ms vacuum analyze: run1: 1681.521 ms; run2: 1688.483 ms 100 tuples, 10 array elements each insert: run1: 122069.072 ms; run2: 116476.058 ms vacuum analyze: run1: 3349.210 ms; run2: 2826.328 ms With Fast Insert Patch (work_mem = 1MB): 1 tuples, 1000 array elements each insert: run1: 111376.670 ms; run2: 110733.455 ms vacuum analyze: run1: 2078.427 ms; run2: 1781.152 ms 100 tuples, 10 array elements each insert: run1: 65743.075 ms; run2: 65252.698 ms vacuum analyze: run1: 3458.500 ms; run2: 3719.916 ms With Fast Insert Patch (work_mem = 1GB): 1 tuples, 1000 array elements each insert: run1: 69419.986 ms; run2: 68985.635 ms vacuum analyze: run1: 57521.067 ms; run2: 43385.682 ms 100 tuples, 10 array elements each insert: run1: 48343.827 ms; run2: 49192.153 ms vacuum analyze: run1: 21134.267 ms; run2: 20474.775 ms With the fast insert patch, the total time for insert + vacuum isn't much different with increased work_mem, but increased work_mem clearly defers a lot of the work to VACUUM. For comparison, building the index afterward is: 1 tuples, 1000 array elements each insert time:7531.645 ms index build time: 24393.737 ms 100 tuples, 10 array elements each insert time: 11564.604 ms index build time: 12753.891 ms So, unfortunately, the fast insert patch does not appear to bring the overall time anywhere close to building the index from scratch. When the work_mem is set to 1GB, the VACUUM took about twice as long to run than the entire index build. Teodor, can you explain why that might be? It does show improvement, and I think my test case might just be too small. It seems like a lot of time is used inserting into the pending list, but it seems like it should be a simple operation. Maybe that can be optimized? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] V4 of PITR performance improvement for 8.4
Koichi Suzuki koichi@gmail.com writes: It's simply because we should not refer to system catalog during the recovery. I don't understand how this is connected to anything to do with prefetching? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PostgreSQL Updated Revision (r1460)
Sorry, I attached incorrect patch file. It is the correct one. KaiGai Kohei wrote: Robert, The attached patch is a draft to replace RedHat/Fedora RPM centric expressions, to add a reference at Database Roles and Privileges chapter and a bit cleanups for the latest revision (r1467). In the previous revision, it noted users to check the version of RPM package, but the revised one notes actually required features. The version number is rewritten as a hint. What is your opinion? Thanks, KaiGai Kohei wrote: Robert Haas wrote: On Fri, Jan 23, 2009 at 12:30 AM, KaiGai Kohei kai...@ak.jp.nec.com wrote: The patch set of SE-PostgreSQL and related stuff were updated (r1460). [1/5] http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1460.patch [2/5] http://sepgsql.googlecode.com/files/sepostgresql-utils-8.4devel-3-r1460.patch [3/5] http://sepgsql.googlecode.com/files/sepostgresql-policy-8.4devel-3-r1460.patch [4/5] http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1460.patch [5/5] http://sepgsql.googlecode.com/files/sepostgresql-tests-8.4devel-3-r1460.patch KaiGai - I read through your docs patch tonight and did some copy editing. Please see the attached patches, which I hope you will find helpful. I have attached my suggested changes both as a patch against v1460 (sepostgresql-docs-rmh-vs-1460.gz) and also as patch against CVS HEAD (sepostgresql-docs-rmh-vs-cvs-head), since I am not sure which is easier for you. I have a couple of general comments about the documentation: Thanks your feedbacks! I basically applied your fixes as is, expect for the following items: - You replaced ! Its providing access controls are not _bypassable_ for any clients ... by ! The access controls implemented by SE-PostgrSQL may not be _biased_ even ... I wanted to express it is unavoidable here, so I changed as: ! The access controls implemented by SE-PostgrSQL may not be _bypassed_ even ... - I found a typo: MAC is described as MAc. And, I have a question about documentation manner. - You represented getpeercon() function as a system call. But, it is actually a wrapper function of getsockopt(2) system call, so the getpeercon(3) is not a system call strictly. Is it necessary to represent these stuffs strictly correct? (Thus, I wrote it as API in the r1460.) 1. The docs as written are very Red Hat-centric, even to the point of making reference to specific versions of Red Hat RPMs. I think that the community will find this unacceptable, as Red Hat is certainly not the only SELinux-enabled distribution and I presume that we want to support all of them to an equal degree. I guess you pointed out about: 1. The Requirement section in Build and Installation assumes RedHat/Fedora's RPM package and its version number. 2. The security context and security policy used to explanation assumes specific security policy. 3. Labeled IPsec seciton points to RedHatEL4 Security Guide, and it assumes the racoon's configuration files are deployed as RPM package doing. About 1, is it necessary to rip the RPM specific version number and replace it as: selinux-policy which includes SE-PostgreSQL related stuffs. About 2, SELinux community provides its default security policy, and distributor's policy (including RedHat's one) is a derivative of the default policy. It is developed independent from distributor's cycle. http://oss.tresys.com/projects/refpolicy http://oss.tresys.com/repos/refpolicy/trunk/policy/modules/services/postgresql.te You can find some of sepgsql_ identifiers in postgresql.te. All the appeared identifiers are upstreamed, so these are not Red Hat specific. About 3, If it rips the link to Red Hat and does not assume specific path of racoon.conf, the explnation become neutral. 2. Some of the information that is documented here properly belongs in other sections of the documentation. For example, the information about GUCs clearly belongs somewhere in the section on server configuration where all of the other GUCs are documented, not in a separate sections about SE-PostgreSQL. These explanations are moved to Security and Authentication section in Chapter 18. Server Configuration. I suspect that all of the information about row-level ACLs should be ripped out of security.sgml and inserted into an appropriate portion of the Database Roles and Privileges chapter, leaving this file to talk just about SE-PostgreSQL. It is indeed an aspect of row-level ACLs. However, it is also a feature on PGACE framework, same as SE-PostgreSQL. An idea is to put a reference to indicate the row-level ACLs section on Database Roles and Privileges chapter, like: PostgreSQL has an enhancement of database roles and privileges mechanism which allows to database ACLs in row-level granuality. See, xref ... for more details. What do you think? 3. It seems to me that the analogy between SQL DAC and Unix user/group DAC is
Re: [HACKERS] FATAL: could not open relation pg_tblspc/491086/467369/491103: No such file or directory
On Mon, 2009-01-26 at 09:48 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: In various places in current HEAD we throw a checkpoint when we want to be certain that all buffers have been flushed. In recovery, a checkpoint isn't always a restartpoint for two reasons: timing and rmgr state. This gives both a cause for the error and an explanation of why it does not occur consistently. ISTM this could likely effect previous releases as well. Were you able to narrow this down? Do you know exactly what command caused it? We know it wasn't any specific command because it caused the bgwriter to crash when HS patch was applied. But no, I'm not looking at it yet, until we're done with HS. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, dropping a tablespace
On Sun, 2009-01-25 at 19:56 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to kill-connected-users if we drop role. Oh, I see, that's still a todo item. But we don't do that during normal operation, so why should we during recovery? LOL. Not many systems allow you to continue working after the access has been removed. But not, as you say, a problem for Hot Standby unless that changes. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, dropping a tablespace
On Sun, 2009-01-25 at 12:13 +, Grzegorz Jaskiewicz wrote: On 2009-01-25, at 09:04, Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction. That sounds like you think there another conclusion? What if subdirectory of that directory is owned by root ? Say I create /home/gj/tablespace1 . Surely emptying it is possible, but should it remove the dir altogether ? It is possible to create tablespace in that directory, even so postgres user doesn't own /home/gj/ directory. So why shouldn't it be possible to drop it ? This is not a question for the Hot Standby patch to answer. Recovery is just trying to emulate what happens in normal running, nothing more. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
--On 25. Januar 2009 13:36:35 -0500 Tom Lane t...@sss.pgh.pa.us wrote: But per spec, UPDATABLE should be the default (if not now, then eventually). Are you proposing CREATE [OR REPLACE] [[NOT] UPDATABLE] VIEW ... ? Seems confusing. Good point. We need a better phrasing to restore the old behavior, maybe CREATE [OR REPLACE] [READ ONLY|UPDATABLE] VIEW ...? I think this looks less confusing. Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, dropping a tablespace
Hi, Simon Riggs wrote: On Sun, 2009-01-25 at 19:56 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to kill-connected-users if we drop role. Oh, I see, that's still a todo item. But we don't do that during normal operation, so why should we during recovery? LOL. Not many systems allow you to continue working after the access has been removed. But not, as you say, a problem for Hot Standby unless that changes. Just like around every unixish system? Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote: If we want to ensure that 8.5 development opens soon, what we have to do is reject those two patches, revert updatable views, and finish up the other stuff (which is all small and could likely be dealt with in a week or two). That puts us in position to go beta by perhaps mid-February with release perhaps on May 1. If we don't, I hereby predict that 8.4 release will not happen before September. Trying to deal with those late, large features will add *at least* one more month to commitfest and *at least* one more month to beta (you think they'll be bug-free?). As our new president has been reminding us, it's time to start making hard choices. We're clearly ahead of where we were in 8.3. Any release with big features in it will take longer, whether you wait a year, or not. I think the choice here is currently between features or release schedule, but we could look at things differently and generate some other options. What would our users want? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote: If we want to ensure that 8.5 development opens soon, what we have to do is reject those two patches, revert updatable views, and finish up the other stuff (which is all small and could likely be dealt with in a week or two). That puts us in position to go beta by perhaps mid-February with release perhaps on May 1. If we don't, I hereby predict that 8.4 release will not happen before September. Expecting a release around September with those patches a bit optimistic, IMHO. We already experienced that a release just after summer is not possible. So, unless there is a way to commit SELinux+Hot Standby+Sync Replication in the next two weeks (or so), as you wrote, we should reject those patches and release 8.4 on time. (Oh, personally I'd like to see all these features at 8.4, from advocacy side). -- Devrim GÜNDÜZ, RHCE devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Simon Riggs wrote: On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote: Any release with big features in it will take longer, whether you wait a year, or not. Well, big features that land early in the release cycle don't delay the release. Just the ones that are submitted in the last commit fest. I think the choice here is currently between features or release schedule, but we could look at things differently and generate some other options. Yeah, that's the tradeoff. features vs. release schedule. I don't see how else you could look at things. What would our users want? I'm sure it depends on the user. Users that are more interested in the features we already have in the bag like window functions and WITH-clause, will obviously prefer to release earlier without hot standby. And users that want hot standby (or SE-postgresql) will prefer to delay the release and have those features included. I don't personally have an opinion, because I don't work with any real-world databases. It's a tradeoff and both options seem good to me. I'm going to stick around and keep reviewing Hot Standby as long as it takes, and when it's ready I can commit it to either 8.4 or 8.5. It's the same amount of work for me either way. There's still a list of non-resolved issues that I estimate will take about two weeks to address. And there's a good possibility that new issues arise, which require yet more time. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
There is another thing that's bothering me, though, which is that the present approach to dumping rules isn't adequate. Consider the following scenario: 1. You create a view that the system considers updatable, so it creates some automatic rules. 2. You don't want those rules, so you delete them, leaving you with the traditional behavior where attempted inserts etc on the view fail. Is that why other db's only make views updateable, that are created WITH CHECK OPTION ? Should we also follow that path ? Andreas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Mon, Jan 26, 2009 at 11:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I'm sure it depends on the user. Users that are more interested in the features we already have in the bag like window functions and WITH-clause, will obviously prefer to release earlier without hot standby. And users that want hot standby (or SE-postgresql) will prefer to delay the release and have those features included. At LinuxLive (UK) the overwhelming majority of people I spoke to over three days wanted hot standby and replication (preferably multi-master, but thats another story). Window functions, recursive queries, SE PostgreSQL, updatable views and other new features were barely mentioned. There's still a list of non-resolved issues that I estimate will take about two weeks to address. And there's a good possibility that new issues arise, which require yet more time. That doesn't seem like it's worth deferring such a useful feature for 12+ months for - especially as the work is clearly ongoing and can happen in parallel with work on other patches. As I've pointed out before, we're not a commercial company working for our shareholders, we're a FOSS project working for our end users. If we can include an important and popular feature like this at the expense of a few weeks extra wait for the release, it seems to me that we'll be serving our users far better overall than making a fair percentage of them wait another 12 months for work that is more or less complete. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with archive_command as suggested by documentation
Simon Riggs wrote: I don't think that particular example is a good one since the whole point of the archive is that it should be off-server. If we're going to be exact about the example then we should give a more realistic one, like using scp. Unfortunately, there is no secure-remote-move command, so doing the above with scp would resend the whole file 3 times. I think it's better to write a script... The problem is that most people do copy the stuff straight out of the documentation. And for those that do write a separate script, there are still a lot of possibilities to get it wrong. There are only about a handful of transportation protocols appearing in practice, so it would be very helpful to provide carefully reviewed scripts or recipes for these. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] reloptions - RELOPT_KIND_ALL
Zdenek Kotala wrote: Alvaro Herrera píše v pá 23. 01. 2009 v 11:04 -0300: Zdenek Kotala wrote: I attached patch which add capability to have single record for all realation kind in the reloption list. It is usefull in situation when all parameters are same for all relation kinds. Do you have an example use case for this? I use it in my space reservation patch. I going to send it soon. The default value is zero in all relkind and max value as well. It seems to me waste a space to copy and paste same lines several times. Aside from the other issues mentioned, ISTM the amount of space reservation wanted depends on the exact changes across releases that make that space reservation necessary in the first place. And depending on what those changes are, the space reservation requirements could be wildly different for each access and storage method. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Compiler warnings fix
ITAGAKI Takahiro wrote: Here is a patch to surpress compiler warnings in pg_locale.c and pg_regress.c. There are following warnings if nls is enabled: pg_locale.c: In function `pg_perm_setlocale': pg_locale.c:161: warning: assignment discards qualifiers from pointer target type and if nls is disabled: pg_locale.c:615: warning: 'IsoLocaleName' defined but not used There is also a warning in pg_regress.c: pg_regress.c: In function `wait_for_tests': pg_regress.c:1367: warning: passing arg 2 of `GetExitCodeProcess' from incompatible pointer type Which platform, which compiler, what configure options? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Sun, Jan 25, 2009 at 12:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: Particularly with regard to hot standby, which by any sane reading was not close to being committable on 1 November (a fortiori from the fact that it's *still* not committable despite large amounts of later work). While I haven't follwed every detail of this patch set, I'm not quite sure I see that as being very fair to Simon. Simon has put a lot of time into Hot Standby and has followed the pseudo-defacto community process from design through what he believes to be near-completion; he can't be sure of completion until someone reviews his work. Honestly, I'm not trying to marginalize your effort reviewing other patches, and I know everyone is busy. Hell, as much as I'd love to have this feature, I too have been unable to find enough time to review Simon's stuff. However, over the past few months, I seem to recall seeing Simon submit countless requests for review and regardless of whether it was completely ready to go November 1st or not, at this time I don't think anyone but Simon has a complete view of what his patch(es) even look like. Yet, albeit with almost no review from the committers, Simon has continually worked through testing, revising his patches, and requesting information and suggestions from the community. Frankly Tom, I don't know of anyone in the community with as much experience in the recovery code as you and Simon. So, any of the major edge case problems will probably only be found by you regardless of how many of us review Simon's work. I do know that this is a feature which a large number of Postgres users really want and were counting on being in 8.4. Looking forward, if no one wanted to review these patches in November, and seemingly no one wants to review them now, how can we expect this to change for 8.5? Can anyone point out something Simon did wrong in this process? -- Jonah H. Harris, Senior DBA myYearbook.com
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote: Particularly with regard to hot standby, which by any sane reading was not close to being committable on 1 November (a fortiori from the fact that it's *still* not committable despite large amounts of later work). I've wasted much time in two main areas in recent months: The first area I've wasted time on is patch review. Had I known that some patches on the queue would not be properly reviewed and that time was going to be a factor then I would not have wasted a single second on patch review. My main review item has been Sync Replication, which I did because of Core's statement that we would include that feature in this release. It would appear that my time on that has been both pointless and damaging to my own situation. The second is in developing an answer to the FATAL errors problem, which I asked a direct question on and received a very specific answer: http://archives.postgresql.org/pgsql-hackers/2008-09/msg01809.php I spent two weeks working out a difficult solution to that, a week subsequently arguing with Heikki (possibly longer) about whether it was necessary and then two weeks refactoring it back out again when you were silent. From my side, I've spent time and money supporting your views and decisions, without argument with you, though most would not see that. I learned on Jan 12 that you'd been too busy to consider this work and that a priori you argued for patch rejection. It would be much simpler if I had badly screwed up or if we'd found an awkward showstopper. That hasn't happened. In fact, I've done the full dance so far. I fail to see how rejecting unreviewed patches provides benefit for users, developers or sponsors. How does de-committing from features voted most-popular or promised by core sit with users? How will developers react when they know that helping in patch review assists their own demise? How will sponsors react when they see another set of patches rejected? If there is a problem with timing it's because we need more planning, scheduling and prioritisation of resources. But it isn't possible to do it retrospectively without causing problems, and those problems may be worse than slipping a month, or three. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Simon has put a lot of time into Hot Standby and has followed the pseudo-defacto community process from design through what he believes to be near-completion; he can't be sure of completion until someone reviews his work. I think this is a fair critique. Yet, albeit with almost no review from the committers, Simon has continually worked through testing, revising his patches, and requesting information and suggestions from the community. There really was not a lot of review from mid-October through the end of the year. That is partly because Simon was out of commission for about three weeks and did not respond (in particular) to several requests to separate ICfR from HS. That having been said, since this is such an important feature to so many people, I think it would have been better if more effort could have been put into doing what additional reviewing was still possible. However, since the turn of the year, it looks to me like Heikki has actually put quite a bit of time into reviewing and responding to issues. Still, I agree that if there's anything we should be putting our effort into as a community right now, it's this feature. If we got Hot Standby in the next release and everything else in the CommitFest got bumped, I think a lot of people would consider that a good trade (though probably not the authors of the patches that got bumped). For example, I would much rather see Tom revert the updatable views patch and work on this than spend the next week fixing updatable views. At a minimum, I think the following patches from the CommitFest wiki should be returned with feedback or rejected: 1. SE-PostgreSQL. We handled this one badly, but there's not enough time to fix it now. 8.5. 2. rmgr hooks and contrib/rmgr_hook. Reject because Tom and Heikki don't believe it's the right approach. Need better use cases. 3. Synchronous log-shipping replication. We handled this one well, but it's not in good enough shape. 8.5. 4. pg_upgrade script. I haven't heard much about this in a while... I am doubtful that it is production-quality, but maybe I'm wrong? 5. Reducing some DDL Locks to ShareLock. No activity in a long time, no time to wait for this to be finished. 8.5. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, conflict resolution
On Sun, 2009-01-25 at 16:19 +, Simon Riggs wrote: On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Ok, then I think we have a little race condition. The startup process doesn't get any reply indicating that the target backend has processed the SIGINT and set the cached conflict LSN. The target backend might move ahead using the old LSN for a little while, even though the startup process has already gone ahead and replayed a vacuum record. Another tiny issue is that it looks like a new conflict LSN always overwrites the old one. But you should always use the oldest conflicted LSN in the checks, not the newest. That makes it easier, because it is either not set, or it is set and does not need to be reset as new conflict LSNs appear. I can see a simple scheme emerging, which I will detail tomorrow morning. Rather than signalling, we could use a hasconflict boolean for each proc in a shared data structure. It can be read without spinlock, but should only be written while holding spinlock. Each time we read a block we check if hasconflict is set. If it is, we grab spinlock, recheck if it is set, if so read the conflict details, clear the flag and drop the spinlock. The aim of this type of conflict resolution was to reduce the footprint of users that would be effected and defer it as much as possible. We've spent time getting the latestCompletedXid, but we know deriving that value is very difficult in the btree case at least. So what I would like to do is pass the relid of a conflict across as well and use that to reduce the footprint, now that we are performing the test inside the buffer manager. We would keep a relid cache with a very small number of relids, perhaps just one, maybe as many as 4 or 8, so that we can fit relids and associated LSNs in a single cache line. We can match the relid using a simple for loop, which we know is well optimised when there is no dependency between the elements of the loop and the loop has a compile-time fixed number of iterations. I would be inclined to make this a separate shared memory area rather than try to weld that onto PGPROC. We could index that using backendid. If the relid cache overflows, we just apply a general LSN value. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote: Any release with big features in it will take longer, whether you wait a year, or not. Well, big features that land early in the release cycle don't delay the release. Just the ones that are submitted in the last commit fest. Has that ever happened? :-) I don't think its chance we get big patches in last commit fest. No sponsor I ever met was happy to both sponsor and to wait. The two seem mutually exclusive. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Simon Riggs wrote: On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote: Well, big features that land early in the release cycle don't delay the release. Just the ones that are submitted in the last commit fest. Has that ever happened? :-) I don't think its chance we get big patches in last commit fest. No sponsor I ever met was happy to both sponsor and to wait. The two seem mutually exclusive. How come it works for Linux? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Jonah H. Harris wrote: Looking forward, if no one wanted to review these patches in November, I did, and many others were active in the discussion too. and seemingly no one wants to review them now, I do. Thank you for your appreciation :-(. how can we expect this to change for 8.5? Can anyone point out something Simon did wrong in this process? Not really, except maybe started 6 months too late. Big patches simply take a long time to mature. Looking back at the timeline for hot standby, it doesn't look unreasonable at all: September: First discussion about the user-visible behavior, transaction isolation etc. Big architectural decisions are made, like where that snapshots are taken. Infrastructure changes for recovery patch is reviewed, and pushed to next commitfest because of issues. Discussion started on how to handle (heavy-weight) locks. October: Discussion on various administration commands, how to handle b-tree splits, and some other stuff. More discussion on how to derive snapshots. First complete hot standby patch is submitted. A patch to change the way subtransactions is submitted, and committed. November: Various people test and review the patch, including Mark Kirkwood and Pavan Deolasee. Bugs are found and fixed. Decision is made that SIGINT should kill an idle-in-transaction transaction as well. December: Discussion on slotids and the B-tree killed items problem. January: Major changes; slotids eliminated, conflict resolution mechanism rewritten. RestoreBkpBlocks refactoring committed. More bugs in conflict resolution unearthed. Discussion on adding a GUC. There has been steady progress, starting from basic design discussions and decisions, moving on to implementation details. Progress never stalled for a long time. I can see no wrongdoing on Simon's part, and there is also no grounds to say that the community has neglected this patch. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Robert Haas robertmh...@gmail.com wrote: Still, I agree that if there's anything we should be putting our effort into as a community right now, it's this feature. If we got Hot Standby in the next release and everything else in the CommitFest got bumped, I think a lot of people would consider that a good trade Based on discussions at Milwaukee BarCamp, improvements in replication would do more to break down barriers to migration to PostgreSQL than anything else. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Robert Haas wrote: At a minimum, I think the following patches from the CommitFest wiki should be returned with feedback or rejected: 1. SE-PostgreSQL. We handled this one badly, but there's not enough time to fix it now. 8.5. Unacceptable. Please make it clear how many items should be fixed/reworked before rejecting it due to the lack of time, at least. (I'm optimistic for its design and quality.) In addition, why does it impossible to be worked in parallel? Maybe, I cannot contribute Simon's work, but I can fix/rework for SE-PostgreSQL in parallel, with my highest priority. As I noted in another message, the current trend strongly requires availability and security for the platform of SaaS/PaaS or cloud-computing. So, we should also include SE-PostgreSQL within v8.4. I believe it makes PostgreSQL more widespreadly applied. 2. rmgr hooks and contrib/rmgr_hook. Reject because Tom and Heikki don't believe it's the right approach. Need better use cases. 3. Synchronous log-shipping replication. We handled this one well, but it's not in good enough shape. 8.5. IMO, this feature is also key factor for availablity. If author can have enough activity, we should not give up right now... 4. pg_upgrade script. I haven't heard much about this in a while... I am doubtful that it is production-quality, but maybe I'm wrong? 5. Reducing some DDL Locks to ShareLock. No activity in a long time, no time to wait for this to be finished. 8.5. ...Robert -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Mon, 2009-01-26 at 12:05 -0300, Alvaro Herrera wrote: Simon Riggs wrote: On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote: Well, big features that land early in the release cycle don't delay the release. Just the ones that are submitted in the last commit fest. Has that ever happened? :-) I don't think its chance we get big patches in last commit fest. No sponsor I ever met was happy to both sponsor and to wait. The two seem mutually exclusive. How come it works for Linux? Not sure. How come up it happens here is a more relevant question. I can only tell you that people only get their cheque books out when they are certain somebody else will not do it for them for free. This happened to you also, or did you have this terrible desire for CRC checks on data blocks quite late in release cycle? If there hadn't been blockers you'd be being lightly barbecued on the patch queue also. :-) -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs si...@2ndquadrant.com writes: I fail to see how rejecting unreviewed patches provides benefit for users, developers or sponsors. Nobody has suggested rejecting either sync replication or standby database. The debate here is over whether to commit into 8.4 or into 8.5. Put another way, the choice here is whether to have a half-baked delayed 8.4 release in 6 months or a polished on-time 8.5 release in 12 months. Either way the feature ships and on a not terribly different timeline either. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Not really, except maybe started 6 months too late. Big patches simply take a long time to mature. Looking back at the timeline for hot standby, it doesn't look unreasonable at all: September: First discussion about the user-visible behavior, transaction isolation etc. Is that right? I had the impression Simon had started working on it previously. If so then given that feature freeze was scheduled to be November 1st starting in September does seem like it was more than a bit unrealistic. Large patches should really be targeted towards the *first* commitfest of the cycle, not the last. Targeting the last is putting all your eggs in one basket as far as getting everything right the first time. Someone else asked why this works for Linux. The reason it works is because work is *always* committed early in the release cycle. The first couple weeks of the release cycle are the *only* time significant patches are taken. The entire rest of the cycle is spent in feature freeze with development happening exclusively on subsystem maintainer's private branches. When the next release cycle begins subsystem maintainers send up patches for all the changes that have happened since the last cycle that they think are ready for release. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with archive_command as suggested by documentation
Peter Eisentraut wrote: I don't think that particular example is a good one since the whole point of the archive is that it should be off-server. If we're going to be exact about the example then we should give a more realistic one, like using scp. Unfortunately, there is no secure-remote-move command, so doing the above with scp would resend the whole file 3 times. I think it's better to write a script... The problem is that most people do copy the stuff straight out of the documentation. And for those that do write a separate script, there are still a lot of possibilities to get it wrong. There are only about a handful of transportation protocols appearing in practice, so it would be very helpful to provide carefully reviewed scripts or recipes for these. I'll try to come up with something, but it will take some time; an impending addition to the family currently has priority... Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark st...@enterprisedb.com writes: Put another way, the choice here is whether to have a half-baked delayed 8.4 release in 6 months or a polished on-time 8.5 release in 12 months. Either way the feature ships and on a not terribly different timeline either. This is pretty much exactly how I see it. *Hot standby is not ready*, and committing it into 8.4 isn't going to magically make that better. The earliest we are going to have a HS feature that I would trust my data to is probably ten-twelve months off. The decision we need to make now is whether that release will be called 8.4 or 8.5; in the former case meaning that all the stuff already in 8.4 will not reach users' hands for close to a year more. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
On Mon, Jan 26, 2009 at 6:48 AM, Zeugswetter Andreas OSB sIT andreas.zeugswet...@s-itsolutions.at wrote: Is that why other db's only make views updateable, that are created WITH CHECK OPTION ? Should we also follow that path ? no, the standard says that if the query expression is updatable the view is updatable -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Andrew Dunstan wrote: Something happened about 80 hours ago that caused my mingw buildfarm member (gcc 3.4.2 on Win XP Pro SP2) to hang at the check stage. It looks like it's hung in initdb. I wonder if it could be this commit: Log Message: --- Make win32 builds always do SetEnvironmentVariable() when doing putenv(). Also, if linked against other versions than the default MSVCRT library (for example the MSVC build which links against MSVCRT80), also update the cache in the default MSVCRT at the same time. I note that the change is not apparently limited to MSVC builds. The MSVC animal that runs on the same machine appears unaffected. I see one other mingw buildfarm member that is having problems that started a few days ago (yak) and another that looks like it is a few hours overdue to report, so it might also be hung (vaquita). Further to this: I see that vaquita has now reported in, and is happy. Also, I can run happily on my Vista box (vaquita is also a Vista box). I therefore suspect that we have a problem specifically with XP (both dawn_bat and yak are XP boxes). cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane t...@sss.pgh.pa.us writes: Gregory Stark st...@enterprisedb.com writes: Put another way, the choice here is whether to have a half-baked delayed 8.4 release in 6 months or a polished on-time 8.5 release in 12 months. Either way the feature ships and on a not terribly different timeline either. This is pretty much exactly how I see it. *Hot standby is not ready*, and committing it into 8.4 isn't going to magically make that better. The earliest we are going to have a HS feature that I would trust my data to is probably ten-twelve months off. The decision we need to make now is whether that release will be called 8.4 or 8.5; in the former case meaning that all the stuff already in 8.4 will not reach users' hands for close to a year more. I'm not even so much worried about the stuff already in 8.4. I'm worried about the queue for the first 8.5 commitfest. Nobody's going to be reviewing new patches until 8.4 is out which will leave all that stuff sitting in limbo for months. Worse, it'll take months to sort through it and nobody will be able to work on anything which depends on stuff in that queue for, uh, 2*months. This is stuff like sync replication, join removal, the posix_fadvise for WAL replay, index-only scans, etc. If this stuff isn't ready to be committed until next September we'll be back in the same boat again. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] foreign_data test fails with non-C locale
Zdenek Kotala wrote: Andrew Dunstan píše v pá 23. 01. 2009 v 23:57 -0500: Zdenek Kotala wrote: Andrew Dunstan píše v pá 09. 01. 2009 v 12:16 -0500: Sure, we can easily have buildfarm's initdb step set any locale (and encoding, for that matter) we like. That's a simple change. Will be possible to set more locales and run tests without recompilation on all of them? For example I have installed all Solaris'es locales on my animal, but currently it means that I need perform whole cycle for each locale. I'm working on this. Yes, you will be able to specify a list of locales to check. For each locale the following tests will be run: installcheck, pl-installcheck, and contrib-installcheck. thanks Example run with locales C, en_US.utf8 and french: http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=dungbeetledt=2009-01-26%2012:44:01 cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: Gregory Stark st...@enterprisedb.com writes: Put another way, the choice here is whether to have a half-baked delayed 8.4 release in 6 months or a polished on-time 8.5 release in 12 months. Either way the feature ships and on a not terribly different timeline either. This is pretty much exactly how I see it. *Hot standby is not ready*, and committing it into 8.4 isn't going to magically make that better. The earliest we are going to have a HS feature that I would trust my data to is probably ten-twelve months off. So can you give us an idea of what parts of the code are in need of rethinking etc? I assume you've looked at it now if you can estimate it's going to take another 10 -12 months? I ask because I've only seen Heikki doing any in depth public review, and he suggested a couple of weeks for the outstanding issues he's aware of. If there are fundamental problems which will take 10 - 12 months to resolve to our normal standards, then I do believe 8.5 would be more appropriate. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Jonah H. Harris wrote: how can we expect this to change for 8.5? Can anyone point out something Simon did wrong in this process? Not really, except maybe started 6 months too late. Big patches simply take a long time to mature. Yeah, exactly. When we decided at PgCon that we needed to accept replication into the core, we thought that there was a realistic chance of it being in 8.4 if full-tilt development started *immediately*. For one reason or another nothing much happened until September. Simon's certainly been working hard on it since then, but it just takes longer than four months for something like this to happen. The alternatives that we realistically have got are: * ship 8.4 now (ie before the summer) without HS, plan for HS in 8.5 * delay 8.4 to late summer and ship it with an unpolished HS feature * delay 8.4 long enough to have a stable, polished HS feature; I claim this will mean it's not out before Christmas. I think anyone who thinks alternatives 2 or 3 will happen faster than sketched here is living in dreamland. So my feeling is that prudent project management would suggest that HS go into 8.5 near the beginning of that development cycle, rather than trying to push it into 8.4. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Tom Lane t...@sss.pgh.pa.us wrote: Gregory Stark st...@enterprisedb.com writes: Put another way, the choice here is whether to have a half-baked delayed 8.4 release in 6 months or a polished on-time 8.5 release in 12 months. Either way the feature ships and on a not terribly different timeline either. This is pretty much exactly how I see it. *Hot standby is not ready*, and committing it into 8.4 isn't going to magically make that better. The earliest we are going to have a HS feature that I would trust my data to is probably ten-twelve months off. The decision we need to make now is whether that release will be called 8.4 or 8.5; in the former case meaning that all the stuff already in 8.4 will not reach users' hands for close to a year more. What about a compromise solution: release 8.4 now, then focus on wrapping up the big ticket items that didn't make it into 8.4 into a quick (as possible) 8.5 release. This means no fests. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure mmonc...@gmail.com wrote: What about a compromise solution: release 8.4 now, then focus on wrapping up the big ticket items that didn't make it into 8.4 into a quick (as possible) 8.5 release. This means no fests. That would depend on timing then. Trying to get people to upgrade to 8.4 is going to be difficult if they're waiting on Hot Standby, which means less in-the-field testing of the 8.4 code base until the 8.5 release. Similarly, if we're looking at a quick 8.5 around September/October (having no commit fests), that means it will probably be early 2011 for 8.6, which is fairly unacceptable for the other patches currently in the queue. -- Jonah H. Harris, Senior DBA myYearbook.com
Re: [HACKERS] 8.4 release planning
Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: This is pretty much exactly how I see it. *Hot standby is not ready*, So can you give us an idea of what parts of the code are in need of rethinking etc? I assume you've looked at it now if you can estimate it's going to take another 10 -12 months? No, I'm just estimating that based on the amount of design churn that's still going on according to the mailing list discussions. I haven't looked at the code at all. (If you expect me to sign off on it you can figure it'll be a couple of months even for that to happen...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: This is pretty much exactly how I see it. *Hot standby is not ready*, So can you give us an idea of what parts of the code are in need of rethinking etc? I assume you've looked at it now if you can estimate it's going to take another 10 -12 months? No, I'm just estimating that based on the amount of design churn that's still going on according to the mailing list discussions. I haven't looked at the code at all. (If you expect me to sign off on it you can figure it'll be a couple of months even for that to happen...) Well there is one of the problems imho - the project is getting too big and the patches are getting too complex for us to be able to rely on you to sign off on every feature or major patch. We need put more trust in the judgement of the other committers and senior developers otherwise the project will simply bottleneck as you get more and more overworked. /me hopes that comes across as intended :-) -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with archive_command as suggested by documentation
On Mon, Jan 26, 2009 at 04:40:12PM +0100, Albe Laurenz wrote: Peter Eisentraut wrote: I don't think that particular example is a good one since the whole point of the archive is that it should be off-server. If we're going to be exact about the example then we should give a more realistic one, like using scp. Unfortunately, there is no secure-remote-move command, so doing the above with scp would resend the whole file 3 times. I think it's better to write a script... The problem is that most people do copy the stuff straight out of the documentation. And for those that do write a separate script, there are still a lot of possibilities to get it wrong. There are only about a handful of transportation protocols appearing in practice, so it would be very helpful to provide carefully reviewed scripts or recipes for these. I'll try to come up with something, but it will take some time; an impending addition to the family currently has priority... You understand that this means you need to provide blue-elephant-themed photos, right? Congratulations! :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Jonah H. Harris jonah.har...@gmail.com writes: On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure mmonc...@gmail.com wrote: What about a compromise solution: release 8.4 now, then focus on wrapping up the big ticket items that didn't make it into 8.4 into a quick (as possible) 8.5 release. This means no fests. That would depend on timing then. Trying to get people to upgrade to 8.4 is going to be difficult if they're waiting on Hot Standby, which means less in-the-field testing of the 8.4 code base until the 8.5 release. [ deja vu... ] Just like no one was going to bother upgrading to 8.3 because what they wanted wouldn't be there till 8.4, and the similar claims we heard about 8.2 and 8.1 before that ... Similarly, if we're looking at a quick 8.5 around September/October (having no commit fests), that means it will probably be early 2011 for 8.6, which is fairly unacceptable for the other patches currently in the queue. Right, one of the major considerations here is allowing other development to get started again (and not be looking at two years wait to see the light of day). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Jonah H. Harris jonah.har...@gmail.com wrote: On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure mmonc...@gmail.com wrote: What about a compromise solution: release 8.4 now, then focus on wrapping up the big ticket items that didn't make it into 8.4 into a quick (as possible) 8.5 release. This means no fests. That would depend on timing then. Trying to get people to upgrade to 8.4 is going to be difficult if they're waiting on Hot Standby, which means less in-the-field testing of the 8.4 code base until the 8.5 release. Similarly, if we're looking at a quick 8.5 around September/October (having no commit fests), that means it will probably be early 2011 for 8.6, which is fairly unacceptable for the other patches currently in the queue. I think that's baked in the cards no matter what. IMO, the issue is that code to review is building up faster than it is getting reviewed, so maybe a festless release is nesessary to flush out the difference (along with the other held over patches from 8.4). The other thing going on here is that HS is extremely important to the project. If it was up to me, 100% effort would be geared to getting it out as quickly as possible. I'm just looking for a way to get HS in the hands of people as quickly as possible that is fair to everybody. (I would actually prefer to hold 8.4 for it, but that's starting to sound unrealistic based on Tom's comments). merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 11:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: That would depend on timing then. Trying to get people to upgrade to 8.4 is going to be difficult if they're waiting on Hot Standby, which means less in-the-field testing of the 8.4 code base until the 8.5 release. [ deja vu... ] Just like no one was going to bother upgrading to 8.3 because what they wanted wouldn't be there till 8.4, and the similar claims we heard about 8.2 and 8.1 before that ... I'm not trying to be an alarmist, I'm just stating what I saw when I was @ EDB. Customers, especially those with large databases or small admin teams, would definitely wait for features before upgrading. Some people waited specifically for HOT or features that would benefit them specifically. My only gripe with a small window between 8.4 and 8.5 was just that I believe people would be more likely to wait until 8.5 rather than upgrading twice in the same year. Though, as I generally like people to be using the latest version of PG, I'd certainly be happy to be wrong on this. Similarly, if we're looking at a quick 8.5 around September/October (having no commit fests), that means it will probably be early 2011 for 8.6, which is fairly unacceptable for the other patches currently in the queue. Right, one of the major considerations here is allowing other development to get started again (and not be looking at two years wait to see the light of day). Agreed. -- Jonah H. Harris, Senior DBA myYearbook.com
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Tom Lane t...@sss.pgh.pa.us wrote: Jeff Davis pg...@j-davis.com writes: There you see a snapshot of the table that never existed. Either the snapshot was taken before the UPDATE, in which case i=3 should be included, or it was taken after the UPDATE, in which case i=4 should be included. So atomicity is broken for WHERE. This assertion is based on a misunderstanding of what FOR UPDATE in read-committed mode is defined to do. It is supposed to give you the latest available rows. Well, technically it's violating the Isolation part of ACID, not the Atomicity, since the UPDATE transaction will either commit or roll back in its entirety, but another transaction can see it in an intermediate (partially applied) state.[1] I guess the issue of whether this violation of ACID properties should be considered a bug or a feature is a separate discussion, but calling it a feature seems like a hard sell to me. -Kevin [1] http://en.wikipedia.org/wiki/ACID -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pgtune: postgresql.conf wizard
pgtune is now on pgFoundry: http://pgfoundry.org/projects/pgtune/ I just released an update to there, and attached here for the archives is that same code. Progress since the last code drop to this list: -All of the necessary integer handling needed was extracted from guc.c and implemented, in particular memory unit handling that matches the server for input and SHOW-style output. One feature that's already allowed is it can sort out: Input: shared_buffers='32768' tuned: shared_buffers='256MB' Notice those are the same value, and then not do anything to that setting. This will make round trips through pgtune much less disruptive. And all it should take to eventually add 8.1 compatibility (before memory units like kB worked) is turning off my SHOW-style implementation and just working with the integer values. -Lots of code cleanup based on all the hiccups reported back to me Primary open issues at this point: -Update the model to limit shared memory use on Windows -Degrade more gracefully on low memory systems -Optional module to estimate shared memory use and output sysctl suggestions The next time I get a block of time to work on this I'll swat all those, getting past all the memory unit stuff was the main thing I've been stuck on recently. There are a few TODO items tagged in the code to finish cleaning up too. I'm on track to have everything I wanted in a 1.0 (only targeting 8.4) ready by Feb 1 in time for beta. Things I was hoping for some input on: -Using default_stats_target=100 doesn't seem controversial anymore, which makes we wonder if it makes sense to restore the original DW suggestion of 400 Josh suggested? -There was some talk about making constraint_exclusion have an additional smart mode aimed at inheritance. Not sure where that's at, but it would impact the model were it available. I also made a number of Python style improvements based on review from Nathan. If you're not interested in that topic you should certainly stop reading here as that's all I cover below. On Fri, 5 Dec 2008, Nathan Boley wrote: - The windows specific try block ( line 16 ) raises a ValueError... I rewrote that whole mess based on all the feedback I got and I hope this interface is much more robust. - from ctypes import * ( 18 ) makes the block difficult to read and pollutes the namespace. Noted, will try to reduce what it calls when I do my final Windows testing. Am not sure exactly what it needs so I won't know if I broke it trying to reduce the import use until then. - all classes ( 58, 135, 205 ) are 'old-style' classes. Converted to new style ones. - The doc strings ( 59, 136, 206 ) don't follow standard conventions, described here http://www.python.org/dev/peps/pep-0257/. Improved doc strings are on my list for later cleanup, haven't gotten to it yet. - Tuple unpacking doesn't require the tuple boundaries ( 446 and others ). ie, options, args = ReadOptions() works. Fixed a bunch of those, if you notice any I missed please point them out. overloading __str__ and __repr__ are the python ways to return string representations of an object Converted my toString call to use __str__ and str(). Have you considered the use of properties http://www.python.org/download/releases/2.2.3/descrintro/#property ? I wasn't fully aware of properties before. Now that I am, I find I dislike them. For this type of app, there doesn't seem to be much benefit for the extra code I'd put in. The main reason I provided those getter functions rather than having things directly reference the object's fields was so I could change the underlying implementation without breaking the API. I didn't really care about making things private. I now realize that I can change the API and make the caller still think they're directly accessing the field with properties, so I just made most things pull right from the fields now. That cut out some code that was only supporting that poor data hiding. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD pgtune-0.9.0.tar.gz Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane t...@sss.pgh.pa.us writes: Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: This is pretty much exactly how I see it. *Hot standby is not ready*, So can you give us an idea of what parts of the code are in need of rethinking etc? I assume you've looked at it now if you can estimate it's going to take another 10 -12 months? No, I'm just estimating that based on the amount of design churn that's still going on according to the mailing list discussions. That's, uh, not a very useful metric. It should argue in *favour* of the patch being near completion that the patch is getting feedback and Simon is rapidly taking such suggestions to heart and making changes responsively. And incidentally most of the changes Heikki's been making have been simplifications. Every bit which is simplified is one fewer area to find further problems in. It's not like he's been finding problems which require new code which in turn could have more problems. I haven't looked at the code at all. (If you expect me to sign off on it you can figure it'll be a couple of months even for that to happen...) Well, given that you're basing your judgements on the mailing list traffic and haven't read the code I guess we have to go on Heikki's judgement that there are about two known weeks of work and possibly more afterwards. The real question is how long will the beta period be with or without Hot Standby? How many users have been pounding on it so far and how many more will we get in beta? If we assume it's about a month before the patch is committed, and then the beta period is an extra month or two long (on top of what I assume will be about a month) then we're looking at 3-4 months before 8.4 or about May. I fear that 3-4 months will already be long enough to back up development on 8.5 and cause a big queue for the first commitfest. Frankly, at 3 months I think feature freeze has _already_ gone on too long, I don't think we need the doom scenario of 10-12 months before we create a serious problem again. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, conflict resolution
Simon Riggs wrote: Rather than signalling, we could use a hasconflict boolean for each proc in a shared data structure. It can be read without spinlock, but should only be written while holding spinlock. Each time we read a block we check if hasconflict is set. If it is, we grab spinlock, recheck if it is set, if so read the conflict details, clear the flag and drop the spinlock. Yeah, that seems workable. The aim of this type of conflict resolution was to reduce the footprint of users that would be effected and defer it as much as possible. We've spent time getting the latestCompletedXid, but we know deriving that value is very difficult in the btree case at least. So what I would like to do is pass the relid of a conflict across as well and use that to reduce the footprint, now that we are performing the test inside the buffer manager. I agree that would be useful, but I'd prefer to keep it simple for now... -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dave Page wrote: On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: This is pretty much exactly how I see it. *Hot standby is not ready*, So can you give us an idea of what parts of the code are in need of rethinking etc? I assume you've looked at it now if you can estimate it's going to take another 10 -12 months? No, I'm just estimating that based on the amount of design churn that's still going on according to the mailing list discussions. I haven't looked at the code at all. (If you expect me to sign off on it you can figure it'll be a couple of months even for that to happen...) Well there is one of the problems imho - the project is getting too big and the patches are getting too complex for us to be able to rely on you to sign off on every feature or major patch. We need put more trust in the judgement of the other committers and senior developers otherwise the project will simply bottleneck as you get more and more overworked. About another issued patch, SE-PostgreSQL has been also said large and complex. However, I cannot believe the reputation reflects the correct shape. As I noted in another message, the main patch of SE-PostgreSQL changes 110 files, inserts 9813 lines, deletes 16 lines and modifies 924 lines. It might seem the patch is large/complex. However, 8207 of 9813 lines are on src/backend/security/* and src/include/security/* because of its modular design. It means most of patch simply adds new files. The 504 of 924 lines modifications are due to a new field in pg_attribute system catalog, so it is obvious change. Rest of relatively-large changes are 293 lines in copy.c and 209 lines in execMain.c. I don't think it is too complex patch to hesiate reviewing. So, I believe we can move it to ready to commit state within reasonable term, as long as we make it progress. Thanks, [kai...@fedora10 ~]$ diffstat sepostgresql-sepgsql-8.4devel-3-r1467.patch configure | 113 + configure.in | 13 src/Makefile.global.in|1 src/backend/Makefile |7 src/backend/access/common/heaptuple.c | 35 src/backend/access/common/reloptions.c| 22 src/backend/access/common/tupdesc.c | 12 src/backend/access/heap/heapam.c | 19 src/backend/access/heap/tuptoaster.c | 19 src/backend/bootstrap/bootparse.y | 13 src/backend/bootstrap/bootstrap.c |8 src/backend/catalog/Makefile |1 src/backend/catalog/aclchk.c |2 src/backend/catalog/catalog.c |4 src/backend/catalog/heap.c| 91 ! src/backend/catalog/index.c | 16 src/backend/catalog/pg_aggregate.c|3 src/backend/catalog/pg_largeobject.c |5 src/backend/catalog/pg_proc.c |6 src/backend/catalog/toasting.c|3 src/backend/commands/cluster.c| 11 src/backend/commands/copy.c | 293 +++! src/backend/commands/dbcommands.c | 20 src/backend/commands/functioncmds.c | 29 src/backend/commands/lockcmds.c |3 src/backend/commands/proclang.c |6 src/backend/commands/tablecmds.c | 25 src/backend/commands/trigger.c| 25 src/backend/executor/execJunk.c |6 src/backend/executor/execMain.c | 209 +++ src/backend/executor/execQual.c |4 src/backend/executor/execScan.c | 40 src/backend/executor/execTuples.c | 19 src/backend/executor/execUtils.c | 10 src/backend/executor/functions.c |6 src/backend/executor/nodeAgg.c|5 src/backend/executor/nodeMergejoin.c |2 src/backend/executor/nodeSubplan.c|4 src/backend/executor/nodeWindowAgg.c |4 src/backend/executor/spi.c|4 src/backend/libpq/be-fsstubs.c| 16 src/backend/nodes/copyfuncs.c | 30 src/backend/nodes/equalfuncs.c| 22 src/backend/nodes/outfuncs.c | 34 src/backend/nodes/readfuncs.c | 43 src/backend/optimizer/plan/createplan.c |6 src/backend/optimizer/plan/planner.c |1 src/backend/optimizer/util/clauses.c |5 src/backend/optimizer/util/relnode.c |1 src/backend/parser/analyze.c | 46 src/backend/parser/gram.y | 64 ! src/backend/parser/parse_target.c | 64 ! src/backend/postmaster/postmaster.c | 43 src/backend/rewrite/rewriteHandler.c |3
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On Mon, 2009-01-26 at 10:48 -0600, Kevin Grittner wrote: I guess the issue of whether this violation of ACID properties should be considered a bug or a feature is a separate discussion, but calling it a feature seems like a hard sell to me. I think I understand the other perspective on this now: SELECT FOR UPDATE/SHARE is an entirely separate command that is more similar (in transactional semantics) to UPDATE than to SELECT. In fact, it's probably most similar to UPDATE ... RETURNING, which will give the same result (that breaks atomicity or isolation, depending on your point of view), which is correct for READ COMMITTED isolation level. Because the command begins with SELECT, I would expect it to follow the rules of SELECT with the side effect of locking. I would think that the standard would have something to say about this*. I certainly don't think it's intuitive behavior. Regards, Jeff Davis. *: It appears that SELECT ... FOR UPDATE is not in the standard, which would indicate to me that the SELECT statement should still behave according to SELECT isolation/snapshot rules. But when I guess about the standard, I'm usually wrong. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Jeff Davis pg...@j-davis.com wrote: In fact, it's probably most similar to UPDATE ... RETURNING, which will give the same result (that breaks atomicity or isolation, depending on your point of view), which is correct for READ COMMITTED isolation level. READ COMMITTED is not supposed to be able to view the work of a concurrent transactions as PARTLY applied and PARTLY committed, which is what's happening here. If one statement in a READ COMMITTED transaction sees the uncommitted view of the data and the next statement sees the committed view, that's compliant. It may not surprise someone who is intimately familiar with PostgreSQL internals for a single SELECT statement to see PART of a transactions work, but it would surprise most users, and is certainly not compliant with the standard. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... (If you expect me to sign off on it you can figure it'll be a couple of months even for that to happen...) Well there is one of the problems imho - the project is getting too big and the patches are getting too complex for us to be able to rely on you to sign off on every feature or major patch. We need put more trust in the judgement of the other committers and senior developers otherwise the project will simply bottleneck as you get more and more overworked. Well, actually, Bruce and I are pretty much leaving it to Heikki's judgement as to when this patch will be ready to commit. He's spent far more time on it than either of us. The reason I started this thread is not to debate when it'll be ready but whether we are prepared to delay 8.4 for however long it will take, realizing that optimistic estimates are very likely to be wrong. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 11:44 AM, Jonah H. Harris jonah.har...@gmail.com wrote: On Mon, Jan 26, 2009 at 11:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: That would depend on timing then. Trying to get people to upgrade to 8.4 is going to be difficult if they're waiting on Hot Standby, which means less in-the-field testing of the 8.4 code base until the 8.5 release. [ deja vu... ] Just like no one was going to bother upgrading to 8.3 because what they wanted wouldn't be there till 8.4, and the similar claims we heard about 8.2 and 8.1 before that ... I'm not trying to be an alarmist, I'm just stating what I saw when I was @ EDB. Customers, especially those with large databases or small admin teams, would definitely wait for features before upgrading. Some people waited specifically for HOT or features that would benefit them specifically. My only gripe with a small window between 8.4 and 8.5 was just that I believe people would be more likely to wait until 8.5 rather than upgrading twice in the same year. Though, as I generally like people to be using the latest version of PG, I'd certainly be happy to be wrong on this. We've got folks working on the upgrade to 8.3; 8.4 isn't on our radar yet, particularly in view of the fact that it's getting pretty nebulous when 8.4 will be production-worthy :-). It would surprise me not at all if we never got around to an 8.4 deployment, irrespective of whether we hold off another 6 months to get more of hot standby into it or not. Based on that metric, I suppose I ought to prefer for us to get 8.4 out the door more quickly, and start seeing work progress on the 8.5 backlog, which would presumably include the two large O/S patches (e.g. - hot standby, SE-PG). I suppose there are two, possibly 3 directions to consider, with respective merits and demerits: 1. Suppose we cut things off now, and say push that stuff to 8.5... - Simon and KaiGai will be justifiably angry since they *have* been doing the right kinds of things, and were expecting to see their material in 8.4. - On the other hand, their work would be among the first considered for 8.5, so that they'd not be behind the way they were for 8.4. - We *won't* irritate the somewhat numerous set of people with patches already awaiting 8.5,http://wiki.postgresql.org/wiki/CommitFest_2009-First and allow those patches to bit rot in the interim. 2. Alternatively, we may press on... - We may put off 8.4 for an extra 4-6 months. - Everyone loses the utility of the features already committed in 8.4 for that extra period of time. - We irritate everyone that was accepting that their contributions would be waiting until the first 8.5 commitfest. 3. There's always the possibility of a worst of all worlds. - All the irritations and losses of #2 - For whatever reason, we don't get usable hot standby / SEPostgreSQL at the end of it. I'm getting increasingly scared of #2 and #3. From a purely selfish perspective, neither of these features are ones I was expecting to be using in the short or medium term (I don't actually see a use case for SELinux, myself). I'll note that as my bias, but I don't think that invalidates the analysis of the 3 directions. -- http://linuxfinances.info/info/linuxdistributions.html Robert Benchley - Drawing on my fine command of the English language, I said nothing. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
I think that's baked in the cards no matter what. IMO, the issue is that code to review is building up faster than it is getting reviewed, so maybe a festless release is nesessary to flush out the difference (along with the other held over patches from 8.4). How is that going to help? People will still write new code in the meanwhile and some of it may be better or more important than the stuff that doesn't get committed to 8.4. Artificially saying that nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't seem to me to solve anything. I think we've actually done a very fine job reviewing code for 8.4. Take a look at how many patches were reviewed and committed. The problem is that we didn't review the big ones. But at least a part of that is due to some oversights in assigning reviewers, rather than an absolute lack of manpower. The other thing going on here is that HS is extremely important to the project. If it was up to me, 100% effort would be geared to getting it out as quickly as possible. I'm just looking for a way to get HS in the hands of people as quickly as possible that is fair to everybody. (I would actually prefer to hold 8.4 for it, but that's starting to sound unrealistic based on Tom's comments). +1. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas wrote: The other thing going on here is that HS is extremely important to the project. If it was up to me, 100% effort would be geared to getting it out as quickly as possible. I'm just looking for a way to get HS in the hands of people as quickly as possible that is fair to everybody. (I would actually prefer to hold 8.4 for it, but that's starting to sound unrealistic based on Tom's comments). +1. +1 Too. Really, the most interesting part of 8.4 for us is sync_rep and hot_standby! Best regards, -- Laurent COUSTET -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 12:47 -0500, Robert Haas wrote: How is that going to help? People will still write new code in the meanwhile and some of it may be better or more important than the stuff that doesn't get committed to 8.4. Artificially saying that nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't seem to me to solve anything. I think we've actually done a very fine job reviewing code for 8.4. We have. This release has been better than any other release in recent memory. We have had more reviewers, and a more transparent process. That is a great thing and should only stand to improve for the next release. Take a look at how many patches were reviewed and committed. The problem is that we didn't review the big ones. I am not sure this is entirely true. SE Postgres for example has been in constant review by Bruce and Hot Standby has been in constant review by Heikki. I think specifically these two patches suffer from two very distinct problems. 1. This community in general doesn't understand SE Linux/Postgres. It also struggles with the very Linux centric nature of it. 2. Hot Standby has two very distinct problems of its own. A. It is a very much buzzword compliant feature. B. It *appears* to be very much in dynamic development. The chatter on the list alone is enough to make conservative hackers very nervous. Myself included. I am personally not in favor of delaying 8.4. To me Hot Standby is patch #8762. It isn't going to make the cut because of timing. That is life. It will be one of the first in the queue for 8.5. Or put another way. Nobody would blink about cutting the new \dfU stuff in favor of getting 8.4 out on time. Why are we delaying for Hot Standby? Sincerely, Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On 2009-01-26, at 17:34, Kevin Grittner wrote: . It may not surprise someone who is intimately familiar with PostgreSQL internals for a single SELECT statement to see PART of a transactions work, but it would surprise most users, and is certainly not compliant with the standard. +1000 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On Mon, 2009-01-26 at 11:34 -0600, Kevin Grittner wrote: READ COMMITTED is not supposed to be able to view the work of a concurrent transactions as PARTLY applied and PARTLY committed, which is what's happening here. If one statement in a READ COMMITTED transaction sees the uncommitted view of the data and the next statement sees the committed view, that's compliant. It may not surprise someone who is intimately familiar with PostgreSQL internals for a single SELECT statement to see PART of a transactions work, but it would surprise most users, and is certainly not compliant with the standard. See 13.2.1: http://www.postgresql.org/docs/8.3/static/transaction-iso.html That explanation seems to be the behavior I would expect from UPDATE in read committed mode. Perhaps I'm just used to PostgreSQL -- what do other database systems do? And what does the standard say? I can't find anything in the standard that handles UPDATEs differently, so that seems to support your position. After the concurrent transaction commits, you basically have three options: 1. Get a new snapshot, and re-run the entire query to find new rows that might match the search condition that were committed between the time you took the original snapshot for UPDATE and the time that the concurrent transaction committed. 2. Blindly proceed with the original snapshot. This would mean that two concurrent updates like set i = i+1 might both see the same value, let's say 5, and both update it to 6, and then commit. However, a serial execution would produce 7. 3. Find the latest version of the same tuples you found from the original snapshot in the original search, and if they still match the search condition, update based on the new version. This is what PostgreSQL currently does. I don't think this is PostgreSQL-specific, I think non-MVCC database systems face this same choice (although the terminology would be different). #3 has a certain intuition about it (i.e. produces the expected result most of the time in simple cases), but in my opinion, that intuition only holds for UPDATE, it doesn't hold for SELECT. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. on the other hand: We held back version 4 months 7.4 for Windows, before it became apparent that there was at least a year more work to do. That was a mistake, and in many ways HS seems like a similar case. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a moving target. I've been waiting for it to settle down so that we can see if upgrade works. It was always true that pg_upgrade would be among the last patches tested; we discussed this at pgCon. == Regarding the Commitfests in general: we still haven't perfected this process. There are a number of issues with it which are hampered by technology, but a lot more by people. Here's my analysis of what's changed over the last year: 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. 2) Having the CFs improves visibility. However, as SEPostgres shows, it doesn't eliminate the ability of major committers to put off dealing with large complex patches which junior reviewers can't be assigned to. Particularly if a major reviewer claims a patch, but then doesn't seem to do a lot of review. 3) I don't feel like I got a real handle on the Round Robin Reviewers and got them processing small patches efficiently until the November review. It just took several cycles to work out how to do it, particularly given my job change this year. Better technology would also help, such as automated tracking of patch changes and when the last time a reviewer spoke up was. Currently, Dave and I have been doing these things by hand and I know we missed a lot of patches which stalled. But the main issue is (and will remain) people and procrastination. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgtune: postgresql.conf wizard
Greg, Things I was hoping for some input on: -Using default_stats_target=100 doesn't seem controversial anymore, which makes we wonder if it makes sense to restore the original DW suggestion of 400 Josh suggested? I'm going to back off from this. Following our discussion, I did some testing on DWs belonging to my clients. The amount of time required to ANALYZE 20m rows of a large text field at 400 is significant, as is the bulk-out of the stats table. Instead, I'm writing a script which increases stats for all *indexed* columns under a named size to a user-specified level. When it's ready, I'll add it to pgTune. So for DW, go for 100. Not 400, though. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] autovacuum, reloptions, and hardcoded pg_class tupdesc
Alvaro Herrera wrote: Alvaro Herrera wrote: I'm not sure that we have any use for the top level you propose; the attached patch just uses the two lower levels, and I think it fits autovacuum usage just fine. Thanks for the idea. Of course, there's no need to pass the relkind; it goes inside the pg_class tuple. Applied. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 11:36 -0800, Josh Berkus wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. Maybe. Most of the people I talk to are more interested in Recursive Queries and Parallel restore. Not that they don't see the benefit of HS of course just that they already have in place solutions for that problem. on the other hand: We held back version 4 months 7.4 for Windows, before it became apparent that there was at least a year more work to do. That was a mistake, and in many ways HS seems like a similar case. Had to read this twice. I think you mean we held back 7.4 four 4 months. Which I had actually forgotten. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. As I posted earlier I think it is mostly understanding of the technology. pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a moving target. I've been waiting for it to settle down so that we can see if upgrade works. It was always true that pg_upgrade would be among the last patches tested; we discussed this at pgCon. I thought pg_upgrade was dead. I am happy to hear it isn't. When can we see a patch that works on anything? 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. Its always something though. Its either Christmas or summer or school starting or school ending or spring break etc... Better technology would also help, such as automated tracking of patch changes and when the last time a reviewer spoke up was. Currently, Dave and I have been doing these things by hand and I know we missed a lot of patches which stalled. But the main issue is (and will remain) people and procrastination. Nod. Sincerely, Joshua D. Drake --Josh -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Josh Berkus j...@agliodbs.com wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. on the other hand: We held back version 4 months 7.4 for Windows, before it became apparent that there was at least a year more work to do. That was a mistake, and in many ways HS seems like a similar case. Not completely. HS is in much better shape than win32 was when it was pulled from 7.4...the build system wasn't even in place yet nor any of the major challenges solved (like fork/exec). HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Jeff Davis pg...@j-davis.com wrote: I don't think this is PostgreSQL-specific, I think non-MVCC database systems face this same choice (although the terminology would be different). A somewhat dated description for Sybase (it predates their support of row level locks and the related predicate locks on indexes) is here: http://manuals.sybase.com/onlinebooks/group-asarc/srv10024/sag/@Generic__BookTextView/41766;pt=41535/* Simplified, in a READ COMMITTED transaction a SELECT takes a lock which conflicts with update before reading, and holds it until the executing statement is done with that table; an UPDATE takes a lock which conflicts with any access (read or write) before it updates a row, and holds it until COMMIT or ROLLBACK. This guarantees that the SELECT doesn't see the results of an incomplete UPDATE, at the expense of taking locks, blocking, and possible serialization failures (in the form of deadlocks). -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Merlin, Not completely. HS is in much better shape than win32 was when it was pulled from 7.4...the build system wasn't even in place yet nor any of the major challenges solved (like fork/exec). HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. OK. I haven't tried it since early December; that's why I was just presenting alternatives. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 2:36 PM, Josh Berkus j...@agliodbs.com wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. if we think we will need a lot of time to get this in form, there will be no difference in the time of the release just in the numbers in which HS comes SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. it hasn't been testing by ours in different platforms (ie: ubuntu has selinux and i want to give it a try, badly enough i have never used selinux so this is new to me)... nor we have any evidence that it doesn't affect to users that doesn't have any variant of selinux (ie: windows)... the real problem here is the base of users we enough knowledge of the tool to make some usefull tests == Regarding the Commitfests in general: we still haven't perfected this process. There are a number of issues with it which are hampered by technology, but a lot more by people. Here's my analysis of what's changed over the last year: 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. +1 -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
I wrote: Simplified, in a READ COMMITTED transaction a SELECT takes a lock which conflicts with update before reading, and holds it until the executing statement is done with that table; That should have said until after the executing statement completes. Apologies, but but I just know someone would have picked up on the hole my misstatement left -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
All, 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. Actually, I'm thinking about this again, and made a mistake about the mistake. The *original plan* was that we were not going to accept any new patches for Nov-CF. Just revised patches from eariler Fests. We didn't stick to that, which is mostly why we are still reviewing now. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On Mon, 2009-01-26 at 13:50 -0600, Kevin Grittner wrote: A somewhat dated description for Sybase (it predates their support of row level locks and the related predicate locks on indexes) is here: http://manuals.sybase.com/onlinebooks/group-asarc/srv10024/sag/@Generic__BookTextView/41766;pt=41535/* Simplified, in a READ COMMITTED transaction a SELECT takes a lock which conflicts with update before reading, and holds it until the executing statement is done with that table; an UPDATE takes a lock which conflicts with any access (read or write) before it updates a row, and holds it until COMMIT or ROLLBACK. This guarantees that the SELECT doesn't see the results of an incomplete UPDATE, at the expense of taking locks, blocking, and possible serialization failures (in the form of deadlocks). That doesn't quite answer the question about UPDATE specifically. Pretend for a second that SELECT ... FOR UPDATE is like UPDATE ... RETURNING. The example in my original email can be easily changed to use UPDATE ... RETURNING. The tricky part is when an UPDATE with a search condition reads, modifies, and writes a value that is used in a search condition for another UPDATE. Every DBMS will block waiting for the first UPDATE to finish. Then what? Do you re-run the query to find new tuples that might now satisfy the search condition that didn't before? Do you update the new value in all the tuples you originally found, regardless of whether they still match the search condition? Do you update the new value in all the tuples that you found that still match the search condition? Do you update the old value, perhaps overwriting changes made by the first UPDATE? What does sybase do in that case? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? This is one of the changes for the better from the past. Though I still think a lot more would try it out once it's committed. Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. While personally I want us to switch to a mode where we commit large patches early in the development cycle I don't believe we would refuse to commit it today if it was ready. And I can't imagine two weeks would make the difference either. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Josh Berkus j...@agliodbs.com wrote: All, 1) having the last CF on Nov. 1 was a mistake. That put us square in the path of the US Christian holidays during the critical integration phase .. which means we haven't really had 3 months of integration, we've had *two*. Actually, I'm thinking about this again, and made a mistake about the mistake. The *original plan* was that we were not going to accept any new patches for Nov-CF. Just revised patches from eariler Fests. We didn't stick to that, which is mostly why we are still reviewing now. I don't recall us discussing that, but it sounds like it might help next cycle. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Jeff Davis pg...@j-davis.com wrote: The tricky part is when an UPDATE with a search condition reads, modifies, and writes a value that is used in a search condition for another UPDATE. Every DBMS will block waiting for the first UPDATE to finish. Then what? Either it's totally safe to proceed, or the second UPDATE was rolled back. Do you re-run the query to find new tuples that might now satisfy the search condition that didn't before? There can't be any. Blocks taken during the reading of rows so far have not been released, and would preclude the update from changing results read so far. Do you update the new value in all the tuples you originally found, regardless of whether they still match the search condition? They have to still match, your locks would preclude their modification. Do you update the new value in all the tuples that you found that still match the search condition? They can't have a new value. Locks. Do you update the old value, perhaps overwriting changes made by the first UPDATE? If you haven't gotten to reading rows yet, you're still OK by the time the other transaction's locks are released. Look over those rules and try some thought experiments to convince yourself. It really is safe, if potentially slow. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09, Gregory Stark st...@enterprisedb.com wrote: Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? This is one of the changes for the better from the past. Though I still think a lot more would try it out once it's committed. Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. I think the best person to answer that is Simon. He got a small reprieve the last time this discussion came up, but it's 'put up or shut up time'...time's up. Is the thing ready to go? While i'm 'pro HS', I'm also very much 'pro get 8.4 out asap'. I think both _may_ be possible. This is probably horribly optimistic. What I can do however is offer to summarize my own experiences testing and invite others to do the same. I'd also like to see Simon and or Heikki make a strong statement on where things stand _right now_ (not in two weeks) :-) merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark st...@enterprisedb.com writes: Here's a thought experiment. If it was committable *today* would we be willing to go with it and plan to release with it? Assume that it would *still* mean a longer beta process, so it would still mean releasing in, say April instead of February or March. April instead of February or March? If we cut off *all* further patch acceptance *today*, we might conceivably be ready to release May 1. It is not physically possible to get to beta from where we are before mid-February, given the open issues, lack of release notes, and already-scheduled back-branch updates that will be consuming some of core's time this week. And I doubt anyone thinks we need less than 2 months in beta. (Or did you mean April of 2010?) While personally I want us to switch to a mode where we commit large patches early in the development cycle I don't believe we would refuse to commit it today if it was ready. And I can't imagine two weeks would make the difference either. Well, if it were ready, we'd not be having this discussion. The problem is that it's not ready and no one is very sure about when it will be. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On Mon, 2009-01-26 at 14:31 -0600, Kevin Grittner wrote: Do you re-run the query to find new tuples that might now satisfy the search condition that didn't before? There can't be any. Blocks taken during the reading of rows so far have not been released, and would preclude the update from changing results read so far. Let's say the sequence is: Data: i j 1 10 2 10 3 20 4 20 Session1: BEGIN; UPDATE a SET j = (j - 10) WHERE i = 2 OR i = 3; Session2: BEGIN; UPDATE a SET j = j + 100 WHERE j = 10; Session1: COMMIT; Session2: COMMIT; In PostgreSQL, the result is: i | j ---+- 4 | 20 2 | 0 3 | 10 1 | 110 (4 rows) Which cannot be obtained by any serial execution. What is the result in Sybase, Oracle, etc.? It seems like it would be a challenge to know that the tuple with i=3 would be updated to a value that matches the search condition j=10. So can you tell me a little more about the mechanism by which Sybase solves this problem? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Please, could I subscribe to this list? Thanks.
bgrosper...@laposte.net Bernard Grosperrin BGSoftFactory Bureau/Domicile 09 53 41 58 18 Mobile 06 75 13 97 17 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 20:12 +, Gregory Stark wrote: Merlin Moncure mmonc...@gmail.com writes: HS is working very well (Simon's ongoing work aside). I am pretty confident based on my personal testing that it would represent the project well if committed today. I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? OK, so maybe its been unclear: Gianni Ciolli has been leading a test team on this, with other people swapping in and out as they go to work on other things. There's been multiple test machines running various kinds of test on different OS. Gianni and I have done many midnight sessions together on this and his help and support has been invaluable. If some bugs have got passed that test screening in recent weeks, I think that's down to the sheer volume of tests and my own descriptions of how things ought to work and exactly where to focus. Gianni has located more issues than Heikki has, just. Hannu provided a lot of early input on the problems of conflict resolution and various alternatives. I'm the main developer, but I don't underrate their contributions. I've had specific bug reports from 6 people and test feedback/questions from about another 4-5 people; Mark isn't credited because I wasn't keeping track of them in earl stages of review. Recent refactoring has been a rich source of regressions and those don't go away overnight, but they don't take months either. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. I've never posted to this list before, but I am an SELinux upstream maintainer. I'd just like to interject here, we (the SELinux community) are very interested in KaiGai's work and have been looking forward to it being upstreamed for quite some time. While we haven't been able to analyze the patches directly to determine whether the security goals are indeed being met we have had much discussion and eventually community agreement on the security model being implemented. This happened years ago and has since been merged into the SELinux reference policy that practically all SELinux users use (distributions start with the reference policy and add rules/modules suitable for them). So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. Joshua Brindle -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Jeff Davis pg...@j-davis.com wrote: On Mon, 2009-01-26 at 14:31 -0600, Kevin Grittner wrote: Do you re-run the query to find new tuples that might now satisfy the search condition that didn't before? There can't be any. Blocks taken during the reading of rows so far have not been released, and would preclude the update from changing results read so far. Let's say the sequence is: Data: i j 1 10 2 10 3 20 4 20 Session1: BEGIN; UPDATE a SET j = (j - 10) WHERE i = 2 OR i = 3; OK, the description would be trivial if we assume page level locks (the default), so I'll assume row level locks. I'll also assume READ COMMITTED. (In SERIALIZABLE, no lock is ever released until COMMIT or ROLLBACK, although the update locks can be downgraded or upgraded.) Simplified a bit, this places an exclusive lock on rows 2 and 3 and then updates these two rows in place (no new tuples are created). Session2: BEGIN; UPDATE a SET j = j + 100 WHERE j = 10; This might update row 1 before blocking on the attempt to read row 2. Let's say, for sake of argument, that it does. So it is holding an exclusive lock on row 1 and attempting to acquire an update lock to read row 2. (It will upgrade this to an exclusive lock if it decides to update it, or downgrade it to shared if it decides not to do so.) Session2 is blocked for now. Session1: COMMIT; After the COMMIT succeeds, the locks from Session1 are released. Session2 acquires its update lock and reads row 2, finds that it doesn't match its update criteria, downgrades the lock to shared, acquires an update lock on row 3, finds that it does match the selection criteria, upgrades the lock to exclusive, updates it, acquires and update lock on row 4 finds that it doesn't match the update criteria, downgrades the lock to shared, hits the end of table, releases the shared locks. Session2: COMMIT; After the COMMIT succeeds, the locks from Session2 are released. In PostgreSQL, the result is: i | j ---+- 4 | 20 2 | 0 3 | 10 1 | 110 (4 rows) Which cannot be obtained by any serial execution. What is the result in Sybase, Oracle, etc.? I can't be sure about Oracle, but I think its results would match PostgreSQL. In Sybase, with either READ COMMITTED or SERIALIZABLE, the result would be: i | j ---+- 1 | 110 2 | 0 3 | 110 4 | 20 (4 rows) If that explanation wasn't clear, let me know. Let me restate -- I don't propose that PostgreSQL implement this locking scheme. I think it can and should do better in approaching compliance with the standard, and with ACID properties, without compromising concurrency and performance to the degree required by this sort of locking and blocking. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Jeff Davis pg...@j-davis.com writes: It seems like it would be a challenge to know that the tuple with i=3 would be updated to a value that matches the search condition j=10. So can you tell me a little more about the mechanism by which Sybase solves this problem? This example is a case of the same issue we were discussing earlier involving predicate locking. To solve it you need a way to lock records that your query *isn't* accessing and may not even exist yet to prevent them from being turned into (or inserted as) records your query should be accessing. As Kevin described it earlier Sybase locks the index pages containing the key range you're accessing preventing anyone from inserting new index pointers in that range. If there's no index it locks the entire table on every select to prevent any updates or inserts in the table until your transaction finishes. In any case note that your example is not *serializable*. (Though in Postgres it can happen even in serializable mode, so that's not much of a defence.) I'm unclear what whether it manifests any of the phenomenon which are prohibited for READ COMMITTED. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 15:47 -0500, Merlin Moncure wrote: I'd also like to see Simon and or Heikki make a strong statement on where things stand _right now_ (not in two weeks) :-) Well, we just found 2 bugs over the weekend, one of which is a regression from refactoring. The second bug is being fixed as part of requested refactoring from Heikki, discussed today. I've insisted on full visibility, so all known bugs have been recorded on Wiki as soon as they are reported, even internally located ones. Most have been fixed on same day found. Check out the Wiki, any day. I still have not completed Prepared Transaction support, but if I had then it just would have slowed down the earlier rework. This is probably the only hanging offence. Please remember that Heikki's rework proposals were still under discussion only 2 weeks ago. Personally, I've very happy with the code as stands, generally, but Heikki may wish to move a few things around yet. The rework has meant I've reread most of my own code, so I have found and fixed many of my own bugs and rewritten comments to better explain things. I found, reported and fixed the issue of drop tablespace for example. If cynical, I could have hidden that and waited a month for people to find it. Going Beta will reveal further feedback on usability, so we might expect a few tweaks there as people moan. We'll see. I won't put a time limit on this since people will just add their own fudge factors to that and we'll be no nearer to a true figure. Without everybody breathing down my neck it might be 2-3 days, but I'm spending time reading email and trying to keep my patches alive. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 15:49 -0500, Tom Lane wrote: The problem is that it's not ready and no one is very sure about when it will be. With respect, I've done more than any other developer has done to give you and the community full information on the patch as it develops. I'm sorry you don't believe what I tell you, but if you don't you should see the patch for yourself or ask the person reviewing it. The Wiki has been there for 4 months now, updated daily for last 8 weeks apart from illness. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus wrote: All, So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I'm convinced that if we took a staw poll, 80% of our users would be in favor of waiting for HS. This one feature will make more of a difference in the number of PG users than any feature since the Windows port. Maybe more. on the other hand: We held back version 4 months 7.4 for Windows, before it became apparent that there was at least a year more work to do. That was a mistake, and in many ways HS seems like a similar case. I can only confirm what Josh is saying here. We would also assume that 80% have been waiting for Simon's work for years. In fact, I have been dealing fulltime with PostgreSQL since 1999 and it has been a missing issue since than. Now that we are so close to fixing this issue for so many people out there, we should give it all the attention we have and support Simon + team wherever we can. I think Simon has responded to all question is almost realtime. We should take that into consideration. Also, Simon is focuing on a very open development model - this naturally means a lot of mailing list traffic. Isn't this what this project is all about? I am in favor of giving this patch a useful timeframe for completion. If people decide to give this patch a chance, we will definitely agree on putting some significant manpower in here as well. We are not the only ones who want to see that in. We already see people saying that they delay migrations because they are hoping for readable slaves to go in. Also, in the past 10 years I have been tortured with when can we have replication each and every day ... I am fed up :). i cannot hear it anymore (... but MySQL has replication *aargh*). best regards, hans -- -- Cybertec Schönig Schönig GmbH Professional PostgreSQL Consulting, Support, Training Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8SEPostgres WAS: .4 release planning
Joshua, So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. Can you please take a look at it ASAP, then? In the next week, we will probably decide on whether or not to defer SEPostgres until 8.5. The fact that we haven't gotten a sign-off from any security expert anywhere is leaning the whole community towards defer. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Monday 26 January 2009 2:12:02 pm Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. Precisely. Thank you. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I am just one person, representing one company...but I do agree, fwiw. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
Gregory Stark st...@enterprisedb.com wrote: This example is a case of the same issue we were discussing earlier involving predicate locking. To solve it you need a way to lock records that your query *isn't* accessing and may not even exist yet to prevent them from being turned into (or inserted as) records your query should be accessing. As Kevin described it earlier Sybase locks the index pages containing the key range you're accessing preventing anyone from inserting new index pointers in that range. If there's no index it locks the entire table on every select to prevent any updates or inserts in the table until your transaction finishes. Well, for READ COMMITTED in Sybase it's only until the end of the statement. Hmmm I'm clearly getting a bit rusty on my Sybase row level locking rules. I got some details wrong in my example, but the outcome would be the same. Broader locks though, leading to more potential blocking. I'm unclear what whether it manifests any of the phenomenon which are prohibited for READ COMMITTED. Interesting question. It's behavior not possible in 2 phase locking, but not explicitly prohibited by the standard. Better watch that kind of talk, though, or they may go and change the standard again. ;-) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/26/09 4:28 PM, Joshua Brindle met...@manicmethod.com wrote: Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: snip SE-Linux: this patch has effectively been in development for 2 years ourside the core process before putting it in; the forked SEPostgres is in use in production. KaiGai has been available for 20 hours a week (or more) to troubleshoot issues and change APIs. I really don't see what the problem is with committing it. The problem, in words of one syllable, is that we are not sure we want it. Do you see a user community clamoring for SEPostgres, or a hacker community that is willing or able to maintain it? If KaiGai-san got run over by a bus tomorrow, this patch would be a dead letter, because there just isn't anyone else who is taking sufficient (any?) interest in it. That doesn't bode well for its future viability. Compare the likely audience for it to previous patches of roughly similar complexity, such as integrated text search or the Windows port, and it's just not in the ballpark. The second problem is that we're not sure it's really the right thing, because we have no one who is competent to review the design from a security standpoint. But unless we get past the first problem the second one is moot. I've never posted to this list before, but I am an SELinux upstream maintainer. I'd just like to interject here, we (the SELinux community) are very interested in KaiGai's work and have been looking forward to it being upstreamed for quite some time. While we haven't been able to analyze the patches directly to determine whether the security goals are indeed being met we have had much discussion and eventually community agreement on the security model being implemented. This happened years ago and has since been merged into the SELinux reference policy that practically all SELinux users use (distributions start with the reference policy and add rules/modules suitable for them). So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. I'd just like to echo Josh's sentiments. I'm also active in the SELinux community, and have been involved in several developments that really needed a database with mandatory access control mechanisms. Unfortunately, these developments have all had maintenance requirements that precluded using KaiGai's code as it was outside not in a commercial distribution. We've been waiting anxiously for it to be merged upstream. Additionally, I've talked to many other end users that really want to deploy LAPP stacks with these security features. They often came to us looking for us to help them build such systems, but we've had to turn them away as there was no supported way to build it. Thanks, Chad Sellers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More FOR UPDATE/FOR SHARE problems
On Mon, 2009-01-26 at 09:26 -0800, Jeff Davis wrote: On Mon, 2009-01-26 at 10:48 -0600, Kevin Grittner wrote: I guess the issue of whether this violation of ACID properties should be considered a bug or a feature is a separate discussion, but calling it a feature seems like a hard sell to me. I think I understand the other perspective on this now: SELECT FOR UPDATE/SHARE is an entirely separate command that is more similar (in transactional semantics) to UPDATE than to SELECT. You can think of SELECT FOR UPDATE as first half of UPDATE command UPDATE is in this case split in two SELECT FOR UPDATE UPDATE WHERE CURRENT which means that yes, it has to follow UPDATE semantics to be of any use in FOR UPDATE case. In fact, it's probably most similar to UPDATE ... RETURNING, which will give the same result (that breaks atomicity or isolation, depending on your point of view), which is correct for READ COMMITTED isolation level. Because the command begins with SELECT, I would expect it to follow the rules of SELECT with the side effect of locking. I would think that the standard would have something to say about this*. I certainly don't think it's intuitive behavior. Regards, Jeff Davis. *: It appears that SELECT ... FOR UPDATE is not in the standard, which would indicate to me that the SELECT statement should still behave according to SELECT isolation/snapshot rules. But when I guess about the standard, I'm usually wrong. -- -- Hannu Krosing http://www.2ndQuadrant.com PostgreSQL Scalability and Availability Services, Consulting and Training -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark wrote: I think a lot of people weren't aware there was anybody testing this patch other than Simon and Heikki -- I wasn't until just today. I wonder how many more people are trying it out? I've been running the patch (I think since Jan 5) on a couple dev instances that were used primarily to test if our software worked with what we expected to turn into 8.4. I can't say I've really been testing the patch specifically, but rather testing my workplace's software against a version with this patch, and hadn't noticed any problems. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
Tom Lane wrote: Bernd Helmle maili...@oopsware.de writes: Or what about CREATE [OR REPLACE] [UPDATABLE] VIEW ... ? This looks closer to TEMP|TEMPORARY VIEW, which we already have. But per spec, UPDATABLE should be the default (if not now, then eventually). Are you proposing CREATE [OR REPLACE] [[NOT] UPDATABLE] VIEW ... ? Seems confusing. UNUPDATABLE? :-) BTW, how do we handle cases where the query cannot be updatable, e.g. aggregates? Do we throw a warning? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
On Mon, Jan 26, 2009 at 5:18 PM, Bruce Momjian br...@momjian.us wrote: Tom Lane wrote: Bernd Helmle maili...@oopsware.de writes: Or what about CREATE [OR REPLACE] [UPDATABLE] VIEW ... ? This looks closer to TEMP|TEMPORARY VIEW, which we already have. But per spec, UPDATABLE should be the default (if not now, then eventually). Are you proposing CREATE [OR REPLACE] [[NOT] UPDATABLE] VIEW ... ? Seems confusing. UNUPDATABLE? :-) BTW, how do we handle cases where the query cannot be updatable, e.g. aggregates? Do we throw a warning? yes. we detect that and send a warning saying that there not be any rules -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
Bruce Momjian wrote: Tom Lane wrote: Bernd Helmle maili...@oopsware.de writes: Or what about CREATE [OR REPLACE] [UPDATABLE] VIEW ... ? This looks closer to TEMP|TEMPORARY VIEW, which we already have. But per spec, UPDATABLE should be the default (if not now, then eventually). Are you proposing CREATE [OR REPLACE] [[NOT] UPDATABLE] VIEW ... ? Seems confusing. I'd frankly look at WITH, which is where we've historically stuck non-SQL extensions. BTW, how do we handle cases where the query cannot be updatable, e.g. aggregates? Do we throw a warning? Error if updatable is specified, warning if not. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8SEPostgres WAS: .4 release planning
Josh Berkus wrote: Joshua, So the security model has been looked at, though not the implementation and we do have a community of developers, users and customers interested in this work. Can you please take a look at it ASAP, then? In the next week, we will probably decide on whether or not to defer SEPostgres until 8.5. The fact that we haven't gotten a sign-off from any security expert anywhere is leaning the whole community towards defer. Yes, I will look at them to the extent I am able. As I am not familiar with the postgresql codebase I won't be able to assert the correctness of the hook placement (that is, where the security functions are called with respect to the data they are protecting being accessed). The postgresql community should be more familiar with the hook call sites and hopefully can assist there. I should be able to handle the security backend and determining whether it matches the security model we agreed on, but the hook placement is just as important since a misplaced or missing hook will allow access that should not be granted. Joshua Brindle -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] FK column doesn't exist error message could use more detail
create table a(a_id serial primary key, a int); create table b(b_id serial primary key, a_id int not null references a (id), b int, c_id int not null references c(id)); NOTICE: CREATE TABLE will create implicit sequence b_id_seq for serial column b.b_id NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index b_pkey for table b ERROR: column id referenced in foreign key constraint does not exist How can I tell which FK constraint that's for? Could we have backend/ tablecmds.c:transformColumnNameList() report the constraint name? Though, that wouldn't be quite enough if you did: CREATE TABLE a(a_id ...) CREATE TABLE b(.., a_id int not null, foreign key(id) references a(id)) Handling that would require passing something into transformColumnNameList() to tell it if it was checking fk_attrs vs pk_attrs. Perhaps that's overkill... Thoughts? -- Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark wrote: I think a lot of people weren't aware there was anybody testing this patch ...I wonder how many more people are trying it out? I think I have an idea to improve this aspect for future commit fests. For a long time at each of my workplaces I've been running a development instance against CVS-HEAD just to make sure our software is more future-proof against up-and-coming releases. We run this system with -enable-debug, asserts, etc, and accept that it's just a development system not expected to be totally stable. If it were just as easy for us to pull from a all 'pending-patches' for-commit-fest-nov that pass regression tests branch, I'd happily pull from that instead. I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). And I hate bringing up the version control flame war again; but git really would make this easier. If all patches were on their own branches; the painful merges into this shared branch would be rare, as the source control system would remember the painful parts of the merges. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] On SCM
On Mon, Jan 26, 2009 at 5:42 PM, Ron Mayer rm...@cheapcomplexdevices.com wrote: I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). And I hate bringing up the version control flame war again; but git really would make this easier. If all patches were on their own branches; the painful merges into this shared branch would be rare, as the source control system would remember the painful parts of the merges. There has been enough experimentation with git usage during the 8.4 process that I expect that enough wisdom is getting collected to let people start taking further steps on this. I would think it a giant disaster to try to change anything at this point, when we're amidst simply assessing whether or not we're trending towards having an 8.4 yet, but if people can report concrete benefits from git usage, that will make it *way* easier to argue in favour of change. -- http://linuxfinances.info/info/linuxdistributions.html George Burns - You can't help getting older, but you don't have to get old. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer rm...@cheapcomplexdevices.com writes: I realize in the current system (emailed patches), this would be a horrible pain to maintain such a branch; but perhaps some of the burden could be pushed down to the patch submitters (asking them to merge their own changes into this merged branch). I've considered maintaining such a repository a few times and dismissed it when I realized how much work it would be to maintain. And I hate bringing up the version control flame war again; but git really would make this easier. If all patches were on their own branches; the painful merges into this shared branch would be rare, as the source control system would remember the painful parts of the merges. We have git repositories, I still think maintaining a merged tree with dozens of patches would be a lot of work. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] On SCM
Christopher Browne wrote: On Mon, Jan 26, 2009 at 5:42 PM, Ron Mayer rm...@cheapcomplexdevices.com wrote: There has been enough experimentation with git usage during the 8.4 ... I certainly didn't mean for the idea to be advocating git nor any changes in 8.4. I was hoping the main idea was that the part you didn't quote: If it were just as easy for us to pull from a all 'pending-patches' for-commit-fest-nov that pass regression tests branch, I'd happily pull from that instead. would be very useful regardless of the source control system it's based on. And if enough people found such a 'staging branch' useful, it'd be worth maintaining even if I had to do it with 'patch' and no SCM tools whatsoever -- simply so not everyone has to merge it. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8SEPostgres WAS: .4 release planning
Joshua Brindle met...@manicmethod.com writes: Yes, I will look at them to the extent I am able. As I am not familiar with the postgresql codebase I won't be able to assert the correctness of the hook placement (that is, where the security functions are called with respect to the data they are protecting being accessed). The postgresql community should be more familiar with the hook call sites and hopefully can assist there. I would suggest looking at the documentation (which I assume the patch does update). If it's not clear that things are in order from the documentation then either it needs better documentation or something's wrong... -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
