Re: [HACKERS] Synch Rep v5
Hi, On Mon, Jan 12, 2009 at 1:10 AM, Simon Riggs wrote: >> > Multiple standby is still possible, but just using old file based >> > mechanisms. We would need to be careful about use of %R in that case. >> >> Yes. Synch Rep can work fine with existing warm-standby mechanism. > > If we want multiple standby servers they wouldn't both be able to trim > files from the archive. So we would need to change pg_standby so it > records the %R from multiple servers on the archive and only trimmed the > max of those %R values. s/max/min ? >> > I believe the max delay is 2* wal_sender_delay. >> >> In async replication case, walsender tries to send the xlogs once per >> wal_sender_delay, and receives the response from the standby on >> demand. So, I think that max delay is wal_sender_delay. Am I missing >> something? > > Sending takes time as well, so it is send_time + delay at least. Right. I'll change the doc into an exact description. Similarly, though the document of "synchronous_commit" says --- When off, there can be a delay between when success is reported to the client and when the transaction is really guaranteed to be safe against a server crash. (The maximum delay is three times wal_writer_delay.) --- write_time also should be added up to the maximum delay. >> > I like the way recovery_trigger_file avoids changing pg_standby, but I >> > guess we still need to plug that gap also one day. But does patch 10 >> > also have the other mechanism? >> >> As you imply, current synch-rep has already not needed the change >> of pg_standby, so I'll get rid of the patch from synch-rep patchset. >> Of course, this patch is still useful for existing warm-standby. I should >> add this patch to commitfest for 8.5? > > May as well leave it in, so people can use it with 8.3. I'd like this patch to be reviewed and committed for 8.4 even if synch-rep is postponed to 8.5. Because this is very useful for existing warm-standby mechanism, and only warm-standby would be still a built-in replication solution in 8.4. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Hi, On Tue, Jan 13, 2009 at 10:52 AM, Robert Haas wrote: >> IMHO, the synchronous replication isn't in such good shape, I'm afraid. I've >> said this before, but I'm not happy with the "built from spare parts" nature >> of it. You shouldn't have to configure an archive, file-based log shipping >> using rsync or whatever, and pg_standby. All that is in addition to the >> direct connection between master and slave. The slave really should be able >> to just connect to the master, and download all the WAL it needs directly. >> That's a huge usability issue if left as is, but requires very large >> architectural changes to fix. > > Yeah, I wasn't thinking about this, but you had mentioned it before, > and I thought (and think) it's a pretty fair criticism. I think the > base backup should be integrated into the mechanism as well. I want > to just be able to configure the master and slave for replication, > fire up the slave, and walk away. Without that, I agree that it's > likely to be too cumbersome for any actual use. I don't think this is essential for replication. It's an optimization, and synch-rep still works fine without it. And, I'm not sure that this should be the job of postgres, from the beginning. Do you think we should develop "rsync" again for postgres? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Hi, On Tue, Jan 13, 2009 at 3:32 AM, Robert Haas wrote: > One thing I find interesting is that the "Infrastructure Changes for > Recovery" patch became the foundation for both "Hot Standby" and > "Synchronous Replication". That implies that those changes might be > of somewhat more general interest, at least as the foundation for > further work. If we HS and/or SR are out of reach, it might be worth > at least looking to see if any of that infrastructure work can be > reasonably be committed for 8.4. +1 If the community determined to postpone replication to 8.5, I'll give up working on it (though I'll work it up to good place to leave off, of course) and focus on reviewing infra-patch instead. I don't think that *all* recovery-related patches are too late for 8.4. We should make a choice for the patches and focus on the high-priority one. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1403)
I updated patch set of SE-PostgreSQL and related stuff (r1403). [1/5] http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1403.patch [2/5] http://sepgsql.googlecode.com/files/sepostgresql-utils-8.4devel-3-r1403.patch [3/5] http://sepgsql.googlecode.com/files/sepostgresql-policy-8.4devel-3-r1403.patch [4/5] http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1403.patch [5/5] http://sepgsql.googlecode.com/files/sepostgresql-tests-8.4devel-3-r1403.patch List of updates: - These are rebased to the latest CVS HEAD. - I removed the argument of "bool validate" from string reloption callbacks due to its improvement on the latest CVS HEAD. There are no changes in rest of the parts. If you saw the previous version partway, here is no matter to switch them. The column-level privileges feature is coming to get merged increasingly. It is not too early to *begin* reviewing the SE-PostgreSQL features now. Please make it progress! Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New patch for Column-level privileges
Tom Lane wrote: > I'm thinking make_var is not the place to do this. The places that are > supposed to be taking care of permissions are the ones that do this: > > /* Require read access --- see comments in setTargetTable() */ > rte->requiredPerms |= ACL_SELECT; Indeed, it seems to me appropriate policy, because CLP feature is a part of SQL standard access control model. The rte->requiredPerms is set on the following places: (1) transformLockingClause() and markQueryForLocking() It adds ACL_SELECT_FOR_UPDATE for listed relations. In this case, column-level priv feature checks ACL_UPDATE for each columns on rte->cols_sel, doesn't it? So, I think it is unnecessary to care about anything here. (2) setTargetTable() It is invoked from transformInsertStmt(), transformUpdateStmt() and transformDeleteStmt(). I thnk it is a proper place to set rte->cols_mod, but the caller does not initialize Query->targetList yet, when it is invoked. So, I think we need put a function call to set cols_mod on caller side based on Query->resultRelation and Query->targetList. (3) scanRTEForColumn() Stephen's original one patched here, but it does not handle RTE_JOIN well, so it made a matter on table-joins. Please revert a code to mark rte->cols_sel, with proper handling for RTE_JOIN cases. (4) addRangeTableEntry() Purpose of the function is to translate RangeVar node to RangeTblEntry node listed on FROM clause and so on. I think we don't need to add any column references here. When the translated relation used for column-references, it is a case that rte->cols_sel is empty. (5) addRangeTableEntryForRelation() It is similar to addRangeTableEntry(). I think we don't need to add something here. (6) ExpandColumnRefStar() and ExpandAllTables() They invoke expandRelAttrs() to handle "SELECT * FROM t;" case. I think here is a proper point to mark refered columns. Please note that here is no guarantee that given RangeTblEntry has RTE_RELATION. Thus, we need the following reworking in my opinion. (a) Implement a function to set rte->cols_sel and rte->cols_mod correctly, even if the given rte has RTE_JOIN. (b) Put a function to set rte->cols_mod on the caller of setTargetTable(). (c) Put a function to set rte->cols_sel on scanRTEForColumn(), expandRelAttrs() and transformWholeRowRef(). > It's possible that we've missed some --- in particular, right at the > moment I am not sure that whole-row Vars are handled properly. Is transformWholeRowRef() a proper place to handle it? If given RangeTblEntry has RTE_JOIN, it has to resolve it and set its source's cols_sel. > And maybe we could refactor a little bit to save some code. > But those are basically the same places that ought to be adding > bits to the column bitmaps. -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documenting pglesslog
Hi, I have no intention to make pglesslog to conflict to PostgreSQL license. Any advice is welcome to make pglesslog available without any license concern. I've a question and ideas. Bruce's modification directly points to my pgfoundry page. I'm not sure what it means. Does it mean that I have to maintain the page for a while? If pglesslog helps for future releases, can it be a part of PostgreSQL release, as contrib module so that all the documentation in pgfoundry (although very simple) is included in the release material? As many hackers know, I've posted another code to speedup PITR after slipping FPW, which does work with 8.3 as external module (pg_readahead). I'm now working to work this with synchronous replication. Maybe it's a good idea to use pglesslog with pg_readahead.Although I'm not sure if pg_readahead integration with synchronous replication will be done within 8.4 development period, I'm quite ready to post pg_readahead for 8.4 sililar to that for 8.3, which also could be in contrib module. Looking forward to inputs. 2009/1/12 Simon Riggs : > > On Sun, 2009-01-11 at 09:47 -0500, Bruce Momjian wrote: >> Simon Riggs wrote: >> > >> > On Sat, 2009-01-10 at 23:38 -0500, Bruce Momjian wrote: >> > >> > > It is BSD licensed. I don't see any copyright issues: >> > > >> > A licence and copyright are different things. Why do we insist on >> > changing copyright on our code if it is unimportant? >> >> Because the BSD copyright has no enforcement > > AFAIK there is no such thing as a BSD copyright. There is Copyright and > there is a BSD licence, issued by the copyright holder. > > In general, IMHO, I don't think it's a good direction to go in to > include links to works of other copyright holders. > > Specifically, I have nothing but good to say about pglesslog and its > authors. > > -- > Simon Riggs www.2ndQuadrant.com > PostgreSQL Training, Services and Support > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- -- Koichi Suzuki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New patch for Column-level privileges
KaiGai Kohei writes: > I reconsidered the previous walker implementation independent > from other parser codes is more simple and better. And slower, and equally subject to this bug, so I'm not convinced. > Stephen, Tom, what is your opinion? I'm thinking make_var is not the place to do this. The places that are supposed to be taking care of permissions are the ones that do this: /* Require read access --- see comments in setTargetTable() */ rte->requiredPerms |= ACL_SELECT; It's possible that we've missed some --- in particular, right at the moment I am not sure that whole-row Vars are handled properly. And maybe we could refactor a little bit to save some code. But those are basically the same places that ought to be adding bits to the column bitmaps. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
> I am just explaining how it works in practice. If the patch is still > being improved, the feeling is that the author wants more time to adjust > things, and with other things on our plate, we are glad to leave their > patch until last. Well, it's good that you have an explanation, but I'm not sure it helps much. :-) Surely the patches that are most likely to change substantially are the big ones, and leaving those until last results in them not making the time-based cutoff. Someone who submitted a 20-line patch isn't likely to revise it substantially; someone who is being paid $20k to write a patch is likely to spend a lot of time working on it. I think the fundamental problem here is the number and bandwidth of the committers, which seems to be pretty limited. Most of the committers are either inactive, or essentially maintainers for a particular subsystem. With the exception of patches authored by the committers themselves, I think the vast majority of patches for this 'fest were committed by Tom and Peter - and I think really mostly Tom. And many of those were significantly modified in the process of being committed, which suggests that efforts to take the load of committers by having non-committers do reviews has not been entirely successful. (It would be interesting to here how much value people think it has added, and get suggestions on how to do things better next time.) I'm not sure what to do about it, though. More committers could be added, but I presume if there were obvious candidates it would have been done already. It's complicated by the fact that you need people who both (1) know what they're doing and (2) have time to review and commit *other people's patches*. In reality, a pretty significant fraction of the current committers are either mostly inactive, or essentially maintainers for one small area of the code. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New patch for Column-level privileges
I can find a matter on JOIN. Please make sure the following function invocation chain: transformSelectStmt() -> transformFromClause() -> transformFromClauseItem() -> expandRTE(), if JoinExpr -> expandRelation(), if rte->rtekind == RTE_RELATION -> expandTupleDesc() expandTupleDesc() set rte->cols_sel for all the user columns. It seems to me unexpected behavior. Maybe, the point to mark rte->cols_sel has to be moved. However, I wonder whether it is actually proper manner to put a code depending its invocation context on existing parser. I can understand Tom's concern, but does it make maintenance difficulty in the future version? For example, when a new future invokes expandRTE(), it also polutes rte->cols_sel implicitly. I reconsidered the previous walker implementation independent from other parser codes is more simple and better. Stephen, Tom, what is your opinion? Thanks, KaiGai Kohei wrote: > Stephen, > > The revised patch can reproduce the original matter, as follows: > > postgres=# CREATE TABLE t1 (a int, b text); > CREATE TABLE > postgres=# CREATE TABLE t2 (x int, y text); > CREATE TABLE > postgres=# GRANT select(a) on t1 to ymj; > GRANT > postgres=# GRANT select(x,y) on t2 TO ymj; > GRANT > postgres=# \c - ymj > psql (8.4devel) > You are now connected to database "postgres" as user "ymj". > postgres=> SELECT a, y FROM t1 JOIN t2 ON a = x; > NOTICE: make_var: attribute 1 added on rte(relid=16398, rtekind=0) > NOTICE: make_var: attribute 1 added on rte(relid=16404, rtekind=0) > NOTICE: make_var: attribute 1 added on rte(relid=0, rtekind=2) > NOTICE: make_var: attribute 4 added on rte(relid=0, rtekind=2) > NOTICE: pg_attribute_aclmask: t1.a required: 0002 allowed: 0002 > NOTICE: pg_attribute_aclmask: t1.b required: 0002 allowed: > ERROR: permission denied for relation t1 > > > I think it is not an essentiality whether it walks on query tree, or not. > So, I also suppose putting this code on make_var(). > However, it does not care the case when rte->relkind == RTE_JOIN, so > it requires column-level privileges on whole of columns including > unrefered ones. > > My suggestiong is case separations. > If rte->relkind == RTE_RELATION, it can keep current behavior, as is. > If rte->relkind == RTE_JOIN, it need to find the source relation > recursively and marks required column. Please note that the source > relation can be RTE_JOIN or others. > Elsewhere, we don't need to care anymore. > > Thanks, > > Stephen Frost wrote: >> Tom, et al, >> >> * Stephen Frost (sfr...@snowman.net) wrote: applyColumnPrivs is misnamed as it's not "applying" any privileges nor indeed doing much of anything directly to do with privileges. It should probably be named something more like findReferencedColumns. And what's with the special exception for SortGroupClause!? >>> I'm not sure what the story with SortGroupClause is.. Might just have >>> been a bit more difficult to do. KaiGai? >> This should be resolved now, since.. >> Actually, though, you probably shouldn't have applyColumnPrivsWalker at all. It requires an additional traversal of the parse tree, and an additional RTE search for each var, for no good reason. Can't we just mark the column as referenced in make_var() and maybe a couple other places that already have their hands on the RTE? >>> I certainly like this idea and I'll look into it, but it might take me a >>> bit longer to find the appropriate places beyond make_var(). >> I've implemented and tested these suggested changes, and have removed >> applyColumnPrivs, etc. It passes all the regression tests, which had a >> variety of tests, so I'm reasonably happy with this. >> pg_attribute_aclmask seems to need a rethink. I don't like the amount of policy copied-and-pasted from pg_class_aclmask, nor the fact that it will fail for system attributes (attnum < 0), nor the fact that it insists on looping over the attributes even if the relation privileges would provide what's needed. (Indeed, you shouldn't need that "merge ACLs" operation at all -- you could be ORing a couple of bitmasks instead, no?) >>> I'll have to think of the entry points for pg_attribute_aclmask. In >>> general, we shouldn't ever get to it if the relation privileges are >>> sufficient but I think it's exposed to the user at some level, where >>> we would be wrong to say a user doesn't have rights on the column >>> when they do have the appropriate table-level rights. Unfortunately, >>> aclmask() uses information beyond the bitmasks (who granted them, >>> specifically) so I don't think we can just OR the bitmasks. >>> >>> With regard to looping through the attributes, I could split it up into >>> two functions, would that be better? A function that handles >>> all-attribute cases (either 'AND'd or 'OR'd together depending on what >>> the caller wants) could be added prett
Re: [HACKERS] New patch for Column-level privileges
Stephen, The revised patch can reproduce the original matter, as follows: postgres=# CREATE TABLE t1 (a int, b text); CREATE TABLE postgres=# CREATE TABLE t2 (x int, y text); CREATE TABLE postgres=# GRANT select(a) on t1 to ymj; GRANT postgres=# GRANT select(x,y) on t2 TO ymj; GRANT postgres=# \c - ymj psql (8.4devel) You are now connected to database "postgres" as user "ymj". postgres=> SELECT a, y FROM t1 JOIN t2 ON a = x; NOTICE: make_var: attribute 1 added on rte(relid=16398, rtekind=0) NOTICE: make_var: attribute 1 added on rte(relid=16404, rtekind=0) NOTICE: make_var: attribute 1 added on rte(relid=0, rtekind=2) NOTICE: make_var: attribute 4 added on rte(relid=0, rtekind=2) NOTICE: pg_attribute_aclmask: t1.a required: 0002 allowed: 0002 NOTICE: pg_attribute_aclmask: t1.b required: 0002 allowed: ERROR: permission denied for relation t1 I think it is not an essentiality whether it walks on query tree, or not. So, I also suppose putting this code on make_var(). However, it does not care the case when rte->relkind == RTE_JOIN, so it requires column-level privileges on whole of columns including unrefered ones. My suggestiong is case separations. If rte->relkind == RTE_RELATION, it can keep current behavior, as is. If rte->relkind == RTE_JOIN, it need to find the source relation recursively and marks required column. Please note that the source relation can be RTE_JOIN or others. Elsewhere, we don't need to care anymore. Thanks, Stephen Frost wrote: > Tom, et al, > > * Stephen Frost (sfr...@snowman.net) wrote: >>> applyColumnPrivs is misnamed as it's not "applying" any privileges nor >>> indeed doing much of anything directly to do with privileges. It should >>> probably be named something more like findReferencedColumns. And what's >>> with the special exception for SortGroupClause!? >> I'm not sure what the story with SortGroupClause is.. Might just have >> been a bit more difficult to do. KaiGai? > > This should be resolved now, since.. > >>> Actually, though, you probably shouldn't have applyColumnPrivsWalker at all. >>> It requires an additional traversal of the parse tree, and an additional RTE >>> search for each var, for no good reason. Can't we just mark the column >>> as referenced in make_var() and maybe a couple other places that already >>> have >>> their hands on the RTE? >> I certainly like this idea and I'll look into it, but it might take me a >> bit longer to find the appropriate places beyond make_var(). > > I've implemented and tested these suggested changes, and have removed > applyColumnPrivs, etc. It passes all the regression tests, which had a > variety of tests, so I'm reasonably happy with this. > >>> pg_attribute_aclmask seems to need a rethink. I don't like the amount >>> of policy copied-and-pasted from pg_class_aclmask, nor the fact that >>> it will fail for system attributes (attnum < 0), nor the fact that it >>> insists on looping over the attributes even if the relation privileges >>> would provide what's needed. (Indeed, you shouldn't need that "merge >>> ACLs" operation at all -- you could be ORing a couple of bitmasks >>> instead, no?) >> I'll have to think of the entry points for pg_attribute_aclmask. In >> general, we shouldn't ever get to it if the relation privileges are >> sufficient but I think it's exposed to the user at some level, where >> we would be wrong to say a user doesn't have rights on the column >> when they do have the appropriate table-level rights. Unfortunately, >> aclmask() uses information beyond the bitmasks (who granted them, >> specifically) so I don't think we can just OR the bitmasks. >> >> With regard to looping through the attributes, I could split it up into >> two functions, would that be better? A function that handles >> all-attribute cases (either 'AND'd or 'OR'd together depending on what >> the caller wants) could be added pretty easily and then >> pg_attribute_aclmask could be reverted to just handling a single >> attribute at a time (which would fix it for system attributes as >> well..). > > I've modified the code to handle system attributes but otherwise kept it > pretty much the same (with the loop and the special values to indicate > how to handle it). I looked at creating a seperate function and it > really seemed like it would be alot of code duplication.. It might > still be possible to refactor it if you'd really like. I'm open to > other thoughts or suggestions you have based on my comments above. > > Updated patch attached. > > Thanks! > > Stephen -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 2:05 PM, Heikki Linnakangas wrote: > Well, I've been keeping an eye on both Hot Standby and Synchronous > Replication patches. IMHO the Hot Standby patch is architecturally sound, > and while I suggested some pretty big changes just recently (which Simon > picked up and did already), it's in pretty good shape. No doubt there's > still some issues that haven't been uncovered, comments to be fixed, > documentation to be written, but no showstoppers or anything that requires a > major rewrite. There's one todo item left: prepared transactions, but I > don't think there's anything fundamentally hard about them, just needs to be > fixed. Simon mentioned usability issues related to who/when queries get > cancelled, but I think we've discussed that to death already and the patch > handles it quite nicely. Cool - that's good to hear. > IMHO, the synchronous replication isn't in such good shape, I'm afraid. I've > said this before, but I'm not happy with the "built from spare parts" nature > of it. You shouldn't have to configure an archive, file-based log shipping > using rsync or whatever, and pg_standby. All that is in addition to the > direct connection between master and slave. The slave really should be able > to just connect to the master, and download all the WAL it needs directly. > That's a huge usability issue if left as is, but requires very large > architectural changes to fix. Yeah, I wasn't thinking about this, but you had mentioned it before, and I thought (and think) it's a pretty fair criticism. I think the base backup should be integrated into the mechanism as well. I want to just be able to configure the master and slave for replication, fire up the slave, and walk away. Without that, I agree that it's likely to be too cumbersome for any actual use. >> One thing I find interesting is that the "Infrastructure Changes for >> Recovery" patch became the foundation for both "Hot Standby" and >> "Synchronous Replication". That implies that those changes might be >> of somewhat more general interest, at least as the foundation for >> further work. If we HS and/or SR are out of reach, it might be worth >> at least looking to see if any of that infrastructure work can be >> reasonably be committed for 8.4. > > Yeah, being able to do an online checkpoint after recovery has some value of > its own. Is there anything standing in the way of committing that patch? I don't think I've seen anything mentioned on -hackers. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] solaris libpq threaded build fails
I supposed Solaris 2.5.1 (release 1996) is just too old to add threading, and this code has been unchanged for years. --- Andrew Chernow wrote: > for anyone interested > > Solaris 2.5.1 with --enable-thread-safety > > configure:25848: gcc -o conftest -O2 -Wall -Wmissing-prototypes > -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels > -fno-strict-aliasing -fwrapv -pthreads -pthreads -D_REENTRANT > -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -DIN_CONFIGURE -lgcc_s > -lvsnprintf conftest.c -lz -lposix4 -lsocket -ldl -lm -lpthread > -lpthread>&5 > In file included from conftest.c:129: > ./src/test/thread/thread_test.c: In function `main': > ./src/test/thread/thread_test.c:179: warning: implicit declaration of > function `mkstemp' > ./src/test/thread/thread_test.c:188: warning: implicit declaration of > function `gethostname' > Undefined first referenced > symbol in file > gethostbyname /var/tmp//ccclix3s.o (symbol > belongs to implicit dependency /usr/lib/libnsl.so.1) > ld: fatal: Symbol referencing errors. No output written to conftest > collect2: ld returned 1 exit status > configure:25851: $? = 1 > configure: program exited with status 1 > configure: failed program was: > > so far: hpux 10.20, solaris 2.5.1 and cygwin builds all fail when thread > safe is enable. > > -- > Andrew Chernow > eSilo, LLC > every bit counts > http://www.esilo.com/ > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Gregory Stark writes: > Bruce Momjian writes: >> As for the process used, I think it is useful to understand how >> committers choose what to work on next. ... > It's not just "unfair". It's counter-productive. It means you're ignoring the > very patches whose authors are mostly likely to be responsive to requests to > change them. And who would be most likely to be fertile ground for further > improvements. I don't think you can honestly argue that the replication-related patches are getting ignored. AFAICT there's quite a lot of review effort going on around them. KaiGai-san probably has a legitimate beef about lack of review on his patch, but the replication patches do not. It's true that stuff isn't going to get *committed* until it seems reasonably stable, but I hope you weren't arguing for that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Gregory Stark wrote: > > Now, maybe this is unfair to patches that are frequently updated, but > > this is the typical process we follow, and it explains why the patches > > above have not gotten near commit status yet. > > It's not just "unfair". It's counter-productive. It means you're ignoring the > very patches whose authors are mostly likely to be responsive to requests to > change them. And who would be most likely to be fertile ground for further > improvements. > > Perhaps it would be useful for you to understand how it looks from a > submitter's point of view. As long as the patch sits in limbo only minor > tweaks and refinements are worth bothering with. Any thoughts of continuing on > any subsequent phases of development are all crushed since all that work might > go down the drain when the committer makes changes to the code it's based on. I am just explaining how it works in practice. If the patch is still being improved, the feeling is that the author wants more time to adjust things, and with other things on our plate, we are glad to leave their patch until last. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
> I feel no need to encourage people to send huge patches uncompressed ;-) > gzip normally gets at least 3x or 4x on large diffs. So a limit around > 250K ought to be enough. To paraphrase a leading authority on PostgreSQL development, and with tongue firmly in cheek, there's something to what you say, but consider that we have pretty much unanimous agreement that 100k is too small. I think we should try to fix the problem, not just gradually ratchet up the value until people start complaining in the other direction. :-) http://archives.postgresql.org/pgsql-hackers/2008-12/msg00809.php ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Bruce Momjian writes: > As for the process used, I think it is useful to understand how > committers choose what to work on next. One criteria is that the patch > has stabilized; if a patch is still be modified regularly, the > committer might as well work on another patch that has stabilized. Now, > a committer could ask for the patch to stabilize to work on it, but if > he has other patches that are stable, there is no point in asking for a > stable version; he might as well work on just stable ones until only > unstable ones are left. > > Now, maybe this is unfair to patches that are frequently updated, but > this is the typical process we follow, and it explains why the patches > above have not gotten near commit status yet. It's not just "unfair". It's counter-productive. It means you're ignoring the very patches whose authors are mostly likely to be responsive to requests to change them. And who would be most likely to be fertile ground for further improvements. Perhaps it would be useful for you to understand how it looks from a submitter's point of view. As long as the patch sits in limbo only minor tweaks and refinements are worth bothering with. Any thoughts of continuing on any subsequent phases of development are all crushed since all that work might go down the drain when the committer makes changes to the code it's based on. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch for str_numth() in PG 7.4
Alvaro Herrera writes: > This was fixed on 1.84 of formatting.c for 8.0 (but not backpatched for > no apparent reason), which also changed some other stuff in that file. > The complete patch is attached. I dunno why I didn't back-patch that; feel free to do so. Better do the whole thing, because anything that chokes on str_numth's abuse of sprintf is not going to like those other cases either. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 08:12:46PM -0500, Tom Lane wrote: > 250K ought to be enough. ...for anybody ;) Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Stefan Kaltenbrunner writes: > Tom Lane wrote: >> David Fetter writes: >>> 1. Remove the messages size limits on -hackers. They serve no useful >>> purpose, and they interfere with our development process. >> >> Agreed, or at least boost it up a good bit more. > the question really is how much "a bit more" is - right now the limit is >10 characters which limits us to ~70KB of attachments (around the > size of the Hot-standby patch if bzip2 compressed). > The SE-Postgres patch for example is ~650KB uncompressed - if we want to > cope with uncompressed patches that large we would have to increase > the current limit by a factor of 10 at least. I feel no need to encourage people to send huge patches uncompressed ;-) gzip normally gets at least 3x or 4x on large diffs. So a limit around 250K ought to be enough. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documenting pglesslog
Bruce Momjian wrote: > In thinking about how to communicate to users about reducing continuous > archiving storage requirements, I realized we don't mention pglesslog in > our official documentation. > > The attached patch documents how to use pglesslog and gzip/gunzip to > reduce storage requirements. Comments? > > Also, I assume pg_lesslog removes the padding we use to make all WAL > files 16MB, effectively doing the function of clearxlogtail too, right? Applied. --- > > -- > Bruce Momjian http://momjian.us > EnterpriseDB http://enterprisedb.com > > + If your life is a hard drive, Christ can be your backup. + [ text/x-diff is unsupported, treating like TEXT/PLAIN ] > Index: doc/src/sgml/backup.sgml > === > RCS file: /cvsroot/pgsql/doc/src/sgml/backup.sgml,v > retrieving revision 2.121 > diff -c -c -r2.121 backup.sgml > *** doc/src/sgml/backup.sgml 9 Nov 2008 17:51:15 - 2.121 > --- doc/src/sgml/backup.sgml 11 Jan 2009 01:41:12 - > *** > *** 1337,1342 > --- 1337,1359 > WAL files are part of the same tar file. > Please remember to add error handling to your backup scripts. > > + > + > + If archive storage size is a concern, use > pg_compresslog, > + http://pglesslog.projects.postgresql.org";>, to > + remove unnecessary and trailing > + space from the WAL files. You can then use > + gzip to further compress the output of > + pg_compresslog: > + > + archive_command = 'pg_compresslog %p - | gzip > > /var/lib/pgsql/archive/%f' > + > + You will then need to use gunzip and > + pg_decompresslog during recovery: > + > + restore_command = 'gunzip < /mnt/server/archivedir/%f | pg_decompresslog > - %p' > + > + > > > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch for str_numth() in PG 7.4
Andreas 'ads' Scherbaum wrote:
>
> Hello all,
>
> On Mon, 12 Jan 2009 22:55:32 +0100 Andreas 'ads' Scherbaum wrote:
>
> > I tracked the problem done to the function str_numth() in
> > src/backend/utils/adt/formatting.c. The fix (attached) is easy: i
> > stole the function code from the latest 8.0 version which looks fare
> > more sane in handling the string pointers.
>
> Now with the correct patch attached, thanks to anyone who pointed this
> out ;-)
This was fixed on 1.84 of formatting.c for 8.0 (but not backpatched for
no apparent reason), which also changed some other stuff in that file.
The complete patch is attached.
revision 1.84
date: 2005-01-12 22:40:13 -0300; author: tgl; state: Exp; lines: +9 -7;
branches: 1.84.4;
Remove unportable assumption that it's okay to use the target buffer
of an sprintf() as a source string. Demonstrably does not work with
recent gcc and/or glibc on some platforms.
--
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Index: src/backend/utils/adt/formatting.c
===
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/utils/adt/formatting.c,v
retrieving revision 1.83
retrieving revision 1.84
diff -c -p -r1.83 -r1.84
*** src/backend/utils/adt/formatting.c 1 Jan 2005 05:43:07 - 1.83
--- src/backend/utils/adt/formatting.c 13 Jan 2005 01:40:13 - 1.84
***
*** 1,7
/* ---
* formatting.c
*
! * $PostgreSQL: pgsql/src/backend/utils/adt/formatting.c,v 1.83 2005-01-01
05:43:07 momjian Exp $
*
*
* Portions Copyright (c) 1999-2005, PostgreSQL Global Development Group
--- 1,7
/* ---
* formatting.c
*
! * $PostgreSQL: pgsql/src/backend/utils/adt/formatting.c,v 1.84 2005-01-13
01:40:13 tgl Exp $
*
*
* Portions Copyright (c) 1999-2005, PostgreSQL Global Development Group
*** get_th(char *num, int type)
*** 1462,1468
static char *
str_numth(char *dest, char *num, int type)
{
! sprintf(dest, "%s%s", num, get_th(num, type));
return dest;
}
--- 1462,1470
static char *
str_numth(char *dest, char *num, int type)
{
! if (dest != num)
! strcpy(dest, num);
! strcat(dest, get_th(num, type));
return dest;
}
*** static int
*** 2057,2062
--- 2059,2065
dch_date(int arg, char *inout, int suf, int flag, FormatNode *node, void
*data)
{
charbuff[DCH_CACHE_SIZE],
+ workbuff[32],
*p_inout;
int i,
len;
*** dch_date(int arg, char *inout, int suf,
*** 2117,2123
switch (arg)
{
-
case DCH_A_D:
case DCH_B_C:
if (flag == TO_CHAR)
--- 2120,2125
*** dch_date(int arg, char *inout, int suf,
*** 2179,2186
}
break;
case DCH_MONTH:
! strcpy(inout, months_full[tm->tm_mon - 1]);
! sprintf(inout, "%*s", S_FM(suf) ? 0 : -9,
str_toupper(inout));
if (S_FM(suf))
return strlen(p_inout) - 1;
else
--- 2181,2188
}
break;
case DCH_MONTH:
! strcpy(workbuff, months_full[tm->tm_mon - 1]);
! sprintf(inout, "%*s", S_FM(suf) ? 0 : -9,
str_toupper(workbuff));
if (S_FM(suf))
return strlen(p_inout) - 1;
else
*** dch_date(int arg, char *inout, int suf,
*** 2242,2249
}
break;
case DCH_DAY:
! strcpy(inout, days[tm->tm_wday]);
! sprintf(inout, "%*s", S_FM(suf) ? 0 : -9,
str_toupper(inout));
if (S_FM(suf))
return strlen(p_inout) - 1;
else
--- 2244,2251
}
break;
case DCH_DAY:
! strcpy(workbuff, days[tm->tm_wday]);
! sprintf(inout, "%*s", S_FM(suf) ? 0 : -9,
str_toupper(workbuff));
if (S_FM(suf))
return strlen(p_inout) - 1;
else
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Tom Lane wrote: > "Christopher Browne" writes: > > On Mon, Jan 12, 2009 at 12:27 PM, Dave Page wrote: > >> On Mon, Jan 12, 2009 at 5:19 PM, Tom Lane wrote: > >>> In general, we have always regretted it in the past when we chose to > >>> slip a release waiting for a specific feature... > >> > >> I don't recall such a time - though perhaps the last time it happened > >> was before I was so heavily involved in the release process (ie. 7.x). > >> What were the reasons for regretting it? > > > I seem to recall us deferring 8.1 (was it 8.1?) for a while on the > > basis that we were waiting for [something I don't recall offhand]. > > The feature that we were *hoping* to get wound up dropped on the floor > > because it just wasn't ready, even *with* the extra time. > > That's happened more than once, though my memory of details is fuzzy > and I don't have time to troll the archives for them right now. > Maybe Bruce can remember without a lot of searching. But our current > policy of time-based releases (ie deadlines) is born of hard experience > with the negative consequences of saying "we'll release when feature X > is ready". The real killer disadvantage is that all other development > tends to stop until X is ready, because no one can plan anything. OK, I had to think about this one, and I didn't want to fan the flames in the discussion either. Basically, I have given up trying to track the many patches around recovery, replication, and hot standby, and I have stated that to several people privately. I have kept an archive of the active emails about the topic: http://momjian.us/cgi-bin/pgsql/pitr Looking at the list on the commit fest wiki: http://wiki.postgresql.org/wiki/CommitFestInProgress#Recovery.2C_Replication.2C_Hot_Standby I think we should focus on the two simplest patches first, "Infrastructure changes for recovery", and "rmgr hooks and contrib/rmgr_hook" because those are probably the easiest to get committed. Based on comments from Heikki, I think "Hot Standby - queries during archive recovery" can be committed, and in fact perhaps Heikki can do the commit. As far as "Synchronous log-shipping replication", there was only a hope that would be completed in time for 8.4, and in fact trying to complete it probably made completing the other patches harder. I think it is time to focus on the first three patches I listed and accept that we are not going to be able to complete synchronous log-shipping in time. I think the code is just too much in flux at this point. Even trying to get it into 8.4, given its late start in the development process, just reflects wishful thinking and not the kind of hard discipline we need to keep our release process organized. Optimism is nice and all, but with so many people and companies relying on us, we don't have the luxury of optimism. If people want to be optimimistic going into the development cycle, fine, but at the end we have to be practical, because failure will lead a disappointment with the community which will be palpable. (Think back to the frustration we have felt about delayed releases, and features we thought we were going to get, but didn't.) As for the process used, I think it is useful to understand how committers choose what to work on next. One criteria is that the patch has stabilized; if a patch is still be modified regularly, the committer might as well work on another patch that has stabilized. Now, a committer could ask for the patch to stabilize to work on it, but if he has other patches that are stable, there is no point in asking for a stable version; he might as well work on just stable ones until only unstable ones are left. Now, maybe this is unfair to patches that are frequently updated, but this is the typical process we follow, and it explains why the patches above have not gotten near commit status yet. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>> > No. A developer can delete, move and rebase branches in his own >> > repository as he likes, and all of those operations "modify >> > history". In fact, a developer can completely destroy or take >> > offline his published repository. It's *not* an archive. > > We could do this using git's configuration: > > http://www.kernel.org/pub/software/scm/git/docs/git-config.html > > See receive.denyNonFastForwards, which is built for just this purpose :) Spiffy. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch for str_numth() in PG 7.4
Hello all,
On Mon, 12 Jan 2009 22:55:32 +0100 Andreas 'ads' Scherbaum wrote:
> I tracked the problem done to the function str_numth() in
> src/backend/utils/adt/formatting.c. The fix (attached) is easy: i
> stole the function code from the latest 8.0 version which looks fare
> more sane in handling the string pointers.
Now with the correct patch attached, thanks to anyone who pointed this
out ;-)
Kind regards
--
Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
diff -rc postgresql-7.4.23.orig/src/backend/utils/adt/formatting.c postgresql-7.4.23.patch/src/backend/utils/adt/formatting.c
*** postgresql-7.4.23.orig/src/backend/utils/adt/formatting.c 2007-06-29 03:52:14.0 +0200
--- postgresql-7.4.23.patch/src/backend/utils/adt/formatting.c 2009-01-12 22:39:47.0 +0100
***
*** 1444,1450
static char *
str_numth(char *dest, char *num, int type)
{
! sprintf(dest, "%s%s", num, get_th(num, type));
return dest;
}
--- 1444,1452
static char *
str_numth(char *dest, char *num, int type)
{
! if (dest != num)
! strcpy(dest, num);
! strcat(dest, get_th(num, type));
return dest;
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch for str_numth() in PG 7.4
Andreas 'ads' Scherbaum writes:
> I tracked the problem done to the function str_numth() in
> src/backend/utils/adt/formatting.c. The fix (attached) is easy: i
> stole the function code from the latest 8.0 version which looks fare
> more sane in handling the string pointers.
I think you attached the wrong patch.
> *** src/bin/psql/describe.c 2008-12-21 13:19:40.0 +0100
> --- src/bin/psql/describe.c.orig 2008-12-21 02:16:31.0 +0100
> ***
> *** 464,476
> " d.datctype as
> \"%s\",\n",
> gettext_noop("Collation"),
> gettext_noop("Ctype"));
> ! if (pset.sversion >= 80100)
> ! appendPQExpBuffer(&buf,
> ! "
> pg_catalog.array_to_string(d.datacl, E'\\n') as \"%s\"",
> ! gettext_noop("Access Privileges"));
> ! else
> ! appendPQExpBuffer(&buf,
> ! "
> pg_catalog.array_to_string(d.datacl, '\\n') as \"%s\"",
> gettext_noop("Access Privileges"));
> if (verbose && pset.sversion >= 80200)
> appendPQExpBuffer(&buf,
> --- 464,471
> " d.datctype as
> \"%s\",\n",
> gettext_noop("Collation"),
> gettext_noop("Ctype"));
> ! appendPQExpBuffer(&buf,
> ! " d.datacl as \"%s\"",
> gettext_noop("Access Privileges"));
> if (verbose && pset.sversion >= 80200)
> appendPQExpBuffer(&buf,
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's RemoteDBA services!
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Patch for str_numth() in PG 7.4
Hello,
even if 7.4 is almost at the end of it's lifetime i found a bug/problem
on a fairly new laptop (not surprising: during upgrade tests). All
timestamp regression tests failed, only "th" was emitted.
I tracked the problem done to the function str_numth() in
src/backend/utils/adt/formatting.c. The fix (attached) is easy: i
stole the function code from the latest 8.0 version which looks fare
more sane in handling the string pointers.
Since this patch is easy enough (replacing one line with three
lines of code), does not seem to break existing code and removes at
least one bug i ask to include this patch into the next 7.4 release (i
assume there will be at least one release if 7.4 reaches eol).
Thanks & kind regards
--
Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
*** src/bin/psql/describe.c 2008-12-21 13:19:40.0 +0100
--- src/bin/psql/describe.c.orig 2008-12-21 02:16:31.0 +0100
***
*** 464,476
" d.datctype as \"%s\",\n",
gettext_noop("Collation"),
gettext_noop("Ctype"));
! if (pset.sversion >= 80100)
! appendPQExpBuffer(&buf,
! " pg_catalog.array_to_string(d.datacl, E'\\n') as \"%s\"",
! gettext_noop("Access Privileges"));
! else
! appendPQExpBuffer(&buf,
! " pg_catalog.array_to_string(d.datacl, '\\n') as \"%s\"",
gettext_noop("Access Privileges"));
if (verbose && pset.sversion >= 80200)
appendPQExpBuffer(&buf,
--- 464,471
" d.datctype as \"%s\",\n",
gettext_noop("Collation"),
gettext_noop("Ctype"));
! appendPQExpBuffer(&buf,
! " d.datacl as \"%s\"",
gettext_noop("Access Privileges"));
if (verbose && pset.sversion >= 80200)
appendPQExpBuffer(&buf,
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>>> "Joshua D. Drake" wrote: > Who has integrated multi-master (transaction and power outage safe) > replication now? As far as I recall, nobody there was that specific about the form of it. PostgreSQL arguably has non-integrated multi-master replication now, and I've seen log-based implementations which operated through daily dial-up connectivity as far back as 1984. (Statewide in Alaska, and city-wide in New York City, neither of which could afford to keep full-time communications up for all the relevant sites.) I did point out the options mentioned here for PostgreSQL: http://en.wikipedia.org/wiki/Multi-master_replication -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 15:07 -0600, Kevin Grittner wrote: > >>> "Dave Page" wrote: > > project that aims to provide it's users with useful features. We > > have two extremely useful features here (hot standby and sync > > replication) which together will make this a killer release for many > > people > > Without taking any particular position on this one way or the other, I > think it's worth noting that when I gave a brief talk about my > experiences with PostgreSQL at Milwaukee BarCamp recently, and > coordinated discussion afterward, the biggest unanswered concern of > people thinking about making the move to PostgreSQL was lack of > integrated "multi-master replication". I'm not entirely sure what > features they would need to make them feel comfortable, but it does > seem to currently be a barrier to migration for some. > > To quantify, there were probably about five or six people out of about > 30 in attendance who seemed particularly concerned about this issue. > Who has integrated multi-master (transaction and power outage safe) replication now? Joshua D. Drake > -Kevin > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 16:02 -0500, Robert Haas wrote: > > However I can say I would be fairly annoyed if everytime I checked > > hackers I was pulling down 5 megs in various patches. > > Oh... really? I thought we were past the day when anyone cared how > large the attachments were. IMO the fact that we email patches at all represents a broken system :P but yes I care how big my attachments are. If -hackers was the only list I was on, I wouldn't care but it obviously isn't. Joshua D. Drake -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>>> "Dave Page" wrote: > project that aims to provide it's users with useful features. We > have two extremely useful features here (hot standby and sync > replication) which together will make this a killer release for many > people Without taking any particular position on this one way or the other, I think it's worth noting that when I gave a brief talk about my experiences with PostgreSQL at Milwaukee BarCamp recently, and coordinated discussion afterward, the biggest unanswered concern of people thinking about making the move to PostgreSQL was lack of integrated "multi-master replication". I'm not entirely sure what features they would need to make them feel comfortable, but it does seem to currently be a barrier to migration for some. To quantify, there were probably about five or six people out of about 30 in attendance who seemed particularly concerned about this issue. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
> However I can say I would be fairly annoyed if everytime I checked > hackers I was pulling down 5 megs in various patches. Oh... really? I thought we were past the day when anyone cared how large the attachments were. At any rate, if we increased the limit from 100k to 1M, you could conceivably get 5M if 5 huge patches had just been posted, but I doubt it would happen every time you checked -hackers. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 21:35 +0100, Stefan Kaltenbrunner wrote: > Tom Lane wrote: > > David Fetter writes: > >> Two things to fix this, and several other problems: > > > >> 1. Remove the messages size limits on -hackers. They serve no useful > >> purpose, and they interfere with our development process. > > > > Agreed, or at least boost it up a good bit more. > > the question really is how much "a bit more" is - right now the limit is >10 characters which limits us to ~70KB of attachments (around the > size of the Hot-standby patch if bzip2 compressed). > > The SE-Postgres patch for example is ~650KB uncompressed - if we want to > cope with uncompressed patches that large we would have to increase > the current limit by a factor of 10 at least. > I wonder if there are people on the list that might not want to receive > mails that large(like users with mobile phones)? Smart mobile phones are not going to pull down the attachment unless the user explicitly says, pull down attachment. However I can say I would be fairly annoyed if everytime I checked hackers I was pulling down 5 megs in various patches. Joshua D. Drake > > > Stefan > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Tom Lane wrote: David Fetter writes: Two things to fix this, and several other problems: 1. Remove the messages size limits on -hackers. They serve no useful purpose, and they interfere with our development process. Agreed, or at least boost it up a good bit more. the question really is how much "a bit more" is - right now the limit is 10 characters which limits us to ~70KB of attachments (around the size of the Hot-standby patch if bzip2 compressed). The SE-Postgres patch for example is ~650KB uncompressed - if we want to cope with uncompressed patches that large we would have to increase the current limit by a factor of 10 at least. I wonder if there are people on the list that might not want to receive mails that large(like users with mobile phones)? Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 12:20 PM, Joshua D. Drake wrote: > > IMO, the reasons to delay a release: > > Our grammar looks like MySQL > mmm... you mean if we add things like VALUES statement, lastval() and things like that? ;) -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL AsesorÃa y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>> I think the problems it would solve for us are (1) emailing huge >> patches around sucks (it sucks unnecessarily because of the >> mailing-list size limit, but even if someone fixes that, it still >> sucks), (2) no need for a CVS-to-GIT conversion that may incur dirty >> reads; (3) retention of history and authorship when merging patches >> into core. It's possible that it might change our workflow in other >> ways too, but even if we got only those three things I think that > > O.k. now the second part :) > > Does bzr, mecurial or monotone offer the same or better solution? Bzr in > particular is in very wide use and I run into mecurial all the time. I'm sure we could make any of them work, but the fact that git.postgresql.org already exists and hg.postgresl.org, bzr.postgresql.org, etc. do not may be suggestive of something - that it would be less work to finish the job, if nothing else. I also think there is some evidence to suggest that git is evolving very rapidly. You could view that as a negative thing, but I don't mean they're fixing bugs: I mean they're adding features. That suggests both that (1) if git doesn't have feature X that you want now, it's likely to have it in the not-too-distant future and (2) a lot of other people like git well enough that they're both using it themselves and contributing changes back to the community, which is an endorsement of the product generally. git definitely has some downsides. The initial repository clone is kind of expensive, and there's a learning curve (though it's much flatter than it was in the past). So I don't think it's the greatest thing ever. But I think it's pretty good, and based on the spike in a/... and b/... paths in recent patches, I'm not the only one. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 02:36:08PM -0500, Robert Haas wrote: > On Mon, Jan 12, 2009 at 1:43 PM, Heikki Linnakangas > wrote: > > Robert Haas wrote: > >> git IS a stable archive of what the patches really were. > > > > No. A developer can delete, move and rebase branches in his own > > repository as he likes, and all of those operations "modify > > history". In fact, a developer can completely destroy or take > > offline his published repository. It's *not* an archive. > > > > There's other reasons why I like git very much over cvs, but > > archiving is not one of them. > > s/IS/CAN BE/, then. > > CVS history can be rewritten, too; it's just harder. We can make a > policy that branches once pushed to git.postgresql.org are not to be > rebased; that's recommended practice with git anyway. I'm not sure > off the top of my head how hard it would be to enforce this in code; > you'd just need to enforce that 'git push' only ever did a > fast-forward. We could do this using git's configuration: http://www.kernel.org/pub/software/scm/git/docs/git-config.html See receive.denyNonFastForwards, which is built for just this purpose :) Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 14:33 -0500, Aidan Van Dyk wrote: > * Joshua D. Drake [090112 14:22]: > > > No. A developer can delete, move and rebase branches in his own > > > repository as he likes, and all of those operations "modify history". In > > > fact, a developer can completely destroy or take offline his published > > > repository. It's *not* an archive. > > > > Yes but I have to pull the whole repo to do it is my point. I can't just > > pull down the 8.3 branch. I have to pull down the whole tree and then > > work on 8.3. > > Not correct. Please, if you're going to say what git "does", please > make sure it's correct. I'm sure people would scream if I said that the > SVN forced you checkout out all of /trunk /branches and /tags (i.e. the > "root" of your SVN repo) into a directory structure simultaneously. They would fall on deaf ears or perhaps on their own flame thrower if they did. > > With git, you pull down the complete *history* of whatever branch, tag, > or reference you want to pull down. The *default* "clone" options are > setup to pull down the history of all available branches and tags, but > that's not mandatory. Oh! O.k. glad to hear it. Then I was misinformed and I am glad that I now know better. > And to top it off, the history in git is usually *smaller* (takes up > less space) than the .svn of a SVN checkout... > > > SVN on the other hand, if I only want to work on trunk, I can check out > > trunk and only work (and commit) into trunk. > > Accept when your busy waiting for the slow SVN operations to do stuff > over the network, you can't do anything... Not to mention, merging, > repeatedly, rebasing, > I am not suggesting that we move to SVN you don't have to start a git is holier than SVN argument. Sincerely, Joshua D. Drake -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 1:43 PM, Heikki Linnakangas wrote: > Robert Haas wrote: >> git IS a stable archive of what the patches really were. > > No. A developer can delete, move and rebase branches in his own repository > as he likes, and all of those operations "modify history". In fact, a > developer can completely destroy or take offline his published repository. > It's *not* an archive. > > There's other reasons why I like git very much over cvs, but archiving is > not one of them. s/IS/CAN BE/, then. CVS history can be rewritten, too; it's just harder. We can make a policy that branches once pushed to git.postgresql.org are not to be rebased; that's recommended practice with git anyway. I'm not sure off the top of my head how hard it would be to enforce this in code; you'd just need to enforce that 'git push' only ever did a fast-forward. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 14:31 -0500, Robert Haas wrote: > > Actually yes we did. There was a bug in git-cvs that we fixed. Its is > > talked about here: > > Actually the work is relatively minimal as we have git infrastructure in > > place. The larger problem is: > > > > What is the problem we are trying to solve? > > Does git actually solve it? > > I think the problems it would solve for us are (1) emailing huge > patches around sucks (it sucks unnecessarily because of the > mailing-list size limit, but even if someone fixes that, it still > sucks), (2) no need for a CVS-to-GIT conversion that may incur dirty > reads; (3) retention of history and authorship when merging patches > into core. It's possible that it might change our workflow in other > ways too, but even if we got only those three things I think that O.k. now the second part :) Does bzr, mecurial or monotone offer the same or better solution? Bzr in particular is in very wide use and I run into mecurial all the time. Sincerely, Joshua D. Drake -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
> Actually yes we did. There was a bug in git-cvs that we fixed. Its is > talked about here: > > http://archives.postgresql.org/pgsql-www/2008-12/msg00182.php > > But... that wasn't really the fault of git. OK, but that's in the past now - good. I thought Tom was saying that it might need to be done again. > There are specific problems with git that people should be aware of > before we start the idea of migrating full to it. The most bothersome to > me is that you must check out the ENTIRE repo. It isn't possible to say: I agree. It's possible that this might change in the future - git has come a long way in a short time. But I'm not betting on it. > Actually the work is relatively minimal as we have git infrastructure in > place. The larger problem is: > > What is the problem we are trying to solve? > Does git actually solve it? I think the problems it would solve for us are (1) emailing huge patches around sucks (it sucks unnecessarily because of the mailing-list size limit, but even if someone fixes that, it still sucks), (2) no need for a CVS-to-GIT conversion that may incur dirty reads; (3) retention of history and authorship when merging patches into core. It's possible that it might change our workflow in other ways too, but even if we got only those three things I think that would be pretty nice. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
* Joshua D. Drake [090112 14:22]: > > No. A developer can delete, move and rebase branches in his own > > repository as he likes, and all of those operations "modify history". In > > fact, a developer can completely destroy or take offline his published > > repository. It's *not* an archive. > > Yes but I have to pull the whole repo to do it is my point. I can't just > pull down the 8.3 branch. I have to pull down the whole tree and then > work on 8.3. Not correct. Please, if you're going to say what git "does", please make sure it's correct. I'm sure people would scream if I said that the SVN forced you checkout out all of /trunk /branches and /tags (i.e. the "root" of your SVN repo) into a directory structure simultaneously. With git, you pull down the complete *history* of whatever branch, tag, or reference you want to pull down. The *default* "clone" options are setup to pull down the history of all available branches and tags, but that's not mandatory. And the benefit of having the whole history of the branch available, is that you can work on the branch *and history* locally, committing, inspecting, reviewing, without needing to go back over the net. And to top it off, the history in git is usually *smaller* (takes up less space) than the .svn of a SVN checkout... > SVN on the other hand, if I only want to work on trunk, I can check out > trunk and only work (and commit) into trunk. Accept when your busy waiting for the slow SVN operations to do stuff over the network, you can't do anything... Not to mention, merging, repeatedly, rebasing, etc ;-) -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] [BUGS] Status of issue 4593
"Kevin Grittner" writes: > Tom Lane wrote: >> If that's what you want then you run the transaction in serializable >> mode. > If you run this at SERIALIZABLE transaction isolation level, would > PostgreSQL currently roll something back before returning rows in an > order different than that specified by the ORDER BY clause? Yes, it would roll back as soon as it found that any of the selected rows had been concurrently modified. The behavior where you get back the updated version of the row is specific to READ COMMITTED mode. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 6:04 PM, Tom Lane wrote: > "Dave Page" writes: >> On Mon, Jan 12, 2009 at 5:20 PM, Joshua D. Drake >> wrote: >>> Well its really nobody's fault except the hacker that didn't step up to >>> do the work. I believe all hackers have already been working diligently. > >> They have - but I see no reason why an imperfect process should delay >> the hard work of developers getting into the hands of users that want >> it for 12 months or more. > > How is it that this argument applies only to work not yet done, as > opposed to work that was already done and committed over the past 12 > months? It doesn't - but those whose work has been committed haven't suffered due to the process. > Really it was possible to foresee this coming months ago. > We knew when we posted > http://archives.postgresql.org/pgsql-hackers/2008-05/msg00913.php > that it was very ambitious to hope for working replication in 8.4. > Then basically nothing happened all summer; Simon didn't ramp up > his effort until around September IIRC. He's done yeoman work > since then, but it can hardly be surprising that we're faced with > a slip-or-cut-the-feature decision now. Simon wasn't working on replication. He's been doing hot standby which has been feature-complete (bar the 2PC stuff which I believe Heikki wanted to hack about in some way) since some time before feature freeze. At this time it's being reviewed and refactored/debugged as a result of the feedback he's received which is precisely what feature freeze is for. The async replication I believe is not in such good shape, having been submitted in a working, but primitive form immediately prior to feature freeze. Although I'd love to see it included in 8.4 (in a form meeting our normal quality requirements of course), I can appreciate it should be bumped if it's not practical to bring it up to par in a reasonable timeframe. I don't believe that decision should be made until it has had a good first review by a couple of committers who can assess what might be required. If it's felt it can then be whipped into shape with a minor delay to the release, then I think it's worth the wait. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 20:43 +0200, Heikki Linnakangas wrote: > Robert Haas wrote: > > git IS a stable archive of what the patches really were. > > No. A developer can delete, move and rebase branches in his own > repository as he likes, and all of those operations "modify history". In > fact, a developer can completely destroy or take offline his published > repository. It's *not* an archive. Yes but I have to pull the whole repo to do it is my point. I can't just pull down the 8.3 branch. I have to pull down the whole tree and then work on 8.3. SVN on the other hand, if I only want to work on trunk, I can check out trunk and only work (and commit) into trunk. > > There's other reasons why I like git very much over cvs, but archiving > is not one of them. Oh don't get me wrong. I am not a CVS user on any level except with PostgreSQL. Joshua D. Drake > > -- >Heikki Linnakangas >EnterpriseDB http://www.enterprisedb.com > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Robert Haas wrote: git IS a stable archive of what the patches really were. No. A developer can delete, move and rebase branches in his own repository as he likes, and all of those operations "modify history". In fact, a developer can completely destroy or take offline his published repository. It's *not* an archive. There's other reasons why I like git very much over cvs, but archiving is not one of them. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, 2009-01-12 at 13:35 -0500, Tom Lane wrote: > I think the behavior Lee is expecting is only implementable with a > full-table write lock, which is exactly what FOR UPDATE is designed > to avoid. There are certain properties you don't get with a partial > lock, and in the end I think we can't do much except document them. > We have LOCK TABLE for those who need the other behavior. > Lee said specifically that he's not using LIMIT, and there's already a pretty visible warning in the docs for using LIMIT with FOR UPDATE. Also, using LIMIT + FOR UPDATE has a dangerous-looking quality to it (at least to me) that would cause me to do a little more investigation before relying on its behavior. I'm not pushing for FOR UPDATE + ORDER BY to be blocked outright, but I think it's strange enough that it should be considered some kind of defect worse than the cases involving LIMIT that you mention. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Robert Haas wrote: That's happened more than once, though my memory of details is fuzzy and I don't have time to troll the archives for them right now. Maybe Bruce can remember without a lot of searching. But our current policy of time-based releases (ie deadlines) is born of hard experience with the negative consequences of saying "we'll release when feature X is ready". The real killer disadvantage is that all other development tends to stop until X is ready, because no one can plan anything. This is a very reasonable concern, and a good policy. But I would feel better about the application of it to this particular case if you, personally, spent a couple of hours reviewing the patches at issue and expressed an opinion about how close they are to being ready to commit. I doubt that many of us would care to substitute our judgment for yours - but it would be a shame to bump them to 8.5 needlessly. Well, I've been keeping an eye on both Hot Standby and Synchronous Replication patches. IMHO the Hot Standby patch is architecturally sound, and while I suggested some pretty big changes just recently (which Simon picked up and did already), it's in pretty good shape. No doubt there's still some issues that haven't been uncovered, comments to be fixed, documentation to be written, but no showstoppers or anything that requires a major rewrite. There's one todo item left: prepared transactions, but I don't think there's anything fundamentally hard about them, just needs to be fixed. Simon mentioned usability issues related to who/when queries get cancelled, but I think we've discussed that to death already and the patch handles it quite nicely. IMHO, the synchronous replication isn't in such good shape, I'm afraid. I've said this before, but I'm not happy with the "built from spare parts" nature of it. You shouldn't have to configure an archive, file-based log shipping using rsync or whatever, and pg_standby. All that is in addition to the direct connection between master and slave. The slave really should be able to just connect to the master, and download all the WAL it needs directly. That's a huge usability issue if left as is, but requires very large architectural changes to fix. One thing I find interesting is that the "Infrastructure Changes for Recovery" patch became the foundation for both "Hot Standby" and "Synchronous Replication". That implies that those changes might be of somewhat more general interest, at least as the foundation for further work. If we HS and/or SR are out of reach, it might be worth at least looking to see if any of that infrastructure work can be reasonably be committed for 8.4. Yeah, being able to do an online checkpoint after recovery has some value of its own. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
>>> Tom Lane wrote: > "Kevin Grittner" writes: >> Tom Lane wrote: >>> A re-sort after locking doesn't really make things all nice and >>> intuitive either. > >> Would it make any sense to roll back and generate a >> SERIALIZATION_FAILURE? > > If that's what you want then you run the transaction in serializable > mode. The point of doing it in READ COMMITTED mode is that you don't > want such a failure. Well, that's a PostgreSQL-specific point of view, although I understand the point of maintaining that guarantee. (In Microsoft SQL Server and Sybase ASE we actually had to run our read-only web application at the READ UNCOMMITTED transaction isolation level because so many SELECT queries were rolled back when they deadlocked with the traffic from replication when they were all running at READ COMMITTED.) If you run this at SERIALIZABLE transaction isolation level, would PostgreSQL currently roll something back before returning rows in an order different than that specified by the ORDER BY clause? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, 2009-01-12 at 14:35 -0400, Jeff Davis wrote: > ate: Mon, 12 Jan 2009 09:52:00 -0800 > On Mon, 2009-01-12 at 08:32 -0500, Tom Lane wrote: > > That code has been working like this for eight or ten years now and > this > > is the first complaint, so taking away functionality on the grounds > that > > someone might happen to update the ordering column doesn't seem like > the > > answer to me. > > > > If they are using FOR UPDATE, they clearly expect concurrent updates. > If they're using ORDER BY, they clearly expect the results to be in > order. > > So who is the target user of this functionality we're trying to > protect? If by the question above you are asking for a rational use-case, I think I can give you one. In my Oracle days I used to use select for update order by, in order to reduce the likelihood of deadlocks. If locks are always taken in the same order, deadlocks become considerably less likely. Unfortunately, I took this construct, which as far as I know works fine in Oracle, and continued to use it in Postgres just assuming that it would work. At least now the source of some of my more mysterious deadlocks is apparent :-) I'd second the request for at least a warning to be issued. __ Marc signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [BUGS] Status of issue 4593
"Kevin Grittner" writes: > Tom Lane wrote: >> A re-sort after locking doesn't really make things all nice and >> intuitive either. > Would it make any sense to roll back and generate a > SERIALIZATION_FAILURE? If that's what you want then you run the transaction in serializable mode. The point of doing it in READ COMMITTED mode is that you don't want such a failure. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
>>> Tom Lane wrote: > A re-sort after locking doesn't really make things all nice and > intuitive either. Suppose that the values of X are 10,20,30,40,50 > and we do SELECT ... ORDER BY x LIMIT 3 FOR UPDATE. Concurrently > someone updates the 20 to 100. The existing code locks the 10,20,30 > rows, then notices the 20 got updated to 100, and returns you > 10,100,30. If it re-sorted you would get 10,30,100, but on what > grounds is that the correct answer and not 10,20,40? If you want > to argue that 10,20,40 is the correct answer, how are you going to > arrive at it without locking more rows than are returned? Would it make any sense to roll back and generate a SERIALIZATION_FAILURE? In other databases I've seen this failure code used for all situations where they can't perform the requested operation due to concurrent transactions, regardless of transaction isolation level. That seems more intuitive to me than any of the alternatives discussed so far, which all seem to involve quietly returning something other than what was specified. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] autovacuum and reloptions
Euler Taveira de Oliveira escribió: > Alvaro Herrera escreveu: > > Alvaro Herrera escribió: > > > >> I have a separate branch on which I keep the old patch from Euler > >> updated to the current reloptions code; so it is probably very similar > >> to what Euler just sent. (I'll have a look at that soon anyway.) > > > > Huh, nevermind -- I thought that Euler had just sent an updated version > > of his patch, but only now I noticed that the message I was looking at > > is dated Nov. 21st. > > > And I did it (It is even listed in the wiki). Last version was [1]. I fixed > some problems and added some docs. Yes, the autovacuum patch I have is loosely based on that one IIRC. (I modified it heavily though.) -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 13:04 -0500, Tom Lane wrote: > Simon didn't ramp up > his effort until around September IIRC. The main topic of snapshot creation was being discussed at PGcon in May and another sponsors got serious then. I started working on a coherent detailed design in July, but didn't publish for another month while I waited for sponsors to confirm. The shape of the patch was coming together over previous months. I published the "infrastructure" patch on 1 Sept, which is about 35% of current patch, deliberately to allow Sync Rep to integrate. > He's done yeoman work since then, Thanks. > but it can hardly be surprising that we're faced with > a slip-or-cut-the-feature decision now. It was always going to be tight; I said as much to Bruce last February and reconfirmed that in May after talking with Suzuki-san's team. I could have ignored Sync Rep completely and made it less tight for myself and more difficult for the project. How much are we going to slip by? Async Commit was committed to CVS on 1 Aug 2007, exactly 4 months after deadline for 8.3. At moment we are 2.4 months part deadline on 8.4. We hoped 8.4 would be different: it has been - we have got much much more done in the same time, just we have a long tail again. I think we should just deal with things as they are now and put in a rule for next time that we auto-chop/no discussion any big patches submitted on last commit fest that haven't already had feedback from earlier fests. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
"Robert Haas" writes: > On Mon, Jan 12, 2009 at 8:32 AM, Tom Lane wrote: >> That code has been working like this for eight or ten years now and this >> is the first complaint, so taking away functionality on the grounds that >> someone might happen to update the ordering column doesn't seem like the >> answer to me. > If the only case where ORDER BY + FOR UPDATE are not strictly > compatible is when the columns being updated are the same as the > columns of the sort, a blanket prohibition against using the two > together seems like it prohibits an awful lot of useful things someone > might want to do. Exactly. > Saying that you can run the query a second time as > a workaround so there's no loss of functionality is true only if you > accept the proposition that performance is not a requirement. Right, and also I'm unconvinced that it is really equivalent. If you've got something like an ORDER BY LIMIT and the ordering columns are changing, you may very well get a different set of rows from the second query ... not all of which got locked the first time, so there's going to be some tail-chasing involved there. A re-sort after locking doesn't really make things all nice and intuitive either. Suppose that the values of X are 10,20,30,40,50 and we do SELECT ... ORDER BY x LIMIT 3 FOR UPDATE. Concurrently someone updates the 20 to 100. The existing code locks the 10,20,30 rows, then notices the 20 got updated to 100, and returns you 10,100,30. If it re-sorted you would get 10,30,100, but on what grounds is that the correct answer and not 10,20,40? If you want to argue that 10,20,40 is the correct answer, how are you going to arrive at it without locking more rows than are returned? And just to bend your brain a bit more, what if the same update command that changed 20 to 100 also changed 50 to 1? Surely if we take the one row change into account in determining the sort order, we ought to notice that one too. However, we aren't even going to *see* that row unless we traverse the entire table. I think the behavior Lee is expecting is only implementable with a full-table write lock, which is exactly what FOR UPDATE is designed to avoid. There are certain properties you don't get with a partial lock, and in the end I think we can't do much except document them. We have LOCK TABLE for those who need the other behavior. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 13:23 -0500, Robert Haas wrote: > >> No, git really does help with this. If Simon were making his changes > >> in git and pushing them to a git branch on git.postgresql.org, you > >> would be able to see exactly what he changed and when he changed it. > > > > Well, if that's actually an archival repository then it would work. > > But wasn't I just reading something about having to wipe that repository > > and re-import the CVS history to fix various problems? > > Not sure; I hope not. Actually yes we did. There was a bug in git-cvs that we fixed. Its is talked about here: http://archives.postgresql.org/pgsql-www/2008-12/msg00182.php But... that wasn't really the fault of git. > I think we'd be well-served by getting rid of > CVS permanently and using git for the master branch. That would > eliminate the possibility of git reading a partial commit from CVS and > any other possible issues of needing to go back and reconstruct git > things based on unexpected wankage in the CVS repository. We could > keep the list of committers exactly the same as what it is now; they'd > just be people with rights to push the master git branch rather than > rights to commit to CVS. > There are specific problems with git that people should be aware of before we start the idea of migrating full to it. The most bothersome to me is that you must check out the ENTIRE repo. It isn't possible to say: git clone https://git.postgresql.org/postgresql/7.3 It is all or nothing. I know why this is but that doesn't mean I like it :) > I am sure this would involve a fair amount of work but I think it Actually the work is relatively minimal as we have git infrastructure in place. The larger problem is: What is the problem we are trying to solve? Does git actually solve it? Sincerely, Joshua D. Drake -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
> That's happened more than once, though my memory of details is fuzzy > and I don't have time to troll the archives for them right now. > Maybe Bruce can remember without a lot of searching. But our current > policy of time-based releases (ie deadlines) is born of hard experience > with the negative consequences of saying "we'll release when feature X > is ready". The real killer disadvantage is that all other development > tends to stop until X is ready, because no one can plan anything. This is a very reasonable concern, and a good policy. But I would feel better about the application of it to this particular case if you, personally, spent a couple of hours reviewing the patches at issue and expressed an opinion about how close they are to being ready to commit. I doubt that many of us would care to substitute our judgment for yours - but it would be a shame to bump them to 8.5 needlessly. One thing I find interesting is that the "Infrastructure Changes for Recovery" patch became the foundation for both "Hot Standby" and "Synchronous Replication". That implies that those changes might be of somewhat more general interest, at least as the foundation for further work. If we HS and/or SR are out of reach, it might be worth at least looking to see if any of that infrastructure work can be reasonably be committed for 8.4. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks
Please commit soon On Fri, 2009-01-09 at 18:30 +0200, Heikki Linnakangas wrote: > The hot standby patch has some hacks to decide which full-page-images > can be restored holding an exclusive lock and which ones need a > vacuum-strength lock. It's not very pretty as is, as mentioned in > comments too. > > How about we refactor things so that redo-functions are responsible for > calling RestoreBkpBlocks? The redo function can then pass an argument > indicating what kind of lock is required. We should also change > XLogReadBufferExtended so that it doesn't lock the page; the caller > knows better what kind of a lock it needs. That makes it more analogous > with ReadBufferExtended too, although I think we should keep > XLogReadBuffer() unchanged for now. > > See attached patch. One shortfall of this patch is that you can pass > only one argument to RestoreBkpBlocks, but there can multiple backup > blocks in one WAL record. That's OK in current usage, though. > -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>> No, git really does help with this. If Simon were making his changes >> in git and pushing them to a git branch on git.postgresql.org, you >> would be able to see exactly what he changed and when he changed it. > > Well, if that's actually an archival repository then it would work. > But wasn't I just reading something about having to wipe that repository > and re-import the CVS history to fix various problems? Not sure; I hope not. I think we'd be well-served by getting rid of CVS permanently and using git for the master branch. That would eliminate the possibility of git reading a partial commit from CVS and any other possible issues of needing to go back and reconstruct git things based on unexpected wankage in the CVS repository. We could keep the list of committers exactly the same as what it is now; they'd just be people with rights to push the master git branch rather than rights to commit to CVS. I am sure this would involve a fair amount of work but I think it would be worth it, and I'd be willing to help with the doing of it. I have resisted learning git for a while but I've come around. I'm even switching to git for development work I do for my employer, where I'm the only developer, because it's just so much easier to do work on a branch and then merge it than it is with CVS or SVN. > (In any case, the URLs I'm complaining of weren't pointing at > git.postgresql.org, but various private servers or wiki pages.) Agreed. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
"Christopher Browne" writes: > On Mon, Jan 12, 2009 at 12:27 PM, Dave Page wrote: >> On Mon, Jan 12, 2009 at 5:19 PM, Tom Lane wrote: >>> In general, we have always regretted it in the past when we chose to >>> slip a release waiting for a specific feature... >> >> I don't recall such a time - though perhaps the last time it happened >> was before I was so heavily involved in the release process (ie. 7.x). >> What were the reasons for regretting it? > I seem to recall us deferring 8.1 (was it 8.1?) for a while on the > basis that we were waiting for [something I don't recall offhand]. > The feature that we were *hoping* to get wound up dropped on the floor > because it just wasn't ready, even *with* the extra time. That's happened more than once, though my memory of details is fuzzy and I don't have time to troll the archives for them right now. Maybe Bruce can remember without a lot of searching. But our current policy of time-based releases (ie deadlines) is born of hard experience with the negative consequences of saying "we'll release when feature X is ready". The real killer disadvantage is that all other development tends to stop until X is ready, because no one can plan anything. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] autovacuum and reloptions
Alvaro Herrera escreveu: > Alvaro Herrera escribió: > >> I have a separate branch on which I keep the old patch from Euler >> updated to the current reloptions code; so it is probably very similar >> to what Euler just sent. (I'll have a look at that soon anyway.) > > Huh, nevermind -- I thought that Euler had just sent an updated version > of his patch, but only now I noticed that the message I was looking at > is dated Nov. 21st. > And I did it (It is even listed in the wiki). Last version was [1]. I fixed some problems and added some docs. [1] http://archives.postgresql.org/pgsql-hackers/2008-12/msg00292.php -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
"Robert Haas" writes: >> This is a red herring, unless your proposal also includes making the >> master CVS^H^H^Hgit repository world-writable. The complaint I have >> about people posting URLs is that there's no stable archive of what the >> patches really were, and just because it came out of someone's local git >> repository doesn't help that. > No, git really does help with this. If Simon were making his changes > in git and pushing them to a git branch on git.postgresql.org, you > would be able to see exactly what he changed and when he changed it. Well, if that's actually an archival repository then it would work. But wasn't I just reading something about having to wipe that repository and re-import the CVS history to fix various problems? (In any case, the URLs I'm complaining of weren't pointing at git.postgresql.org, but various private servers or wiki pages.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 5:30 PM, Joshua D. Drake wrote: > On Mon, 2009-01-12 at 17:27 +, Dave Page wrote: > >> > In general, we have always regretted it in the past when we chose to >> > slip a release waiting for a specific feature... >> >> I don't recall such a time - though perhaps the last time it happened >> was before I was so heavily involved in the release process (ie. 7.x). >> What were the reasons for regretting it? > > 8.2 suffered from horrendous slip. It wasn't delayed for a specific feature though was it? Just because there was so much in the queue it took far longer than planned. Plus it was originally intended as a half-length cycle which clearly didn't work for a number of reasons. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
"Dave Page" writes: > On Mon, Jan 12, 2009 at 5:20 PM, Joshua D. Drake > wrote: >> Well its really nobody's fault except the hacker that didn't step up to >> do the work. I believe all hackers have already been working diligently. > They have - but I see no reason why an imperfect process should delay > the hard work of developers getting into the hands of users that want > it for 12 months or more. How is it that this argument applies only to work not yet done, as opposed to work that was already done and committed over the past 12 months? > It'll annoy users and potentially alienate > important developers - and there are few enough of them able to work > on features of this complexity as it is. Well, we can alienate developers who get annoyed because we won't slip the schedule for their convenience, or we can alienate the ones who met the agreed-on schedule and don't get to see their work shipped in a timely fashion. Really it was possible to foresee this coming months ago. We knew when we posted http://archives.postgresql.org/pgsql-hackers/2008-05/msg00913.php that it was very ambitious to hope for working replication in 8.4. Then basically nothing happened all summer; Simon didn't ramp up his effort until around September IIRC. He's done yeoman work since then, but it can hardly be surprising that we're faced with a slip-or-cut-the-feature decision now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, 2009-01-12 at 12:47 -0500, Robert Haas wrote: > If the only case where ORDER BY + FOR UPDATE are not strictly > compatible is when the columns being updated are the same as the > columns of the sort, a blanket prohibition against using the two > together seems like it prohibits an awful lot of useful things someone > might want to do. As long as the people using it are aware that they can't update the ordering columns, it may make sense to leave that functionality in there. Can you expand on "an awful lot of useful things"? It seems like an edge case to me, and the restriction it imposes is quite awkward to meet. "OK, nobody ever update these fields in this table.". Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Robert Haas wrote: >>> 2. Start using more git... >> This is a red herring, unless your proposal also includes making the >> master CVS^H^H^Hgit repository world-writable. The complaint I have >> about people posting URLs is that there's no stable archive of what the >> patches really were, and just because it came out of someone's local git >> repository doesn't help that. > > No, git really does help with this. ... > git IS a stable archive of what the patches really were. Sorry to re-ignite the flame war, but this is the *perfect* example of the singlemost compelling advantage git over cvs. All of Simon's history remains visible in git on his branch. Better - any patches submitted to Simon by code reviewers that Simon accepts (pulls) into his branch - can also be seen on branches off of Simon's branch with the complete history of where they came from. When/if the patch eventually gets accepted into the master, as as much (or as little, thanks to git-rebase) of the history of that branch can be pulled along with it; as can be seen with the major merges of linux branches here: http://repo.or.cz/git-browser/by-commit.html?r=linux-2.6.git There's no need for the master git to be world-writable. The few with write access choose exactly how much history from Simon's branch (and from the code review's branches) they want to merge in when they pull&merge from his branch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, 2009-01-12 at 08:32 -0500, Tom Lane wrote: > That code has been working like this for eight or ten years now and this > is the first complaint, so taking away functionality on the grounds that > someone might happen to update the ordering column doesn't seem like the > answer to me. > If they are using FOR UPDATE, they clearly expect concurrent updates. If they're using ORDER BY, they clearly expect the results to be in order. So who is the target user of this functionality we're trying to protect? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, Jan 12, 2009 at 8:32 AM, Tom Lane wrote: > Peter Eisentraut writes: >> I can see two ways forward: > >> 1) We document bluntly that ORDER BY + FOR UPDATE can return unordered >> results, or > >> 2) We prohibit ORDER BY + FOR UPDATE, like we do with a number of other >> clauses. (There would be no loss of functionality, because you can run >> the query a second time in the transaction with ORDER BY.) > > That code has been working like this for eight or ten years now and this > is the first complaint, so taking away functionality on the grounds that > someone might happen to update the ordering column doesn't seem like the > answer to me. If the only case where ORDER BY + FOR UPDATE are not strictly compatible is when the columns being updated are the same as the columns of the sort, a blanket prohibition against using the two together seems like it prohibits an awful lot of useful things someone might want to do. Saying that you can run the query a second time as a workaround so there's no loss of functionality is true only if you accept the proposition that performance is not a requirement. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 08:37 -0800, Joshua D. Drake wrote: > On Mon, 2009-01-12 at 11:18 -0500, Tom Lane wrote: > > > > My point is that what Simon currently has (and so what you tested) is > > > different from what is going to be commited (note the "final" in what > > > I wrote) and I suspect there will be a certain number of non > > > negligible adjustments (see the last discussions between Simon and > > > Heikki and I don't think Tom has taken a look at these patches yet). > > > > The thing that's disturbing me is that (to judge by what I've been > > seeing on the mailing list) there's been a steady stream of "non > > negligible adjustments" for the past two months. That's good from > > the standpoint that problems are getting found and fixed, but it's > > not giving me any warm fuzzies about the code being ready to go. > > This is the same thing that makes me nervous. The feature appears to be > "Under heavy development". As I understand the development model the > heavy development is supposed to happen before commit fest. That simply not true. So far there have been 15 resolved issues, shown on Wiki 2 correctness issues, critical if not resolvable, but easily done 1 trivial porting issue 1 minor issue related to disallowed read-write modes 1 minor refactoring 3 minor issues resolved to no action 6 bugs, all easily diagnosed and fixed plus 1 major refactoring, that required significant examination of internals to achieve, but has not changed anything that we now agree is significant (that also was a discussion point). Much of the difficulty caused because the design needs to be both correct and performant, which I believe it is. So far this patch has had much less refactoring than async commit, which reached version > 22 before it was committed. There are 322 separate code chunks touching 69 files, with more than 9000 lines of code (>11000 including patch diffs). In percentage terms, that's pretty good "code quality". There are two main oustanding tasks, which will be complete over the next couple of days. * prepared transactions - left because I knew Heikki wanted to refactor things in a way that would rip out any code written for that part * usability issues related to how/when queries get cancelled. That came out of discussion during review and was not an outstanding item before deadline. There have been no changes to the main mechanisms in the patch, nor in the user interface related parts. To be honest we might reasonably have expected *more* change to have taken place than has done. Some dev might sound like heavy lifting, because the code often involves the internal mechanics of the transaction system, but this is hardly either my or Heikki's first foray into that section of code and we're used to hitting problems and solving them. Anyway, I'm actually going to turn off my email for a few days. Talking won't get it done and that way I can pretend you're all sending me positive waves (not to pick specifically on you Josh). -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 12:27 PM, Dave Page wrote: > On Mon, Jan 12, 2009 at 5:19 PM, Tom Lane wrote: >> In general, we have always regretted it in the past when we chose to >> slip a release waiting for a specific feature... > > I don't recall such a time - though perhaps the last time it happened > was before I was so heavily involved in the release process (ie. 7.x). > What were the reasons for regretting it? I seem to recall us deferring 8.1 (was it 8.1?) for a while on the basis that we were waiting for [something I don't recall offhand]. The feature that we were *hoping* to get wound up dropped on the floor because it just wasn't ready, even *with* the extra time. A good reason to regret the slippage would be if that slippage didn't allow bringing in the new feature. That does seem pretty relevant here: it sure would be a shame if we put off 8.4 because we *hoped* extra time would allow us to get hot standby into place, only to find that it wasn't ready because it *really* Wasn't Ready. We would have several things to regret: Irrespective of the state of Hot Standby, we would have the following regrets: - 8.4 would be put off, which is of nonzero cost - 8.5 would also be put off - Features deferred for 8.5 would be put off further - Developers working on features other than Hot Standby have the fruits of their efforts deferred If Hot Standby turns out to be "8.4-worthy", then there is an anti-regret: - We get Hot Standby in 8.4 On the other hand, if Hot Standby turns out to *not* be ready enough, then add the further regret: - We waited, and deferred more functionality to 8.5, without any attendant gain Each one of the "regrets" is fairly material. I'm not arguing (at the moment) against taking time to put in Hot Standby, but there is certainly grist available to make that argument. -- http://linuxfinances.info/info/linuxdistributions.html Eddie Izzard - "I grew up in Europe, where the history comes from." -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] Status of issue 4593
On Mon, 2009-01-12 at 15:26 +0200, Peter Eisentraut wrote: > 1) We document bluntly that ORDER BY + FOR UPDATE can return unordered > results, or > > 2) We prohibit ORDER BY + FOR UPDATE, like we do with a number of other > clauses. (There would be no loss of functionality, because you can run > the query a second time in the transaction with ORDER BY.) > I like Lee's idea of a WARNING plus a documentation note -- seems like a reasonable compromise. Maybe we can add the prohibition later if we still don't have a fix for it. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Tom Lane writes: > Yeah, but there are already a number of things in 8.4 that are killer > features for various applications --- window functions and WITH to take > two recently-committed examples. Should we sit on those for however > long it will take to make replication release-worthy? Do we know it's not release-worthy now? From what I see Heikki is proposing refactorings which improve the code but hasn't found anything actually broken. I'm all for cleaner simpler code -- especially in critical backup processes since simpler means safer -- but just because there are better ways to do things doesn't mean the current code isn't acceptable. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
>> 2. Start using more git, as many hackers and committers have already >> started to do. This is the kind of situation where CVS just plain >> falls down because branching and merging are unmanageably difficult in >> it, where in git, they're many-times-a-day operations. > > This is a red herring, unless your proposal also includes making the > master CVS^H^H^Hgit repository world-writable. The complaint I have > about people posting URLs is that there's no stable archive of what the > patches really were, and just because it came out of someone's local git > repository doesn't help that. No, git really does help with this. If Simon were making his changes in git and pushing them to a git branch on git.postgresql.org, you would be able to see exactly what he changed and when he changed it. You would therefore be able to assess whether the changes over the last several months were or were not substantial. The whole point of git is that you don't just publish the master branch - everyone can have their own branches, and they can all be published, and everyone can see the whole development history of every project for whatever purpose they care to see it. git IS a stable archive of what the patches really were. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 5:20 PM, Joshua D. Drake wrote: > The community are our shareholders. Exactly - and their dividends are the features we release, not a share of profits we make from pushing out something a few weeks earlier. > Right. Except that isn't really the question at hand is it? The above is > just a potential result of the question at hand. The low level question > is, "do we feel comfortable from a technical (not a whiz bang) level > with the diligence that has been provided this code. I always feel very confident knowing that it won't be committed until it's right. > Well its really nobody's fault except the hacker that didn't step up to > do the work. I believe all hackers have already been working diligently. They have - but I see no reason why an imperfect process should delay the hard work of developers getting into the hands of users that want it for 12 months or more. It'll annoy users and potentially alienate important developers - and there are few enough of them able to work on features of this complexity as it is. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgsql: Tweak order of operations in BitmapHeapNext() to avoid the case
Gregory Stark writes: > t...@postgresql.org (Tom Lane) writes: >> Tweak order of operations in BitmapHeapNext() to avoid the case of >> prefetching >> the same page we are nanoseconds away from reading for real. There should be >> something left to do on the current page before we consider issuing a >> prefetch. > Doesn't this break things if, say, there's precisely one tuple on every page? No. It'd only break things if there were zero tuples on each page, but that's not possible (else the page wouldn't be in the index output to begin with). There is a bit of oddness in how fast the target ramps up, because of the target++ in the "Continuing in previously obtained page" path. I wasn't too happy with that target++ to begin with, but am not sure what's a saner thing to do there. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 17:27 +, Dave Page wrote: > > In general, we have always regretted it in the past when we chose to > > slip a release waiting for a specific feature... > > I don't recall such a time - though perhaps the last time it happened > was before I was so heavily involved in the release process (ie. 7.x). > What were the reasons for regretting it? 8.2 suffered from horrendous slip. Joshua D. Drake > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 5:19 PM, Tom Lane wrote: > "Dave Page" writes: > Yeah, but there are already a number of things in 8.4 that are killer > features for various applications --- window functions and WITH to take > two recently-committed examples. Should we sit on those for however > long it will take to make replication release-worthy? When chatting with 'real' users (as opposed to us lot) at LinuxLive late last year, sync rep and hot standby were the features they all seemed to want. The others are excellent additions as well of course, but most people didn't seem as interested. > In general, we have always regretted it in the past when we chose to > slip a release waiting for a specific feature... I don't recall such a time - though perhaps the last time it happened was before I was so heavily involved in the release process (ie. 7.x). What were the reasons for regretting it? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] per-database locale: createdb switches
Peter Eisentraut wrote: > Bruce Momjian wrote: > > Heikki Linnakangas wrote: > >> Alvaro Herrera wrote: > >>> Tom Lane wrote: > Alvaro Herrera writes: > > Alvaro Herrera wrote: > >> I like Teodor's proposal; I'll see about implementing that. > > Attached. > You missed updating the sgml docs, and personally I'd be inclined to > list -l before the individual --lc switches; otherwise it looks fine. > >>> Thanks, committed that way. I noticed that --lc-ctype and --lc-collate > >>> were forgotten in SGML docs, so I added them too. > >> Should we have a shorthand CREATE DATABASE option like that as well? > > > > createdb is really about convenience; not sure it is warranted for > > CREATE DATABASE. > > I think unless you are doing something completely funny, you would > usually want to have COLLATE and CTYPE equal. The fact that you now > have to enter both to get that result could be pretty annoying in > practice, I would think. I agree but I can't think of many cases where we offer one option which controls two other options; can you? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 16:48 +, Dave Page wrote: > On Mon, Jan 12, 2009 at 4:37 PM, Joshua D. Drake > wrote: > > On Mon, 2009-01-12 at 11:18 -0500, Tom Lane wrote: > > > >> Basically I think we are up against the same type of project management > >> decision we've had several times before: are we willing to slip the > >> 8.4 release schedule for however long it will take for hot standby > >> and the other replication-related features to be ready? > > > > I would certainly not like to see 8.4 slip. > > I would. PostgreSQL is not a commercial application which has to be > released on schedule to satisfy shareholders - it's an Open Source > project that aims to provide it's users with useful features. The community are our shareholders. > We have > two extremely useful features here (hot standby and sync replication) > which together will make this a killer release for many people - we And for others we already have a killer release. > can delay a month or two as required to polish and get them ready for > release, or decide we're willing to wait another 12 - 14 months for > them to be available for end users. Right. Except that isn't really the question at hand is it? The above is just a potential result of the question at hand. The low level question is, "do we feel comfortable from a technical (not a whiz bang) level with the diligence that has been provided this code. If we don't it should push to 8.5. These are "features" not core requirements of the product. They can wait until another release if need be. We already have a gargantuan list of whiz bang features in this release. IMO, the reasons to delay a release: We broke autovacuum MVCC is no longer MVCC Our grammar looks like MySQL Constraints no longer constrain Not: I want super duper feature. > > I'd much rather see them included than deferred (particularly hot > standby, parts of which have been awaiting review for months now > anyway, through no fault of Simons). > Well its really nobody's fault except the hacker that didn't step up to do the work. I believe all hackers have already been working diligently. Sincerely, Joshua D. Drake -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
Peter Eisentraut wrote: > Simon Riggs wrote: >> Recovery doesn't have a test framework as yet. > > I have been having these concerns as well. In fact, I recall > discussions at least 8 years back about how pg_dump doesn't really have > any organized testing, and we also have little regular testing of PITR > aside from specific exercises that users or developers occasionally run. > > The question remains how to do it. A very simple way to start would be to stop emitting shutdown checkpoints. This would mean that any server restart would cause recovery. This wouldn't be exhaustive, but enough people doing that should cause at least some bugs to emerge. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
"Dave Page" writes: > On Mon, Jan 12, 2009 at 4:37 PM, Joshua D. Drake > wrote: >> I would certainly not like to see 8.4 slip. > I would. PostgreSQL is not a commercial application which has to be > released on schedule to satisfy shareholders - it's an Open Source > project that aims to provide it's users with useful features. We have > two extremely useful features here (hot standby and sync replication) > which together will make this a killer release for many people Yeah, but there are already a number of things in 8.4 that are killer features for various applications --- window functions and WITH to take two recently-committed examples. Should we sit on those for however long it will take to make replication release-worthy? In general, we have always regretted it in the past when we chose to slip a release waiting for a specific feature... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal: new border setting in psql
Tom Lane wrote: > Alvaro Herrera writes: > > My vote goes for 1. > > > I wonder why you think it's impossible. Is it because you must scan > > the whole table before being able to print any of it? (For example to > > add extra alignment for the escaping backslashes in a way that doesn't > > render it invalid.) Note that psql already does that in aligned mode, > > to determine the wide of the columns. > > Hmm. If RST is really that brain-damaged, the problem here is that it's > going to take a very large patch to make it work. It's not going to be > a "border" option but a separate output mode like latex or html. And > then we have to get into the discussion of whether there's really enough > demand for this to justify carrying such a large chunk of code. > > In any case, my vote is for either 0 or 1; I'm unimpressed by anything > that emits RST-except-we-skipped-all-the-hard-parts. Yep, that is my analysis as well. If you want a pretty ReST-like output, that can be added later. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgsql: Tweak order of operations in BitmapHeapNext() to avoid the case
t...@postgresql.org (Tom Lane) writes: > Log Message: > --- > Tweak order of operations in BitmapHeapNext() to avoid the case of prefetching > the same page we are nanoseconds away from reading for real. There should be > something left to do on the current page before we consider issuing a > prefetch. Doesn't this break things if, say, there's precisely one tuple on every page? You'll keep raising the prefetch_target but never actually prefetch anything. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 5:48 PM, Dave Page wrote: > I would. PostgreSQL is not a commercial application which has to be > released on schedule to satisfy shareholders - it's an Open Source > project that aims to provide it's users with useful features. It has nothing to do with commercial/non commercial. It's basically a decision of time based releases vs features based releases. For many years now, new versions of PostgreSQL have been released on a time based schedule (one version/year in december) even if it was not a strong decision. > We have > two extremely useful features here (hot standby and sync replication) > which together will make this a killer release for many people - we > can delay a month or two as required to polish and get them ready for > release, or decide we're willing to wait another 12 - 14 months for > them to be available for end users. IMHO, if it takes 4 months to have these patches in the tree, it's not worth it: do we accept other patches during this period or not? If so, on which basis, if not how are we going to deal with 3-4 months of patches waiting for review. Note that delaying 8.4 is also delaying the other features of 8.4 which are ready (new FSM, CTE, Windowing functions). Personnaly, integrated replication is by far the most expected feature but I'm not sure it's the case for everyone, especially if they have to wait 3-4 more months. It's really a matter of how far we are of having these patches in their final form (and I mean after reviewing). And as Tom stated, it's currently hard to know. IMHO, Simon's proposal to identify which parts especially needs attention is a very good idea. I really think these patches need a thorough review sooner than later: we won't make it happen by letting Simon write code alone without feedback. After this first review, we should be able to know if it's a matter of 1 month or 3. Without knowing this, it's hard to take a good decision. -- Guillaume -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore -1 vs -C and -c
Magnus Hagander writes: > On 12 jan 2009, at 17.46, Tom Lane wrote: >> However, one of the properties -1 is supposed to have is that any >> failure aborts the whole restore; it's not immediately clear how to >> preserve that with CREATE DATABASE issued separately. > Good point. Declare as incompatible it is, then :) it's not like it's > hard do create the database before restoring. Works for me. >>> As for -c, the solution would be to issue DROP IF EXISTS >>> statements. Is there any particular reason why we don't? >> >> I think we did that to avoid damaging portability and backwards >> compatibility of the dump files. The backwards compatibility argument >> is pretty weak by now, but the "it's not standard SQL" argument still >> has force. > IIRC the drop statements are generated by pg_restore and not stored in > the archive. So we could do the if exists by default and have a switch > to turn it off for a compatible dump, perhaps? No, the text of the statements is in the archive; though it might not be too painful to have pg_restore edit them to insert "IF EXISTS". You don't need an extra switch, just do this if -1 is in use (and document that that switch reduces the standard-ness of the output...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On 2009-01-12, at 16:48, Dave Page wrote: On Mon, Jan 12, 2009 at 4:37 PM, Joshua D. Drake wrote: On Mon, 2009-01-12 at 11:18 -0500, Tom Lane wrote: Basically I think we are up against the same type of project management decision we've had several times before: are we willing to slip the 8.4 release schedule for however long it will take for hot standby and the other replication-related features to be ready? I would certainly not like to see 8.4 slip. I would. PostgreSQL is not a commercial application which has to be released on schedule to satisfy shareholders - it's an Open Source project that aims to provide it's users with useful features. We have two extremely useful features here (hot standby and sync replication) which together will make this a killer release for many people - we can delay a month or two as required to polish and get them ready for release, or decide we're willing to wait another 12 - 14 months for them to be available for end users. I'd much rather see them included than deferred (particularly hot standby, parts of which have been awaiting review for months now anyway, through no fault of Simons). +1 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore -1 vs -C and -c
On 12 jan 2009, at 17.46, Tom Lane wrote: Magnus Hagander writes: It should be possible to make it compatible with -C by moving the CREATE DATABASE command to outside of the transaction. I have only had a quick look at the code wrt how much work this would be. One thing that hit me quickly: do we support multiple CREATE DATABASE statements in a single dump file? I think not, but the format seems to allow it? If not, it should be "fairly simple" to do. We don't, and a single transaction couldn't restore into multiple databases anyway. So in principle there's no reason we shouldn't allow it to do the CREATE DATABASE, switch into the new DB, and then start the transaction. However, one of the properties -1 is supposed to have is that any failure aborts the whole restore; it's not immediately clear how to preserve that with CREATE DATABASE issued separately. Good point. Declare as incompatible it is, then :) it's not like it's hard do create the database before restoring. Also you need to think about whether this might impact pg_dumpall. Dumps from pg_dumpall aren't restored through pg_restore... As for -c, the solution would be to issue DROP IF EXISTS statements. Is there any particular reason why we don't? I think we did that to avoid damaging portability and backwards compatibility of the dump files. The backwards compatibility argument is pretty weak by now, but the "it's not standard SQL" argument still has force. IIRC the drop statements are generated by pg_restore and not stored in the archive. So we could do the if exists by default and have a switch to turn it off for a compatible dump, perhaps? /Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 05:50:19PM +0100, Magnus Hagander wrote: 2. Start using more git, as many hackers and committers have already started to do. This is the kind of situation where CVS just plain falls down because branching and merging are unmanageably difficult in it, where in git, they're many-times-a-day operations. >>> >>> This is a red herring, unless your proposal also includes making >>> the master CVS^H^H^Hgit repository world-writable. The complaint >>> I have about people posting URLs is that there's no stable archive >>> of what the patches really were, and just because it came out of >>> someone's local git repository doesn't help that. >> >> The master repository need not be world-writeable, but as many >> public ones as needed for development should be. I'd love for >> people to use our infrastructure, but github, etc., would also >> work. > > As much as I'm starting to join the "let's move the main repo to > git" crowd, all you need for what you're suggesting here is a > stable git mirror on git.postgresql.org. Agreed :) Cheers, David (happy to help by setting people up on git.postgresql.org). -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On 12 jan 2009, at 17.42, David Fetter wrote: On Mon, Jan 12, 2009 at 11:33:43AM -0500, Tom Lane wrote: David Fetter writes: Two things to fix this, and several other problems: 1. Remove the messages size limits on -hackers. They serve no useful purpose, and they interfere with our development process. Agreed, or at least boost it up a good bit more. If -hackers isn't already subscriber-only, now would be the time to make it so. Not sure how that's relevant? Spam and wackiness. Consider what Dmitry Turin would do with an unlimited ability to send his "specs" to -hackers. It is. Anything submitted from addresses not subscribed is moderated. I tested this by mistake today by sending from the wrong address. 2. Start using more git, as many hackers and committers have already started to do. This is the kind of situation where CVS just plain falls down because branching and merging are unmanageably difficult in it, where in git, they're many-times-a-day operations. This is a red herring, unless your proposal also includes making the master CVS^H^H^Hgit repository world-writable. The complaint I have about people posting URLs is that there's no stable archive of what the patches really were, and just because it came out of someone's local git repository doesn't help that. The master repository need not be world-writeable, but as many public ones as needed for development should be. I'd love for people to use our infrastructure, but github, etc., would also work. As much as I'm starting to join the "let's move the main repo to git" crowd, all you need for what you're suggesting here is a stable git mirror on git.postgresql.org. /Magnus Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 4:37 PM, Joshua D. Drake wrote: > On Mon, 2009-01-12 at 11:18 -0500, Tom Lane wrote: > >> Basically I think we are up against the same type of project management >> decision we've had several times before: are we willing to slip the >> 8.4 release schedule for however long it will take for hot standby >> and the other replication-related features to be ready? > > I would certainly not like to see 8.4 slip. I would. PostgreSQL is not a commercial application which has to be released on schedule to satisfy shareholders - it's an Open Source project that aims to provide it's users with useful features. We have two extremely useful features here (hot standby and sync replication) which together will make this a killer release for many people - we can delay a month or two as required to polish and get them ready for release, or decide we're willing to wait another 12 - 14 months for them to be available for end users. I'd much rather see them included than deferred (particularly hot standby, parts of which have been awaiting review for months now anyway, through no fault of Simons). -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On 1/12/09, Joshua D. Drake wrote: > > > Basically I think we are up against the same type of project management > > decision we've had several times before: are we willing to slip the > > 8.4 release schedule for however long it will take for hot standby > > and the other replication-related features to be ready? > > > I would certainly not like to see 8.4 slip. 8.4 has already slipped. From my (basically a user's persepective) the thing is ready to drop in right now...it works wonderfully. So, *) Does Simon intend for 8.4 release *) Is the thing up to production standards? *) If so, is a patch ready? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore -1 vs -C and -c
Magnus Hagander writes: > It should be possible to make it compatible with -C by moving the CREATE > DATABASE command to outside of the transaction. I have only had a quick > look at the code wrt how much work this would be. One thing that hit me > quickly: do we support multiple CREATE DATABASE statements in a single > dump file? I think not, but the format seems to allow it? If not, it > should be "fairly simple" to do. We don't, and a single transaction couldn't restore into multiple databases anyway. So in principle there's no reason we shouldn't allow it to do the CREATE DATABASE, switch into the new DB, and then start the transaction. However, one of the properties -1 is supposed to have is that any failure aborts the whole restore; it's not immediately clear how to preserve that with CREATE DATABASE issued separately. Also you need to think about whether this might impact pg_dumpall. > As for -c, the solution would be to issue DROP IF EXISTS statements. Is > there any particular reason why we don't? I think we did that to avoid damaging portability and backwards compatibility of the dump files. The backwards compatibility argument is pretty weak by now, but the "it's not standard SQL" argument still has force. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 5:18 PM, Tom Lane wrote: > Basically I think we are up against the same type of project management > decision we've had several times before: are we willing to slip the > 8.4 release schedule for however long it will take for hot standby > and the other replication-related features to be ready? At this point > I think there can be no question that it will not be a small slip; > in fact I'm not even prepared to guess at how long it will take. What I wouldn't like to see is the replication patches becoming another "Bitmap index on disk" patch. If we release 8.4 and postpone replication to 8.5, we really need a plan to concentrate the efforts of the few people capable of making it happen in the very few months of the 8.5 cycle. -- Guillaume -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 11:33:43AM -0500, Tom Lane wrote: > David Fetter writes: > > Two things to fix this, and several other problems: > > > 1. Remove the messages size limits on -hackers. They serve no > > useful purpose, and they interfere with our development process. > > Agreed, or at least boost it up a good bit more. > > > If -hackers isn't already subscriber-only, now would be the time > > to make it so. > > Not sure how that's relevant? Spam and wackiness. Consider what Dmitry Turin would do with an unlimited ability to send his "specs" to -hackers. > > 2. Start using more git, as many hackers and committers have > > already started to do. This is the kind of situation where CVS > > just plain falls down because branching and merging are > > unmanageably difficult in it, where in git, they're > > many-times-a-day operations. > > This is a red herring, unless your proposal also includes making the > master CVS^H^H^Hgit repository world-writable. The complaint I have > about people posting URLs is that there's no stable archive of what > the patches really were, and just because it came out of someone's > local git repository doesn't help that. The master repository need not be world-writeable, but as many public ones as needed for development should be. I'd love for people to use our infrastructure, but github, etc., would also work. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 11:33 -0500, Tom Lane wrote: > > If -hackers > > isn't already subscriber-only, now would be the time to make it so. > > Not sure how that's relevant? So we don't get spam patches. Joshua D. Drake > > regards, tom lane > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, 2009-01-12 at 11:18 -0500, Tom Lane wrote: > > My point is that what Simon currently has (and so what you tested) is > > different from what is going to be commited (note the "final" in what > > I wrote) and I suspect there will be a certain number of non > > negligible adjustments (see the last discussions between Simon and > > Heikki and I don't think Tom has taken a look at these patches yet). > > The thing that's disturbing me is that (to judge by what I've been > seeing on the mailing list) there's been a steady stream of "non > negligible adjustments" for the past two months. That's good from > the standpoint that problems are getting found and fixed, but it's > not giving me any warm fuzzies about the code being ready to go. This is the same thing that makes me nervous. The feature appears to be "Under heavy development". As I understand the development model the heavy development is supposed to happen before commit fest. > > Basically I think we are up against the same type of project management > decision we've had several times before: are we willing to slip the > 8.4 release schedule for however long it will take for hot standby > and the other replication-related features to be ready? I would certainly not like to see 8.4 slip. > At this point > I think there can be no question that it will not be a small slip; > in fact I'm not even prepared to guess at how long it will take. Not a comforting thought. Sincerely, Joshua D Drake > > regards, tom lane > -- PostgreSQL Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
David Fetter writes: > Two things to fix this, and several other problems: > 1. Remove the messages size limits on -hackers. They serve no useful > purpose, and they interfere with our development process. Agreed, or at least boost it up a good bit more. > If -hackers > isn't already subscriber-only, now would be the time to make it so. Not sure how that's relevant? > 2. Start using more git, as many hackers and committers have already > started to do. This is the kind of situation where CVS just plain > falls down because branching and merging are unmanageably difficult in > it, where in git, they're many-times-a-day operations. This is a red herring, unless your proposal also includes making the master CVS^H^H^Hgit repository world-writable. The complaint I have about people posting URLs is that there's no stable archive of what the patches really were, and just because it came out of someone's local git repository doesn't help that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Assertion failure in plpgsql with INSTEAD OF rule
Heikki Linnakangas writes: > If _SPI_execute_plan reaches the above "if" and sets "my_res = res", > SPI_processed is left at 0. So FOUND is always set to false if the > rewritten command type doesn't match the original. Right, and I think that's probably sane behavior. The original command got rewritten to nothing, therefore it processed zero rows. I'd be happy with code that guaranteed that FOUND got set false in that case. The point I'm making is that the code doesn't guarantee it now (and would not do so after removing those Asserts either). Consider create rule r1 as on insert to foo do instead notify foo_insert; As implemented, this would send us through the SPI_OK_UTILITY path and nothing happens to FOUND. I say that's a bug. So what I'm thinking is: 1. We can't redefine the SPI interface in back branches, so there's probably little alternative but to remove those Asserts there. 2. In HEAD, I think we should have SPI return a new SPI_OK_REWRITTEN code for this case, and have plpgsql respond to that by always setting found = false. With that, the Asserts can stay (in fact the new path should assert !mod_stmt, since it shouldn't have found any canSetTag query). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 11:11:20AM -0500, Greg Stark wrote: > On Mon, Jan 12, 2009 at 9:55 AM, Tom Lane wrote: > > Simon Riggs writes: > >> On Mon, 2009-01-12 at 09:04 -0500, Tom Lane wrote: > > > > Well, one of the things that makes me uncomfortable is that it's > > not even clear exactly which set of patches is currently proposed > > for inclusion. We've seen a whole lot of URLs fly back and forth, > > many of them pointing at pages that aren't there a few days later. > > I've been too busy with non-replication-related patches to pay > > really close attention, but I certainly don't get the impression > > that there's a stable set of patches waiting to be applied. > > See this is one of the things which bothers me. I don't see any > advantage in forcing Simon to stop making improvements -- and there > are always improvements to be made -- just to make his code seem > more stable. Two things to fix this, and several other problems: 1. Remove the messages size limits on -hackers. They serve no useful purpose, and they interfere with our development process. If -hackers isn't already subscriber-only, now would be the time to make it so. 2. Start using more git, as many hackers and committers have already started to do. This is the kind of situation where CVS just plain falls down because branching and merging are unmanageably difficult in it, where in git, they're many-times-a-day operations. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Assertion failure in plpgsql with INSTEAD OF rule
Tom Lane wrote: Heikki Linnakangas writes: The problem is that mod_stmt is determined for the query that has canSetTag set, but in case of an INSTEAD OF rule that rewrites the statement into a different command, an INSERT into a DELETE in this case, canSetTag is not set. The return code of SPI_execute_plan still indicates SPI_OK_DELETE, so we have a mismatch in what that assertion is trying to check. mod_stmt is used to control whether to throw an error if the query returns more than one row and there's an INTO, and ISTM the logic is correct for that use. However, the logic for when to set FOUND is different, so I think the correct fix is to simply remove the assertion. At least for back-branches; you could argue for changing the behavior of FOUND, but that could break existing applications. I think the real problem is this bit at the tail end of _SPI_execute_plan: /* * If none of the queries had canSetTag, we return the last query's result * code, but not its auxiliary results (for backwards compatibility). */ if (my_res == 0) my_res = res; This always seemed a bit dubious to me, and in the light of this example I wonder whether it was even really backwards-compatible. The problem of course is what to return instead --- it almost seems like we'd have to invent a new SPI return code "SPI_OK_REWRITTEN" or some such. In any case, having an INSERT set FOUND on the basis of a rewritten DELETE seems 100% wrong to me. It won't even be self-consistent because the actual value of SPI_processed won't be coming from the DELETE. And what if it's rewritten into a utility statement (eg NOTIFY)? If _SPI_execute_plan reaches the above "if" and sets "my_res = res", SPI_processed is left at 0. So FOUND is always set to false if the rewritten command type doesn't match the original. Note that even when an INSERT is rewritten into another INSERT, there's actually no guarantee that SPI_processed means what the caller would think it means. It could be rewritten into an INSERT of a different table, a log table, for example. I don't much like the idea of leaving FOUND untouched either. If there's an earlier query in the function that sets FOUND, and you have an "IF FOUND ..." block after an INSERT that follows, you might not realize that FOUND is still left at the value set by the earlier query. Hmm, perhaps FOUND should be set to NULL in this case, to mean "unknown". -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery Test Framework
On Mon, Jan 12, 2009 at 11:07 AM, Guillaume Smet wrote: > On Mon, Jan 12, 2009 at 4:56 PM, Merlin Moncure wrote: >> I disagree at least with hot standby. I've been using/testing (as >> have others) it under a variety of workloads for several months now >> with no issues outside of corrected issues in the very early patches. >> Also, a relatively few amount of people update/build from cvs >> frequently so being committed late in the release cycle isn't as >> important as you are claiming...the real 'wider net' testing happens >> when the beta period begins. > > Update/build from CVS != Update/build from CVS + apply the replication > patches + test them explicitely. > > That said, I didn't have the time to test them myself so I feel also > responsible for that. In the general case I think plenty of people update and build from CVS regularly. It's great that the FSM has been in for a couple months before the beta, we've uncovered a couple problems which could easily have slipped through the betas for example. In the case of hot standby and replication I'm not really sure that logic applies. It takes quite a lot of work to test these features and they don't turn up problems in other areas when you're not running them. So I doubt it would really have helped in this case. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
