Re: [HACKERS] Hot standby, recovery procs
Simon Riggs wrote: On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: What benefit would we gain from separating them, especially since we now have working, tested code? Simplicity. That matters a lot. Removing the distinction between unobserved xids and already-observed running transactions would slash a lot of code. It might and it might not, but I don't believe all angles have been evaluated. But I would say that major changes such as this have resulted in weeks of work. More bugs have been introduced since feature freeze than were present beforehand. Here's a rough sketch of how the transaction tracking could work without recovery procs, relying on unobserved xids array only. The unobserved xids is a complete misnomer now, as it tracks all master-transactions, and there's no distinction between observed and unobserved ones. Another big change in this patch is the way xl_xact_assignment records work. Instead of issuing one such WAL record for each subtransaction when they're being assigned recursively, we keep track of which xids have already been reported in the WAL (similar to what you had in an earlier version of the patch). Whenever you hit the limit of 64 unreported subxids, you issue a single WAL record listing all the unreported subxids of this top-level transactions, and mark them as reported. The limit of 64 is chosen arbitrarily, but it should match the number of slots in the unobserved xids array per backend, to avoid running out of slots. This eliminates the need for the xl_topxid field in the WAL record header. I think one WAL record per 64 assigned subtransactions is a small price to pay, considering that a transaction with that many subtransactions is probably doing some interesting work anyway, and the volume of those assignment WAL records is lost in the noise of all the other WAL records the transactions issues. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] effective_cache_size less than shared_buffers
On Thu, Feb 26, 2009 at 1:33 AM, Joshua D. Drake j...@commandprompt.com wrote: On Wed, 2009-02-25 at 17:04 -0800, Josh Berkus wrote: Joshua D. Drake wrote: I would say no. Although I could see an argument for the default effective_cache_size always being the same size as shared_buffers. That's certainly not what we've meant historically by ECS. Since we are already using X amount of shared_buffers we know we have at least X amount of cache. That's not what you wrote, at least how it was understood. It sounds like you're in violent agreement. We can't determine the size of the FS cache. Hence why we have the parameter. We can determine the size of the shared_buffers. The idea here is to eliminate one of those by default PostgreSQL is slow issues. Well we won't eliminate any problems unless we actually override the effective_cache_size setting by clipping it to shared_buffers. I don't really see much of a problem doing that. The only case where that would annoy someone was if they're intentionally understating effective_cache_size to push the planner into avoiding nested loops and I doin't think it's a powerful enough knob to be very likely used that way. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery procs
Simon Riggs wrote: On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: I think if I had not made those into procs you would have said that they are so similar it would aid code readability to have them be the same. And in fact I suggested earlier that we get rid of the unobserved xids array, and only use recovery procs. Last week, I think. Why are these tweaks so important? Heh, actually, I went searching my mail for when I had suggested that, and found that in fact I proposed this exact same method of using the unobserved xids array only back in October: http://archives.postgresql.org/message-id/48f76342.5070...@enterprisedb.com I had since forgotten all about, but now came up with the same idea again during review. In the first reply in that thread you said that The main problem is fatal errors that don't write abort records. By reusing the PROC entries we can keep those to a manageable limit. We're not worried about that anymore. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Thu, Feb 26, 2009 at 7:09 AM, Joshua D. Drake j...@commandprompt.com wrote: I think his reply states that. The long and short is, what Tom was concerned about is true and Heikki has confirmed it. This patch as nice as it would be to have, isn't ready for prime time. It is time to push this patch to 8.5, close up the rest of whatever is left with other patches and move to Beta. The decision on whether it is ready lies with Heikki, and whilst he is prepared to work on it, and there are other patches still in the queue being worked on by other committers there is no reason to bump it yet (unless Heikki wants to look at the other patches). I agree that this patch alone should not delay the release further though. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery procs
On Thu, 2009-02-26 at 10:04 +0200, Heikki Linnakangas wrote: we keep track of which xids have already been reported in the WAL (similar to what you had in an earlier version of the patch) You objected to doing exactly that earlier. Why is it OK to do it now that you are proposing it? You haven't even given a good reason to make these changes. We don't have time to make this change and then shake out everything else that will break as a result. Are you suggesting that you will make these changes and then follow up on all other breakages? Forcing this request seems like a great way to cancel this patch, since it will be marked as author refused to make change. You have spotted a problem elsewhere and I am working to fix that now. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
I haven't been following this thread closely, so pardon if this has been discussed already. The patch doesn't seem to change the cost estimates in the planner at all. Without that, I'd imagine that the planner rarely chooses a multi-batch hash join to begin with. Joshua, in the tests that you've been running, did you have to rig the planner with enable_mergjoin=off or similar, to get the queries to use hash joins? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication Hot standby patches
On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote: Again, I'm not planning to get rid of any existing capabilities Good unless necessary. That is not a caveat I will accept, a priori. What does accept mean above? Are you the sole acceptor for this feature? That is surprising to me. You can say you would vote against it but your wording above seems overly controlling. I would hope my words carry the same weight as others when people speak of what can and cannot be included, when backed by reasonable and logical technical reasons. Some things are important, some not, and I've done my best to use words that indicate my views on that. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery procs
Simon Riggs wrote: On Thu, 2009-02-26 at 10:04 +0200, Heikki Linnakangas wrote: we keep track of which xids have already been reported in the WAL (similar to what you had in an earlier version of the patch) You objected to doing exactly that earlier. I'm talking about the xidMarkedInWAL and hasUnMarkedSubXids fields you had in TransactionState, at least still in version hs.v7.20090112_1.tar.bz2 of the patch. I objected to adding the corresponding flags in the WAL header, and that made tracking the status in TransactionState obsolete in the patch too, since it wasn't used for anything anymore. There's nothing wrong per se about tracking the marked or reported status in master. You haven't even given a good reason to make these changes. Simplicity. We don't have time to make this change and then shake out everything else that will break as a result. Are you suggesting that you will make these changes and then follow up on all other breakages? Forcing this request seems like a great way to cancel this patch, since it will be marked as author refused to make change. I'm not suggesting anything to be canceled. I simply think these are changes that should be made. I wish you could make them, because that means less work for me. But if you're not willing to, I can pick it up myself. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] effective_cache_size less than shared_buffers
Greg, Well we won't eliminate any problems unless we actually override the effective_cache_size setting by clipping it to shared_buffers. I don't really see much of a problem doing that. The only case where that would annoy someone was if they're intentionally understating effective_cache_size to push the planner into avoiding nested loops and I doin't think it's a powerful enough knob to be very likely used that way. My experience from PostgreSQL on Windows: effective_cache_size should reflect the value of system cache from task manager. shared_buffers (on windows) should be rather small. My real-workload-tests (no benchmarks, real usage of DB-Server) showed that big shared buffers on Windows have a negative effect on PostgreSQL performance. I have found no explanation WHY it is this way. Harald -- GHUM Harald Massa persuadere et programmare Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 no fx, no carrier pigeon - EuroPython 2009 will take place in Birmingham - Stay tuned!
Re: [HACKERS] Hot standby, recovery procs
On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should be reduced and now the same argument to put it back again. We don't have time to make this change and then shake out everything else that will break as a result. Are you suggesting that you will make these changes and then follow up on all other breakages? Forcing this request seems like a great way to cancel this patch, since it will be marked as author refused to make change. I'm not suggesting anything to be canceled. I simply think these are changes that should be made. I wish you could make them, because that means less work for me. But if you're not willing to, I can pick it up myself. When you review my code, you make many useful suggestions and I am very thankful. Testing can't find out some of those things. My feeling is that you are now concentrating on things that are optional, yet will have a huge potential for negative impact. If I could please draw your review efforts to other parts of the patch, I would be happy to return to these parts later. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery procs
Simon Riggs wrote: On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should be reduced and now the same argument to put it back again. That was in reference to the slot ids, I'm not suggesting to put that back. If anything, removing the need for the the xl_topxid field in WAL record will further reduce the coupling between master and standby. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery procs
On Thu, 2009-02-26 at 12:19 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should be reduced and now the same argument to put it back again. That was in reference to the slot ids, I'm not suggesting to put that back. If anything, removing the need for the the xl_topxid field in WAL record will further reduce the coupling between master and standby. OK, well, if you feel those changes are necessary prior to commit then I would ask you do that in your public repo and we'll test and provide helpful comments on it from there as quickly as we can. Too many cooks spoil the git. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
On Thu, Feb 26, 2009 at 4:22 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I haven't been following this thread closely, so pardon if this has been discussed already. The patch doesn't seem to change the cost estimates in the planner at all. Without that, I'd imagine that the planner rarely chooses a multi-batch hash join to begin with. AFAICS, a multi-batch hash join happens when you are joining two big, unsorted paths. The planner essentially compares the cost of sorting the two paths and then merge-joining them versus the cost of a hash join. It doesn't seem to be unusual for the hash join to come out the winner, although admittedly I haven't played with it a ton. You certainly could try to model it in the costing algorithm, but I'm not sure how much benefit you'd get out of it: if you're doing this a lot you're probably better off creating indices. Joshua, in the tests that you've been running, did you have to rig the planner with enable_mergjoin=off or similar, to get the queries to use hash joins? I didn't have to fiddle anything, but Josh's tests were more exhaustive. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: For fear of passing an ill formed fragment of xml to the processor, we strip the xml declaration if any and surround what's left with 'x and '/x' and prepend '/x' to the supposed xpath. This is just horrible. I seem to recall having complained about that at the time, but I didn't (and don't) know enough about xpath to do any better. Well, a few of us do. I guess I took my eye off the ball a bit back when we were putting this into 8.3. This whole thing is a mess, and I suspect the only fix for now is to undo all the mangling of both the xml and the xpath expression. I don't think we should change the behavior if it's just to arrive at another less-than-desirable behavior. Whacking semantics around afresh with each release does not endear us to users. If we know how to fix it right, great; but if we can't then we should keep compatibility with 8.3 until we can. Honestly, this is a bug, pure and simple. There really can't be an argument about that. For the stable branch, we could make the following changes that should result in a Pareto improvement (nothing gets worse while some things get better): * only do the xml transformation if the xml is known not to be be well formed * if we need to mangle the xpath expression due to doing the xml transformation, then unless the xpath expression begins with a '/', prepend it with '/x//'. (two slashes corresponds to the descendent axis in xpath - in effect it stands for any number of descendent elements). But that's just a holding operation. For 8.4 we should stop this nonsense and simply say that it is up to the programmer to ensure that the xml passed to the processor is well formed. The thing that is so very bad about this is that if the programmer *has* made sure that the inputs are correct, s/he can still end up with broken results. If we're going to try to fix bad inputs, we must make damn sure that we don't break on correct inputs as a result. However, I can't off hand think of a lightning way to fix bad inputs that doesn't carry some danger to good inputs. Until someone comes up with something tolerably bulletproof, I suggest that we simply say that it is the programmer's responsibility. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
On Wed, Feb 25, 2009 at 10:24:21PM -0500, Robert Haas wrote: I don't think we're really doing this the right way. EXPLAIN ANALYZE has a measurable effect on the results, and we probably ought to stop the database and drop the VM caches after each query. Are the Z1-Z7 datasets on line someplace? I might be able to rig up a script here. ...Robert They're automatically generated by the dbgen utility, a link to which was originally published somewhere in this thread. That tool creates a few text files suitable (with some tweaking) for a COPY command. I've got the original files... the .tbz I just made is 1.8 GB :) Anyone have someplace they'd like me to drop it? - Josh signature.asc Description: Digital signature
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Thu, Feb 26, 2009 at 3:47 AM, Dave Page dp...@pgadmin.org wrote: I think his reply states that. The long and short is, what Tom was concerned about is true and Heikki has confirmed it. This patch as nice as it would be to have, isn't ready for prime time. It is time to push this patch to 8.5, close up the rest of whatever is left with other patches and move to Beta. The decision on whether it is ready lies with Heikki, and whilst he is prepared to work on it, and there are other patches still in the queue being worked on by other committers there is no reason to bump it yet (unless Heikki wants to look at the other patches). Well, this is a key point. If Heikki thinks that the patch isn't going to be ready for 8.4 (and it sounds like that's the case), then it would be more beneficial to the project for him to lay it aside for a few weeks and help clean out the rest of the patch queue rather than continuing to work on that patch and leaving the other patches to the other committers. Our rate of progress on cleaning out the November CommitFest seems to be asymptotically approaching zero (9 patches committed in December, 6 in January, 4 so far in February...) so benching one of the main committers for a feature that won't be ready anyway is not a good trade-off. Of course, if Heikki can't or isn't interested in working on any of the remaining patches, then he might as well keep plugging away at Hot Standby rather than leaving it for another day. It can live in git until 8.4 is released and then get merged after the tree is branched, and anyone else who wants to do further development in that area can get started knowing what the final shape of that code will be. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
On Thu, Feb 26, 2009 at 08:22:52AM -0500, Robert Haas wrote: On Thu, Feb 26, 2009 at 4:22 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Joshua, in the tests that you've been running, did you have to rig the planner with enable_mergjoin=off or similar, to get the queries to use hash joins? I didn't have to fiddle anything, but Josh's tests were more exhaustive. The planner chose hash joins for the queries I was running, regardless of whether the patch was applied. I didn't have to mess with any settings to get hash joins. - Josh signature.asc Description: Digital signature
Re: [HACKERS] Have \d show child tables that inherit from the specified parent
On Wed, Feb 25, 2009 at 07:30:30PM -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: Should the patch (and the feature) use WITH RECURSIVE in order to get the entire tree? See the note at the top of that file that all queries are expected to work with server versions back to 7.4. Would it be OK to have a different code path for pre-8.4 databases? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Have \d show child tables that inherit from the specified parent
David Fetter da...@fetter.org writes: On Wed, Feb 25, 2009 at 07:30:30PM -0500, Tom Lane wrote: See the note at the top of that file that all queries are expected to work with server versions back to 7.4. Would it be OK to have a different code path for pre-8.4 databases? I don't really see the point here. describe.c exists to get the job done, not to show off fancy features. BTW, has anyone thought about what's gonna happen if the table has got thousands of inheritance children? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
Heikki's got a point here: the planner is aware that hashjoin doesn't like skewed distributions, and it assigns extra cost accordingly if it can determine that the join key is skewed. (See the bucketsize stuff in cost_hashjoin.) If this patch is accepted we'll want to tweak that code. Still, that has little to do with the current gating issue, which is whether we've convinced ourselves that the patch doesn't cause a performance decrease for cases in which it's unable to help. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication Hot standby patches
Simon Riggs wrote: On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote: Again, I'm not planning to get rid of any existing capabilities Good unless necessary. That is not a caveat I will accept, a priori. What does accept mean above? Are you the sole acceptor for this feature? That is surprising to me. You can say you would vote against it but your wording above seems overly controlling. I would hope my words carry the same weight as others when people speak of what can and cannot be included, when backed by reasonable and logical technical reasons. Some things are important, some not, and I've done my best to use words that indicate my views on that. I think you need to work on improving your word choice then. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Robert Haas wrote: On Thu, Feb 26, 2009 at 3:47 AM, Dave Page dp...@pgadmin.org wrote: I think his reply states that. The long and short is, what Tom was concerned about is true and Heikki has confirmed it. This patch as nice as it would be to have, isn't ready for prime time. It is time to push this patch to 8.5, close up the rest of whatever is left with other patches and move to Beta. The decision on whether it is ready lies with Heikki, and whilst he is prepared to work on it, and there are other patches still in the queue being worked on by other committers there is no reason to bump it yet (unless Heikki wants to look at the other patches). Well, this is a key point. If Heikki thinks that the patch isn't going to be ready for 8.4 (and it sounds like that's the case), then it would be more beneficial to the project for him to lay it aside for a few weeks and help clean out the rest of the patch queue rather than continuing to work on that patch and leaving the other patches to the other committers. Our rate of progress on cleaning out the November CommitFest seems to be asymptotically approaching zero (9 patches committed in December, 6 in January, 4 so far in February...) so benching one of the main committers for a feature that won't be ready anyway is not a good trade-off. Agreed, we could use Heikki's help in closing the other outstanding patches, and if hot standby isn't going to make 8.4, which I agree with, it would be good for him to put hot standby aside and help on other patches. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GIN fast insert
Teodor, can you confirm * we WAL log the insert into the pending list * we WAL log the move from the pending list to the main index Yes, I can and I confirm * that we maintain the pending list correctly during redo so that it can be accessed by index scans Not sure about correct locking in ginxlog.c - but it should fixable rather easy. The main thing with Hot Standby is that we can't do any writes. So a pending list cannot change solely because of a gingettuple call on the *standby*. But that's easy to disable. If all the inserts happened on Right. And suggest to increase work_mem. the primary node and all the reads happened on the standby, then pending list would never be cleaned up if the cleanup is triggered only by read. I would suggest that we trigger cleanup by read at threshold size X and trigger cleanup by insert at threshold size 5X. That avoids the strange threshold is not helpful here because for gininsert and gingettuple it depends on work_mem setting which could be very different on slave(s) and master. Next, it's possible that master never executes read-only queries (i.e. gingettuple will not be called) by application's design. This is a strong objection for invocation of cleanup from gingettuple. So, I'm inclined to Tom's idea about removing gingettuple interface although GIN with fastupdate=off doesn't affect discussed problem . I found many parts of the patch and docs quite confusing because of the way things are named. For me, this is a deferred or delayed insert technique to allow batching. I would prefer if everything used one description, rather than fast, pending, delayed etc. Terminology was taken from inverted index technique. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication Hot standby patches
Bruce Momjian br...@momjian.us wrote: Simon Riggs wrote: On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote: Again, I'm not planning to get rid of any existing capabilities Good unless necessary. That is not a caveat I will accept, a priori. What does accept mean above? Are you the sole acceptor for this feature? That is surprising to me. You can say you would vote against it but your wording above seems overly controlling. I would hope my words carry the same weight as others when people speak of what can and cannot be included, when backed by reasonable and logical technical reasons. Some things are important, some not, and I've done my best to use words that indicate my views on that. I think you need to work on improving your word choice then. Anything which breaks existing techniques for log shipping, PITR recovery, or warm standby would present a significant obstacle to adoption in our environment. Based on off-list emails received when I mentioned what we do, I know we're not alone. Some comments have made me nervous on this count, but responses to questions about it have generally been reassuring. The unless necessary seemed a little ominous. I won't get too excited unless someone thinks a change is necessary and I see the nature of the change. As long as there is some way to ship base backups and WAL files to multiple targets, not all of which are running any PostgreSQL software, and the database can be restored from such copies, we're not looking a more than a few weeks of staff time to adjust our scripts and procedures. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1608)
KaiGai Kohei wrote: The series of SE-PostgreSQL patches for v8.4 were updated: [1/5] http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch [2/5] http://sepgsql.googlecode.com/files/sepgsql-utils-8.4devel-r1608.patch [3/5] http://sepgsql.googlecode.com/files/sepgsql-policy-8.4devel-r1608.patch [4/5] http://sepgsql.googlecode.com/files/sepgsql-docs-8.4devel-r1608.patch [5/5] http://sepgsql.googlecode.com/files/sepgsql-tests-8.4devel-r1608.patch - List of updates: * bugfix: sepgsqlCheckProcedureEntrypoint() was invoked twice when security invoker functions are invoked. Rest of parts are unchanged. Don't mind contracted filename. Please comment anything. It will help to improve our code. I did an analysis of the core file: http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch changed lines 3226 new files 4075 syscatalog 9977 total 17278 The good news is that 3226 is the affect on the non-system-catalog main core code, and is a context diff size, not total changed lines. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Andrew Dunstan wrote: For fear of passing an ill formed fragment of xml to the processor, we strip the xml declaration if any and surround what's left with 'x and '/x' and prepend '/x' to the supposed xpath. This is just horrible. It will break for every xpath expression that doesn't begin with a '/' and probably for many that do. This whole thing is a mess, and I suspect the only fix for now is to undo all the mangling of both the xml and the xpath expression. If the programmer passes an ill formed piece of xml to the processor that is their lookout, but I think we should ensure that we give back correct results on well formed input. It's not about ill-formed pieces, it is about (well-formed) content fragments that are not full documents (exactly one root element). libxml2 doesn't support xpath on content fragments. The tradeoff here is either supporting no xpath at all on content fragments or supporting some xpath on both kinds of xml data. Whoever made the initial implementation of this (Nikolay, Cc) decided for the latter, but I can't say I'm happy with it either. I'd be OK with changing it, but I haven't had time to get to it, unfortunately. See also http://wiki.postgresql.org/wiki/XPath_Todo#XPath. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GIN fast insert
it be? So far we've ruled out using the planner to prevent index scans when the pending list is long (because it's not reliable) and cleaning up the pending list during insert when needed (because it won't work with Hot Standby). We haven't decided what WILL work, During insert it will work with Hot Standby because there is no any limitation for number of pages touched or WAL records. There is a problem with cleanup invoked by gingettuple - slave could not start cleanup process at all and hence it could emit an error if tidbitmap becomes lossy. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GIN fast insert
On Thu, Feb 26, 2009 at 11:41 AM, Teodor Sigaev teo...@sigaev.ru wrote: it be? So far we've ruled out using the planner to prevent index scans when the pending list is long (because it's not reliable) and cleaning up the pending list during insert when needed (because it won't work with Hot Standby). We haven't decided what WILL work, During insert it will work with Hot Standby because there is no any limitation for number of pages touched or WAL records. There is a problem with cleanup invoked by gingettuple - slave could not start cleanup process at all and hence it could emit an error if tidbitmap becomes lossy. I didn't think about that option - might be reasonable. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication Hot standby patches
On Thu, 2009-02-26 at 11:02 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote: Again, I'm not planning to get rid of any existing capabilities Good unless necessary. That is not a caveat I will accept, a priori. What does accept mean above? Are you the sole acceptor for this feature? That is surprising to me. You can say you would vote against it but your wording above seems overly controlling. I would hope my words carry the same weight as others when people speak of what can and cannot be included, when backed by reasonable and logical technical reasons. Some things are important, some not, and I've done my best to use words that indicate my views on that. I think you need to work on improving your word choice then. I vote against this proposal. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
From: Tom Lane Heikki's got a point here: the planner is aware that hashjoin doesn't like skewed distributions, and it assigns extra cost accordingly if it can determine that the join key is skewed. (See the bucketsize stuff in cost_hashjoin.) If this patch is accepted we'll want to tweak that code. Those modifications would make the optimizer more likely to select hash join, even with skewed distributions. For the TPC-H data set that we are using the optimizer always picks hash join over merge join (single or multi-batch). Since the current patch does not change the cost function, there is no change in the planning cost. It may or may not be useful to modify the cost function depending on the effect on planning cost. Still, that has little to do with the current gating issue, which is whether we've convinced ourselves that the patch doesn't cause a performance decrease for cases in which it's unable to help. Although we have not seen an overhead when the optimization is by-passed, we are looking at some small code changes that would guarantee that no extra statements are executed for the single batch case. Currently, an if optimization_on check is performed on each probe tuple which, although minor, should be able to be avoided. The patch's author, Bryce Cutt, is defending his Master's thesis Friday morning (on this work), so we will provide some updated code right after that. Since these code changes are small, they should not affect people trying to test the performance of the current patch. -- Ramon Lawrence -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance ofMulti-BatchHash Join for Skewed Data Sets
They're automatically generated by the dbgen utility, a link to which was originally published somewhere in this thread. That tool creates a few text files suitable (with some tweaking) for a COPY command. I've got the original files... the .tbz I just made is 1.8 GB :) Anyone have someplace they'd like me to drop it? Just a note that the Z7 data set is really a uniform data set Z0. The generator only accepts skew in the range from Z0 to Z4. The uniform, Z0, data set is typically used when benchmarking data warehouses. It turns out the data is not perfectly uniform as the top 100 suppliers and products represent 2.3% and 1.5% of LineItem. This is just enough skew that the optimization will sometimes be triggered in the multi-batch case (currently 1% skew is the cutoff). I have posted a pg_dump of the TPCH 1G Z0 data set at: http://people.ok.ubc.ca/rlawrenc/tpch1g0z.zip (Note that ownership commands are in the dump and make sure to vacuum analyze after the load.) I can also post the input text files if that is easier. -- Ramon Lawrence -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Peter Eisentraut wrote: Andrew Dunstan wrote: For fear of passing an ill formed fragment of xml to the processor, we strip the xml declaration if any and surround what's left with 'x and '/x' and prepend '/x' to the supposed xpath. This is just horrible. It will break for every xpath expression that doesn't begin with a '/' and probably for many that do. This whole thing is a mess, and I suspect the only fix for now is to undo all the mangling of both the xml and the xpath expression. If the programmer passes an ill formed piece of xml to the processor that is their lookout, but I think we should ensure that we give back correct results on well formed input. It's not about ill-formed pieces, it is about (well-formed) content fragments that are not full documents (exactly one root element). libxml2 doesn't support xpath on content fragments. The tradeoff here is either supporting no xpath at all on content fragments or supporting some xpath on both kinds of xml data. Whoever made the initial implementation of this (Nikolay, Cc) decided for the latter, but I can't say I'm happy with it either. I'd be OK with changing it, but I haven't had time to get to it, unfortunately. See also http://wiki.postgresql.org/wiki/XPath_Todo#XPath. I don't think it is our responsibility to remedy the deficiencies of libxml2, especially if it involves breaking the processing of valid xpath expressions on well formed XML. If someone can come up with a correct scheme for handling fragments, I'm all ears, but otherwise I think a) we should rip this out of 8.4 and b) we should try to make 8.3 slightly less broken at least, along the lines of my earlier suggestion. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
Hi, On Fri, Jan 30, 2009 at 7:47 PM, Simon Riggs si...@2ndquadrant.com wrote: That whole area was something I was leaving until last, since immediate shutdown doesn't work either, even in HEAD. (Fujii-san and I discussed this before Christmas, briefly). This problem remains in current HEAD. I mean, immediate shutdown may be unable to kill the startup process because system() which executes restore_command ignores SIGQUIT while waiting. When I tried immediate shutdown during recovery, only the startup process survived. This is undesirable behavior, I think. The following code should be added into RestoreArchivedFile()? if (WTERMSIG(rc) == SIGQUIT) exit(2); Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Have \d show child tables that inherit from the specified parent
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 This one is very basic, it just shows the child tables of a specific table when you type \d tablename in psql : I'm not so jazzed about this, as I work on systems that have literally hundreds of child tables. When I do a \d on the master table, that's not what I want to see. Frankly, a good naming system goes a long way towards that, as children usually line up quick neatly in \d output. Can we make this only showup in \d+ perhaps? Or not do this at all? Or for \d, make it say Use \d+ to view all children? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation PGP Key: 0x14964AC8 200902261222 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkmm0AEACgkQvJuQZxSWSsjKSgCgrWjNIewEWmhZbIn1qO8fUQNf U4sAn2+AYLnCtq02e8s9IVA55EmgRlk3 =Y9Pi -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance ofMulti-BatchHash Join for Skewed Data Sets
I have posted a pg_dump of the TPCH 1G Z0 data set at: http://people.ok.ubc.ca/rlawrenc/tpch1g0z.zip That seems VERY useful - can you post the other ones (Z1, etc.) so I can download them all? Thanks, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance ofMulti-BatchHash Join for Skewed Data Sets
That seems VERY useful - can you post the other ones (Z1, etc.) so I can download them all? The Z1 data set is posted at: http://people.ok.ubc.ca/rlawrenc/tpch1g1z.zip I have not generated Z2, Z3, Z4 for 1G, but I can generate the Z2 and Z3 data sets, and in a hour or two they will be at: http://people.ok.ubc.ca/rlawrenc/tpch1g2z.zip http://people.ok.ubc.ca/rlawrenc/tpch1g3z.zip Note that Z3 and Z4 are not really useful as the skew is extreme (98% of the probe relation covered by top 100 values). Using the Z2/Z3 data set should be enough to show the huge win if you do *really* have a skewed data set. BTW, is there any particular form/options of the pg_dump command that I should use to make the dump? -- Ramon Lawrence -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Have \d show child tables that inherit from the specified parent
On Thu, Feb 26, 2009 at 12:23 PM, Greg Sabino Mullane g...@turnstep.com wrote: This one is very basic, it just shows the child tables of a specific table when you type \d tablename in psql : I'm not so jazzed about this, as I work on systems that have literally hundreds of child tables. When I do a \d on the master table, that's not what I want to see. Frankly, a good naming system goes a long way towards that, as children usually line up quick neatly in \d output. Can we make this only showup in \d+ perhaps? Or not do this at all? Or for \d, make it say Use \d+ to view all children? Yeah, there is already an awful lot of output from regular old \d Typically, I get indexes, check constraints, foreign key constraints, and triggers. It's not real easy to read. It would be nice if there were a way to suppress this output or only show the part you're interested in at the moment, but it's hard to think of what syntax might be reasonable. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Andrew Dunstan and...@dunslane.net writes: I don't think it is our responsibility to remedy the deficiencies of libxml2, especially if it involves breaking the processing of valid xpath expressions on well formed XML. If someone can come up with a correct scheme for handling fragments, I'm all ears, but otherwise I think a) we should rip this out of 8.4 and b) we should try to make 8.3 slightly less broken at least, along the lines of my earlier suggestion. I'm okay with removing this for 8.4, but I think it's too late to change the behavior of 8.3. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: I don't think it is our responsibility to remedy the deficiencies of libxml2, especially if it involves breaking the processing of valid xpath expressions on well formed XML. If someone can come up with a correct scheme for handling fragments, I'm all ears, but otherwise I think a) we should rip this out of 8.4 and b) we should try to make 8.3 slightly less broken at least, along the lines of my earlier suggestion. I'm okay with removing this for 8.4, but I think it's too late to change the behavior of 8.3. It's never too late to fix a bug. The current behaviour is just plain nonsense ... if your xpath expression is 'foo/bar' it ends up searching for '/xfoo/bar' which can't possibly be right. Surely we can at least fix that. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I'm okay with removing this for 8.4, but I think it's too late to change the behavior of 8.3. It's never too late to fix a bug. When the proposed fix involves breaking cases that used to behave usefully, you need a much stronger argument than that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I'm okay with removing this for 8.4, but I think it's too late to change the behavior of 8.3. It's never too late to fix a bug. When the proposed fix involves breaking cases that used to behave usefully, you need a much stronger argument than that. What I have proposed for 8.3 should not break a single case that currently behaves usefully. If anyone has a counter-example please show it. What I have proposed for 8.4 possibly would break current useful behaviour (FSVO useful), but should be done anyway on correctness grounds. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
Fujii Masao wrote: On Fri, Jan 30, 2009 at 7:47 PM, Simon Riggs si...@2ndquadrant.com wrote: That whole area was something I was leaving until last, since immediate shutdown doesn't work either, even in HEAD. (Fujii-san and I discussed this before Christmas, briefly). This problem remains in current HEAD. I mean, immediate shutdown may be unable to kill the startup process because system() which executes restore_command ignores SIGQUIT while waiting. When I tried immediate shutdown during recovery, only the startup process survived. This is undesirable behavior, I think. Yeah, we need to fix that. The following code should be added into RestoreArchivedFile()? if (WTERMSIG(rc) == SIGQUIT) exit(2); I don't see how that helps, as we already have this in there: signaled = WIFSIGNALED(rc) || WEXITSTATUS(rc) 125; ereport(signaled ? FATAL : DEBUG2, (errmsg(could not restore file \%s\ from archive: return code %d, xlogfname, rc))); which means we already ereport(FATAL) if the restore command dies with SIGQUIT. I think the real problem here is that pg_standby traps SIGQUIT. The startup process doesn't receive the SIGQUIT because it's in system(), and pg_standby doesn't propagate it to the startup process either because it traps it. I think we should simply remove the signal handler for SIGQUIT from pg_standby. Or will that lead to core dump by default? In that case, we need pg_standby to exit(128) or similar, so that RestoreArchivedFile understands that the command was killed by a signal. Another approach is to check that the postmaster is still alive, like we do in walwriter and bgwriter: /* * Emergency bailout if postmaster has died. This is to avoid the * necessity for manual cleanup of all postmaster children. */ if (!PostmasterIsAlive(true)) exit(1); However, I'm afraid there's a race condition with that. If we do that right after system(), postmaster might've signaled us but not exited yet. We could check that in the main loop, but if we wrongly interpret the exit of the recovery command as a file not found - go ahead and start up, the damage might be done by the time we notice that the postmaster is gone. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
What I have proposed for 8.3 should not break a single case that currently behaves usefully. If anyone has a counter-example please show it. What I have proposed for 8.4 possibly would break current useful behaviour (FSVO useful), but should be done anyway on correctness grounds. I dunno, aren't XML document fragments sort of a pretty common case? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1608)
Bruce Momjian wrote: KaiGai Kohei wrote: The series of SE-PostgreSQL patches for v8.4 were updated: [1/5] http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch [2/5] http://sepgsql.googlecode.com/files/sepgsql-utils-8.4devel-r1608.patch [3/5] http://sepgsql.googlecode.com/files/sepgsql-policy-8.4devel-r1608.patch [4/5] http://sepgsql.googlecode.com/files/sepgsql-docs-8.4devel-r1608.patch [5/5] http://sepgsql.googlecode.com/files/sepgsql-tests-8.4devel-r1608.patch - List of updates: * bugfix: sepgsqlCheckProcedureEntrypoint() was invoked twice when security invoker functions are invoked. Rest of parts are unchanged. Don't mind contracted filename. Please comment anything. It will help to improve our code. I did an analysis of the core file: http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch changed lines 3226 new files 4075 syscatalog 9977 total 17278 The good news is that 3226 is the affect on the non-system-catalog main core code, and is a context diff size, not total changed lines. Hum...? What utility did you use to compute the lines? It seems to me the changed lines except for system catalogs are larger than actual one. The diffstat says: 65 files changed, 4769 insertions(+), 11 deletions(-), 4945 modifications(!) The (4244 + 500) of 4945 modifications come from pg_proc.h and pg_attribute.h due to a new field to store security label of procedures and columns. The new files adds 4014 in total, so rest of (755 + 11 + 201 = 967) lines are estimated changes in the main core code. Anyway, I believe the burden of reviewer became smaller than the prior full-set version. Thanks, - [kai...@masu ~]$ diffstat ~/sepgsql-core-8.4devel-r1608.patch configure | 113 configure.in | 13 src/Makefile.global.in|1 src/backend/Makefile |7 src/backend/access/heap/heapam.c | 12 src/backend/bootstrap/bootparse.y |4 src/backend/bootstrap/bootstrap.c |3 src/backend/catalog/aclchk.c | 11 src/backend/catalog/heap.c| 94 src/backend/catalog/index.c |8 src/backend/catalog/pg_aggregate.c|3 src/backend/catalog/pg_proc.c |9 src/backend/catalog/toasting.c|3 src/backend/commands/cluster.c|4 src/backend/commands/copy.c |9 src/backend/commands/dbcommands.c | 33 src/backend/commands/foreigncmds.c|7 src/backend/commands/functioncmds.c | 77 src/backend/commands/lockcmds.c |4 src/backend/commands/proclang.c |6 src/backend/commands/tablecmds.c | 99 src/backend/commands/trigger.c|6 src/backend/executor/execMain.c | 22 src/backend/nodes/copyfuncs.c | 25 src/backend/nodes/equalfuncs.c| 21 src/backend/nodes/outfuncs.c | 28 src/backend/nodes/readfuncs.c | 41 src/backend/optimizer/plan/planner.c |1 src/backend/parser/gram.y | 63 src/backend/postmaster/postmaster.c | 43 src/backend/rewrite/rewriteHandler.c |6 src/backend/security/Makefile | 11 src/backend/security/sepgsql/Makefile | 16 src/backend/security/sepgsql/avc.c| 1157 +++ src/backend/security/sepgsql/checker.c| 902 + src/backend/security/sepgsql/core.c | 235 + src/backend/security/sepgsql/dummy.c | 37 src/backend/security/sepgsql/hooks.c | 576 +++ src/backend/security/sepgsql/label.c | 360 ++ src/backend/security/sepgsql/perms.c | 463 ++ src/backend/storage/ipc/ipci.c|2 src/backend/tcop/utility.c|5 src/backend/utils/cache/catcache.c| 32 src/backend/utils/cache/syscache.c| 15 src/backend/utils/fmgr/dfmgr.c| 10 src/backend/utils/fmgr/fmgr.c |8 src/backend/utils/init/postinit.c | 11 src/backend/utils/misc/guc.c | 18 src/backend/utils/misc/postgresql.conf.sample |3 src/include/catalog/heap.h|9 src/include/catalog/pg_attribute.h| 500 !!! src/include/catalog/pg_class.h| 12 src/include/catalog/pg_database.h |6 src/include/catalog/pg_proc.h | 4244 !! src/include/catalog/pg_proc_fn.h |3 src/include/fmgr.h| 10 src/include/nodes/nodes.h
[HACKERS] a proposal for an extendable deparser
While writing a shared-library extension for Postgres, I needed to output
SQL expressions from trees. The only facility for doing that seems to be the
deparse code in ruleutils.c, which is really intended for outputing rules
and constraints, not for producing general SQL, so it didn't do quite what I
wanted. I ended up having to copy the entire ruleutils.c file and making a
few minor changes deep in the file. Of course, copy-and-paste is not a very
maintainable form of code reuse so I'm not too happy with this solution.
What I would like is a generic pretty-printing mechanism that can be
tailored for specific needs. I'm willing to do this work if I think it's
likely to be accepted into the main code line.
Here is the proposal: get_rule_expr() consists of a switch statement that
looks like this:
switch (nodeTag(node))
{
case T_Var:
(void) get_variable((Var *) node, 0, true, context);
break;
case T_Const:
get_const_expr((Const *) node, context, 0);
break;
case T_Param:
appendStringInfo(buf, $%d, ((Param *) node)-paramid);
break;
case ...
I would replace this with a table-driven deparser:
deparse_table[nodeTag(node)](node, context);
where deparse_table[] is an array of function pointers containing functions
similar to get_variable(), get_const_expr(), etc. The functions would have
to be adapted to a consistent signature using a more generic context object.
To create a modified deparser, you just copy deparse_table[] and replace
some of its members with your own specialized replacements.
The above description is a bit over-simplified. I would probably end up
making deparse_table[] into a struct with various flags and user-defined
data in addition to the table of function pointers. Also, it might have more
than one table of function pointers. I think a table of RTE types would be
useful, for example, and maybe a table of operators. It would support pretty
printing entire queries, not just rules, constraints, and fragments.
I'd lke to get some feedback on this from the Postgres developers. Is there
any interest in this kind of facility? Would it be likely to be accepted?
Re: [HACKERS] Hot standby, recovery infra
Hi, On Fri, Feb 27, 2009 at 3:38 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think the real problem here is that pg_standby traps SIGQUIT. The startup process doesn't receive the SIGQUIT because it's in system(), and pg_standby doesn't propagate it to the startup process either because it traps it. Yes, you are right. I think we should simply remove the signal handler for SIGQUIT from pg_standby. +1 I don't see how that helps, as we already have this in there: signaled = WIFSIGNALED(rc) || WEXITSTATUS(rc) 125; ereport(signaled ? FATAL : DEBUG2, (errmsg(could not restore file \%s\ from archive: return code %d, xlogfname, rc))); which means we already ereport(FATAL) if the restore command dies with SIGQUIT. SIGQUIT should kill the process immediately, so I think that the startup process as well as other auxiliary process should call exit(2) instead of ereport(FATAL). Right? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
On Thu, 2009-02-26 at 20:38 +0200, Heikki Linnakangas wrote: I think we should simply remove the signal handler for SIGQUIT from pg_standby. If you do this, please make it release dependent so pg_standby behaves correctly for the release it is being used with. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] a proposal for an extendable deparser
Dave Gudeman dave.gude...@gmail.com writes: I would replace this with a table-driven deparser: deparse_table[nodeTag(node)](node, context); I don't actually see what this is going to buy for you. You didn't say exactly why ruleutils doesn't work for you, but reading between the lines suggests that you want to add new node types. There are a *ton* of places that need to change for that, typically, and this isn't going to fix it. I've occasionally speculated about the possible value of switching over to method-table-based node types (or maybe just biting the bullet and going to C++) but it never really looked like it would be worth the trouble. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
Simon Riggs wrote: On Thu, 2009-02-26 at 20:38 +0200, Heikki Linnakangas wrote: I think we should simply remove the signal handler for SIGQUIT from pg_standby. If you do this, please make it release dependent so pg_standby behaves correctly for the release it is being used with. Hmm, I don't think there's a way for pg_standby to know which version of PostgreSQL is calling it. Assuming there is, how would you want it to behave? If you want no change in behavior in old releases, can't we just leave it unfixed in back-branches? In fact, it seems more useful to not detect the server version, so that if you do want the new behavior, you can use a 8.4 pg_standby against a 8.3 server. In back-branches, I think we need to decide between fixing this, at the risk of breaking someone's script that is using killall -QUIT pg_standby or similar to trigger failover, and leaving it as it is knowing that immediate shutdown doesn't work on a standby server. I'm not sure which is best. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Robert Haas wrote: What I have proposed for 8.3 should not break a single case that currently behaves usefully. If anyone has a counter-example please show it. What I have proposed for 8.4 possibly would break current useful behaviour (FSVO useful), but should be done anyway on correctness grounds. I dunno, aren't XML document fragments sort of a pretty common case? They are. So what? How does that contradict either of the statements made above? Programmers using libxml2 are used to handling this problem. Why must postgres alone use a totally brain dead and utterly incorrect wrapping to solve a problem that everyone else leaves up to the programmer to handle? People in this thread are not concentrating on the fact that what we do now can break correct input. That makes it an unquestionable bug, IMNSHO. There seems to be agreement about what to do for 8.4, so we seem to be arguing about what to do for 8.3. Are you *really* arguing that prepending the xpath expression with '/x' in all cases should be allowed to continue? If you are I can only assume your use of xml/xpath is probably fairly light. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
On Thu, Feb 26, 2009 at 3:18 PM, Andrew Dunstan and...@dunslane.net wrote: I dunno, aren't XML document fragments sort of a pretty common case? They are. So what? How does that contradict either of the statements made above? Programmers using libxml2 are used to handling this problem. Why must postgres alone use a totally brain dead and utterly incorrect wrapping to solve a problem that everyone else leaves up to the programmer to handle? I can't think of any reason, especially not when you put it that way. :-) People in this thread are not concentrating on the fact that what we do now can break correct input. That makes it an unquestionable bug, IMNSHO. There seems to be agreement about what to do for 8.4, so we seem to be arguing about what to do for 8.3. Are you *really* arguing that prepending the xpath expression with '/x' in all cases should be allowed to continue? If you are I can only assume your use of xml/xpath is probably fairly light. Mine is very light indeed. But the change you're proposing seems like it could CONCEIVABLY break a working application that counts on PostgreSQL's particular flavor of misbehavior, and therefore it seems questionable to put that into a stable branch. The fact that the application perhaps should not have been written that way to begin with is neither here nor there. We don't want people to be afraid of upgrading to the latest point release when we fix, say, a backend crash or a data corruption bug. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
On Thu, 2009-02-26 at 15:37 -0500, Robert Haas wrote: Mine is very light indeed. But the change you're proposing seems like it could CONCEIVABLY break a working application that counts on PostgreSQL's particular flavor of misbehavior, That is what release notes are for. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Robert Haas wrote: But the change you're proposing seems like it could CONCEIVABLY break a working application that counts on PostgreSQL's particular flavor of misbehavior, and therefore it seems questionable to put that into a stable branch. The fact that the application perhaps should not have been written that way to begin with is neither here nor there. We don't want people to be afraid of upgrading to the latest point release when we fix, say, a backend crash or a data corruption bug. Let me reiterate the changes that I propose, and again challenge you to provide a working counter-example if you believe it will break anything not currently broken, even cases involving fragments. First, I propose that we abandon this mangling, if, and only if, the xml is in fact a well formed XML document. Since the whole point of the mangling is to handle situations where the XML is not a well formed document, that seems fairly straight-forward. If this change were to upset any user, it must be because they are relying on undisputably incorrect results. Second, I propose that, in the remaining cases, where we do mangle the XML, if the xpath expression does not begin with a '/', instead of prepending it with '/x/, which can not possibly be correct under any circumstance, we prepend it with '/x//' which has some possibility of giving correct results. IOW, these proposals will expand the number of correct results from the code, without contributing any new incorrect cases. These changes are *very* conservative, as is usual when we fix things on stable branches. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
I wrote: Second, I propose that, in the remaining cases, where we do mangle the XML, if the xpath expression does not begin with a '/', instead of prepending it with '/x/, which can not possibly be correct under any circumstance, we prepend it with '/x//' which has some possibility of giving correct results. ^^^ '/x' cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Andrew Dunstan and...@dunslane.net writes: First, I propose that we abandon this mangling, if, and only if, the xml is in fact a well formed XML document. Since the whole point of the mangling is to handle situations where the XML is not a well formed document, that seems fairly straight-forward. If this change were to upset any user, it must be because they are relying on undisputably incorrect results. Second, I propose that, in the remaining cases, where we do mangle the XML, if the xpath expression does not begin with a '/', instead of prepending it with '/x/, which can not possibly be correct under any circumstance, we prepend it with '/x//' which has some possibility of giving correct results. Hmm, does this proposal require adding a test of well-formed-ness to a code path that doesn't currently have one? If so, is that likely to contribute any noticeable slowdown? I can't offhand see an objection to this other than possible performance impact. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed Patch to Improve Performance of Multi-BatchHash Join for Skewed Data Sets
The patch originally modified the cost function but I removed that part before we submitted it to be a bit conservative about our proposed changes. I didn't like that for large plans the statistics were retrieved and calculated many times when finding the optimal query plan. The overhead of the algorithm when the skew optimization is not used ends up being roughly a function call and an if statement per tuple. It would be easy to remove the function call per tuple. Dr. Lawrence has come up with some changes so that when the optimization is turned off, the function call does not happen at all and instead of the if statement happening per tuple it is run just once per join. We have to test this a bit more but it should further reduce the overhead. Hopefully we will have the new patch ready to go this weekend. - Bryce Cutt On Thu, Feb 26, 2009 at 7:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Heikki's got a point here: the planner is aware that hashjoin doesn't like skewed distributions, and it assigns extra cost accordingly if it can determine that the join key is skewed. (See the bucketsize stuff in cost_hashjoin.) If this patch is accepted we'll want to tweak that code. Still, that has little to do with the current gating issue, which is whether we've convinced ourselves that the patch doesn't cause a performance decrease for cases in which it's unable to help. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] DTrace doc patch for new probes in 8.4
Attached is the doc patch for the recently added probes. -Robert Index: doc/src/sgml/monitoring.sgml === RCS file: /projects/cvsroot/pgsql/doc/src/sgml/monitoring.sgml,v retrieving revision 1.63 diff -u -3 -p -r1.63 monitoring.sgml --- doc/src/sgml/monitoring.sgml11 Nov 2008 20:06:21 - 1.63 +++ doc/src/sgml/monitoring.sgml26 Feb 2009 22:18:31 - @@ -1009,23 +1009,25 @@ SELECT pg_stat_get_backend_pid(s.backend productnamePostgreSQL/productname provides facilities to support dynamic tracing of the database server. This allows an external utility to be called at specific points in the code and thereby trace - execution. Currently, this facility is primarily intended for use by - database developers, as it requires substantial familiarity with the code. + execution. /para para - A number of probes or trace points are already inserted - into the source code. By default these probes are not compiled into the + A number of probes or trace points are already inserted into the source + code. These probes are intended to be used by database developers and + administrators. By default the probes are not compiled into the binary, and the user needs to explicitly tell the configure script to make the probes available in productnamePostgreSQL/productname. /para para - Currently, only the DTrace utility is supported, which is available - on Solaris Express, Solaris 10, and Mac OS X Leopard. It is expected that - DTrace will be available in the future on FreeBSD. - Supporting other dynamic tracing utilities is theoretically possible by - changing the definitions for the macros in + Currently, only the + ulink url=http://opensolaris.org/os/community/dtrace/;DTrace/ulink + utility is supported, which is available + on OpenSolaris, Solaris 10, and Mac OS X Leopard. It is expected that + DTrace will be available in the future on FreeBSD and possibly other + operating systems. Supporting other dynamic tracing utilities is + theoretically possible by changing the definitions for the macros in filenamesrc/include/utils/probes.h/. /para @@ -1045,93 +1047,387 @@ SELECT pg_stat_get_backend_pid(s.backend titleBuilt-in Probes/title para - A few standard probes are provided in the source code - (of course, more can be added as needed for a particular problem). - These are shown in xref linkend=trace-point-table. + A number of standard probes are provided in the source code, and more + can certainly be added to enhance PostgreSQL's observability. There are two categories + of probes, those that are targeted toward database administrators and those for developers. + They are shown in xref linkend=admin-trace-point-table and + xref linkend=dev-trace-point-table. /para - table id=trace-point-table - titleBuilt-in Probes/title + table id=admin-trace-point-table + titleBuilt-in Probes for Administrators/title tgroup cols=3 thead row entryName/entry entryParameters/entry - entryOverview/entry + entryDescription/entry /row /thead tbody + row entrytransaction-start/entry - entry(int transactionId)/entry - entryThe start of a new transaction./entry + entry(LocalTransactionId)/entry + entryProbe that fires at the start of a new transaction. arg0 is the transaction id./entry /row row entrytransaction-commit/entry - entry(int transactionId)/entry - entryThe successful completion of a transaction./entry + entry(LocalTransactionId)/entry + entryProbe that fires when a transaction completes successfully. arg0 is the transaction id./entry /row row entrytransaction-abort/entry - entry(int transactionId)/entry - entryThe unsuccessful completion of a transaction./entry + entry(LocalTransactionId)/entry + entryProbes that fires when a transaction does not complete successfully. arg0 is the transaction id./entry +/row +row + entryquery-start/entry + entry(const char *)/entry + entryProbe that fires when the execution of a statement is started. arg0 is the query string./entry +/row +row + entryquery-done/entry + entry(const char *)/entry + entryProbe that fires when the execution of a statement is complete. arg0 is the query string./entry +/row +row + entryquery-parse-start/entry + entry(const char *)/entry + entryProbe that fires when the parsing of a query is started. arg0 is the query string./entry +/row +row + entryquery-parse-done/entry + entry(const char *)/entry + entryProbe that fires when the parsing of a query is complete. arg0 is the query string./entry +/row +row + entryquery-rewrite-start/entry + entry(const char *)/entry + entryProbe that fires when the rewriting of a query is started.
Re: [HACKERS] xpath processing brain dead
On Thu, Feb 26, 2009 at 3:54 PM, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: But the change you're proposing seems like it could CONCEIVABLY break a working application that counts on PostgreSQL's particular flavor of misbehavior, and therefore it seems questionable to put that into a stable branch. The fact that the application perhaps should not have been written that way to begin with is neither here nor there. We don't want people to be afraid of upgrading to the latest point release when we fix, say, a backend crash or a data corruption bug. Let me reiterate the changes that I propose, and again challenge you to provide a working counter-example if you believe it will break anything not currently broken, even cases involving fragments. First, I propose that we abandon this mangling, if, and only if, the xml is in fact a well formed XML document. Since the whole point of the mangling is to handle situations where the XML is not a well formed document, that seems fairly straight-forward. If this change were to upset any user, it must be because they are relying on undisputably incorrect results. Second, I propose that, in the remaining cases, where we do mangle the XML, if the xpath expression does not begin with a '/', instead of prepending it with '/x/, which can not possibly be correct under any circumstance, we prepend it with '/x//' which has some possibility of giving correct results. IOW, these proposals will expand the number of correct results from the code, without contributing any new incorrect cases. These changes are *very* conservative, as is usual when we fix things on stable branches. cheers andrew You are right. Nuff said, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
On Feb 26, 2009, at 9:37 AM, Peter Eisentraut wrote: It's not about ill-formed pieces, it is about (well-formed) content fragments that are not full documents (exactly one root element). libxml2 doesn't support xpath on content fragments. exslt:node-set() to the rescue? Or is that/equivalent functionality not easily accessed at the C-level with libxml2? http://www.exslt.org/exsl/functions/node-set/index.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
James Pye wrote: On Feb 26, 2009, at 9:37 AM, Peter Eisentraut wrote: It's not about ill-formed pieces, it is about (well-formed) content fragments that are not full documents (exactly one root element). libxml2 doesn't support xpath on content fragments. exslt:node-set() to the rescue? Or is that/equivalent functionality not easily accessed at the C-level with libxml2? http://www.exslt.org/exsl/functions/node-set/index.html A node-set isn't a document. In any case, this functionality doesn't appear to be in libxml2, it's in libxslt according to the reference you provided. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1608)
Bruce Momjian wrote: KaiGai Kohei wrote: Bruce Momjian wrote: KaiGai Kohei wrote: The series of SE-PostgreSQL patches for v8.4 were updated: [1/5] http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch [2/5] http://sepgsql.googlecode.com/files/sepgsql-utils-8.4devel-r1608.patch [3/5] http://sepgsql.googlecode.com/files/sepgsql-policy-8.4devel-r1608.patch [4/5] http://sepgsql.googlecode.com/files/sepgsql-docs-8.4devel-r1608.patch [5/5] http://sepgsql.googlecode.com/files/sepgsql-tests-8.4devel-r1608.patch - List of updates: * bugfix: sepgsqlCheckProcedureEntrypoint() was invoked twice when security invoker functions are invoked. Rest of parts are unchanged. Don't mind contracted filename. Please comment anything. It will help to improve our code. I did an analysis of the core file: http://sepgsql.googlecode.com/files/sepgsql-core-8.4devel-r1608.patch changed lines 3226 new files 4075 syscatalog 9977 total 17278 The good news is that 3226 is the affect on the non-system-catalog main core code, and is a context diff size, not total changed lines. Hum...? What utility did you use to compute the lines? It seems to me the changed lines except for system catalogs are larger than actual one. The diffstat says: 65 files changed, 4769 insertions(+), 11 deletions(-), 4945 modifications(!) The (4244 + 500) of 4945 modifications come from pg_proc.h and pg_attribute.h due to a new field to store security label of procedures and columns. The new files adds 4014 in total, so rest of (755 + 11 + 201 = 967) lines are estimated changes in the main core code. Anyway, I believe the burden of reviewer became smaller than the prior full-set version. I manually created the three files and then got totals; I have attached the files. OK, I can understand. It also counts unchanged lines, like: | *** base/src/backend/Makefile Sat Jan 3 13:01:35 2009 | --- sepgsql/src/backend/MakefileFri Feb 6 13:13:08 2009 | *** include $(top_builddir)/src/Makefile.glo | *** 16,22 | |SUBDIRS = access bootstrap catalog parser commands executor foreign lib libpq \ | main nodes optimizer port postmaster regex rewrite \ | ! storage tcop tsearch utils $(top_builddir)/src/timezone | |include $(srcdir)/common.mk | | --- 16,22 | |SUBDIRS = access bootstrap catalog parser commands executor foreign lib libpq \ | main nodes optimizer port postmaster regex rewrite \ | ! security storage tcop tsearch utils $(top_builddir)/src/timezone | |include $(srcdir)/common.mk But it is not a point to be discussed here. The significant point is the scale of patches are indeed reduced as I promised before, by separating a few facilities. :) I've waited for reviewer's comment. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] graph representation of data structures in optimizer
Adriano Lange adri...@c3sl.ufpr.br wrote: The patch attached is an implementation for graph generation of data structures in postgres. The file debuggraph.c is a simple library that generate graphs in the format supported by graphviz. It's interesting, but I don't think it is suitable for a core feature. Could you split the patch into a core-patch and an extension module? The extension module would be put in contrib or in a pgFoundry project. XML might be good for communications between the core and the module; XML-explain was ongoingly discussed, but had not been completed yet. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Kerberos V5 required for PostgreSQL installation on Windows
If Kerberos V5 is not installed on a Windows platform, the following error dialog is returned upon attempted installation: Posgres.exe - Unable to Locate Component This application has failed to start because krb5_32.dll was not found. Re-installing the application may fix this problem. [OK] -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
On Feb 26, 2009, at 5:41 PM, Andrew Dunstan wrote: http://www.exslt.org/exsl/functions/node-set/index.html A node-set isn't a document. yes.. :) I guess what I'm saying is that if: tinman=# SELECT XML'foo/bar/'; xml -- foo/bar/ (1 row) is considered to be valid per *SQL-XML*, then it should probably be treated as a node-set in the context of xpath, not mangled with x.../x.. Certainly, I would expect an implicit node-set() call long before wrapping the fragment in x.../x and prefixing my xpath query. However, I find the validity of the above, XML'foo/bar/', a bit disturbing to begin with. :P In any case, this functionality doesn't appear to be in libxml2, it's in libxslt according to the reference you provided. I think that's *just* referencing the list of xslt implementations that the extension function is known to be available in.. I doubt that means to imply that the function or equivalent functionality is not available in libxml2 itself. I'd wager that equivalent functionality could be implemented if it weren't directly/already available.. =) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Any reason for no pg_get_operatordef?
Hackers, Is there any reason we don't have an pg_get_operatordef() in 8.4, other than nobody writing the code? I don't see any issues in archives ... --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
James Pye wrote: On Feb 26, 2009, at 5:41 PM, Andrew Dunstan wrote: http://www.exslt.org/exsl/functions/node-set/index.html A node-set isn't a document. yes.. :) I guess what I'm saying is that if: tinman=# SELECT XML'foo/bar/'; xml -- foo/bar/ (1 row) is considered to be valid per *SQL-XML*, then it should probably be treated as a node-set in the context of xpath, not mangled with x.../x.. Certainly, I would expect an implicit node-set() call long before wrapping the fragment in x.../x and prefixing my xpath query. However, I find the validity of the above, XML'foo/bar/', a bit disturbing to begin with. :P In any case, this functionality doesn't appear to be in libxml2, it's in libxslt according to the reference you provided. I think that's *just* referencing the list of xslt implementations that the extension function is known to be available in.. I doubt that means to imply that the function or equivalent functionality is not available in libxml2 itself. I'd wager that equivalent functionality could be implemented if it weren't directly/already available.. =) James, can you point me at any call in libxml2 which will evaluate an xpath expression in the context of a nodeset instead of a document? Quite apart from anything else, xpath requires there to be a (single) context node (see http://www.w3.org/TR/xpath20/#context ). For a doc, we set that node to the document node, but what would it be for a node-set or a fragment? If we can't get over that hurdle we're screwed in pursuing your line of thought. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath processing brain dead
Tom Lane wrote: Hmm, does this proposal require adding a test of well-formed-ness to a code path that doesn't currently have one? If so, is that likely to contribute any noticeable slowdown? I can't offhand see an objection to this other than possible performance impact. Yeah, testing the well-formedness might cost a bit. We could short-circuit the test by applying some comparatively fast heuristic tests. Or we could decide that we'll just fix the xpath prefix part for 8.3 and keep the wrapping. I don't want to spend a huge effort on fixing something I regard as fundamentally broken. I'll do some tests to see what the cost of extra xml parsing might be. cheers andrew We -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] a proposal for an extendable deparser
I don't need to add new node types or add any syntax; it is the output that I'm concerned with. What I want is a way to print a tree according to some pretty strict rules. For example, I want a special syntax for function RTEs and I don't want the v::type notation to be output (the flag to turn it off doesn't do what I want). There are lots of uses for specialized pretty-printing. Sometimes you have a simplified syntax reader that can't handle the fully general syntax. For example, you might write an extension in Perl that needs to understand the parse trees. One way to make this work is to print out a simplified syntax from the parse tree and then reparse in Perl. Another use is for general pretty-printing. For example, I modified ruleutils.c to let me print a nice representation of the SQL statement after all of the source-to-source transformations but before the planning. This was a big help in debugging the source-to-source transformations I was working on. As a general rule, you want the list of node types to appear in as few places as possible in order to increase the maintainability of the code. One way to do that is with generic walkers like the ones in Postgresql (a nice solution, by the way). This works well in the case where only a small number of node types need special consideration. The case of printing different though. In printing, there is a default thing to do with each node type --but something different for each type. You want to do the default thing for most of the nodes, but something special for a few types. The best way I know to abstract that sort of process is with a table-driven walker. As for future plans, if you ever get serious about making a big change in the parsing and tree-manipulating code then you might want to look into some of the open-source attribute-grammar and tree-transformation software rather than C++. Those tools are specialized for that kind of work while C++ has some weaknesses in that area. I think some of these tools have BSD-style licenses. The downside is that they require the maintainers to know yet another language. The upside is that they let you work at a higher level of abstraction. And most of them come with built-in pretty printers :-). On Thu, Feb 26, 2009 at 11:54 AM, Tom Lane t...@sss.pgh.pa.us wrote: Dave Gudeman dave.gude...@gmail.com writes: I would replace this with a table-driven deparser: deparse_table[nodeTag(node)](node, context); I don't actually see what this is going to buy for you. You didn't say exactly why ruleutils doesn't work for you, but reading between the lines suggests that you want to add new node types. There are a *ton* of places that need to change for that, typically, and this isn't going to fix it. I've occasionally speculated about the possible value of switching over to method-table-based node types (or maybe just biting the bullet and going to C++) but it never really looked like it would be worth the trouble. regards, tom lane
