Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
The attached patch is revised one based on the V3 approach. The only difference from V3 is that it also applies checks on the AT_AlterColumnType option, not only renameatt(). The performance was almost same as the V3 case. * CVS HEAD 0.828s 0.828s 0.833s 0.829s 0.838s - ALTER RENAME TO with V5 patch 2.419s 2.418s 2.418s 2.426s I also checked ALTER ... TYPE cases. It is relatively heavy operation than renameatt(), so its affects was relatively smaller. - ALTER ... TYPE with CVS HEAD 28.888s 29.948s 30.738s 30.600s - ALTER ... TYPE with V5 patch 28.067s 28.212s 28.038s 29.497s (2010/01/26 10:10), KaiGai Kohei wrote: (2010/01/26 1:11), Bernd Helmle wrote: --On 25. Januar 2010 11:39:21 +0900 KaiGai Koheikai...@ak.jp.nec.com wrote: (echo CREATE TABLE t (a int); for i in `seq 0 9`; do echo CREATE TABLE s$i (b int) INHERITS(t); for j in `seq 0 9`; do echo CREATE TABLE v$i$j (c int) INHERITS(s$i); for k in `seq 0 9`; do echo CREATE TABLE w$i$j$k (d int) INHERITS(v$i$j); for l in `seq 0 9`; do echo CREATE TABLE x$i$j$k$l (e int) INHERITS(w$i$j$k); done done done done) | psql test Well, each table inherits one table in your test. In my test, I inherit from multiple tables for each table. My script generates the following inheritance tree (and wins a price of copy paste ugliness, see attachment): A1, A2, A3, ..., Am B1 INHERITS(A1...A10), B2 INHERITS(A1...A10, B3 INHERITS(A1...A10), ...Bn C1 INHERITS(B1...B10), C2 INHERITS(B1...B10), ... Co D1 INHERITS(C1...C10), ..., Dp m = 10 n = 10 o = 10 p = 1000 Repeating this on my MacBook gives: ALTER TABLE a1 RENAME COLUMN acol1 TO xyz; -HEAD: Time: 382,427 ms Time: 375,974 ms Time: 385,478 ms Time: 371,067 ms Time: 410,834 ms Time: 386,382 ms Recent V4 patch: Time: 6065,673 ms Time: 3823,206 ms Time: 4037,933 ms Time: 3873,029 ms Time: 3899,607 ms Time: 3963,308 ms Hmm... I also could observe similar result in 4 times iteration of ALTER TABLE with your test_rename.sql. I agree the recent V4 patch is poor in performance perspective. * CVS HEAD 0.828s 0.828s 0.833s 0.829s 0.838s * Rcent V4 patch: 10.283s 10.135s 10.107s 10.382s 10.162s * Previous V3 patch: 2.607s 2.429s 2.431s 2.436s 2.428s The V3 patch is designed to compute an expected inhcount for each relations to be altered at first, then it shall be compared to pg_attribute.inhcount to be renamed. Basically, its execution cost is same order except for a case when a relation has diamond inheritance tree. The find_all_inheritors() does not check child relations which is already scanned. However, in this case, we have to check how many times is the child relation inherited from a common origin. I guess it is reason of the different between -HEAD and V3. For example, if we have the following inheritance tree, A2A5 / \ \ A1A4 \ / \ A3 -- A6 The find_all_inheritors() checks existence of directly inherited relations at A1, ... , A6 without any duplications, because this function does not intend to compute how many times was it inherited. The find_all_inheritors_with_inhcount() in V3 patch checks existence of directly inherited relations, even if the target relation is already checked, because it also has to return the times to be inherited from a common origin. In this example, it considers the above structure is same as the following tree. In this diagram, we can find A4 and A5 twice, and A6 thrice. A5 / A2 - A4 - A6 \ A1 \ A3 - A4 - A6 \\ A6 A5 Thus, the test_rename.sql was the worst case test for V3 also. However, I don't think we should keep the bug in the next release. The CVS HEAD's performance is the result of omission for necessary checks. I think we should back to the V3 patch approach, and also reconsider the logic in ATPrepAlterColumnType(). Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com pgsql-fix-inherit-rename.5.patch Description: application/octect-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] About Our CLUSTER implementation is pessimal patch
Consider multi-column indexes, ie: CREATE INDEX i_foo ON foo (length(a), length(b)); Ok, I've never thought of expression indexes that way (in the (expr1,expr2,exprN) form): that is a good example. Maybe you're confusing expression indexes with partial indexes? No no, that was exactly what I needed to know. Thank you very much Leonardo -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. oooh, I could really use this. +1 to put it in 9.1-first CF. however, \conninfo is probably the better name. And what about a +1 on that name. postgresql function version for non-psql connections? How could that function possibly know what the connection looks like from the client side? Think NAT, think proxies, think connection poolers. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] testing cvs HEAD - HS/SR - missing file
Fujii Masao wrote:
*** a/src/backend/replication/walsender.c
--- b/src/backend/replication/walsender.c
***
*** 661,666 XLogSend(StringInfo outMsg)
--- 661,673
sentPtr = endptr;
+ if (sentPtr.xrecoff = XLogFileSize)
+ {
+ /* crossing a logid boundary */
+ sentPtr.xlogid += 1;
+ sentPtr.xrecoff = 0;
+ }
+
/*
* Read the log directly into the output buffer to prevent
* extra memcpy calls.
Before that, endptr is advanced using XLByteAdvance() macro, which does
handle xlogid boundaries. Is XLByteAdvance() broken?
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] testing cvs HEAD - HS/SR - missing file
On Wed, Jan 27, 2010 at 7:05 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Before that, endptr is advanced using XLByteAdvance() macro, which does handle xlogid boundaries. Is XLByteAdvance() broken? No. The cause of the bug is that endptr might be set to the SendRqstPtr that has crossed a xlogid boundary in the following code. /* if we went beyond SendRqstPtr, back off */ if (XLByteLT(SendRqstPtr, endptr)) endptr = SendRqstPtr; Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
How about using the psql prompt to convey this information? IIRC the psql prompt can be configured to show the hostname, server, port and other fields. Wouldn't this be enough? or am I missing something? - Martin - On 27 Jan 2010, at 13:01, Magnus Hagander wrote: 2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. oooh, I could really use this. +1 to put it in 9.1-first CF. however, \conninfo is probably the better name. And what about a +1 on that name. postgresql function version for non-psql connections? How could that function possibly know what the connection looks like from the client side? Think NAT, think proxies, think connection poolers. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] testing cvs HEAD - HS/SR - missing file
Fujii Masao wrote: On Wed, Jan 27, 2010 at 7:05 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Before that, endptr is advanced using XLByteAdvance() macro, which does handle xlogid boundaries. Is XLByteAdvance() broken? No. The cause of the bug is that endptr might be set to the SendRqstPtr that has crossed a xlogid boundary in the following code. /* if we went beyond SendRqstPtr, back off */ if (XLByteLT(SendRqstPtr, endptr)) endptr = SendRqstPtr; But SendRqstPtr comes from LogwrtResult.Write, surely that's correct, no? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
I think the idea is that if you do that, it'll be there all the time, potentially crowding the space. //Magnus 2010/1/27 Martin Atukunda matl...@gmail.com: How about using the psql prompt to convey this information? IIRC the psql prompt can be configured to show the hostname, server, port and other fields. Wouldn't this be enough? or am I missing something? - Martin - On 27 Jan 2010, at 13:01, Magnus Hagander wrote: 2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. oooh, I could really use this. +1 to put it in 9.1-first CF. however, \conninfo is probably the better name. And what about a +1 on that name. postgresql function version for non-psql connections? How could that function possibly know what the connection looks like from the client side? Think NAT, think proxies, think connection poolers. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] testing cvs HEAD - HS/SR - missing file
On Wed, Jan 27, 2010 at 8:37 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: But SendRqstPtr comes from LogwrtResult.Write, surely that's correct, no? Right. But the point is that LogwrtResult.Write might indicate 0/FF00 because it's the last byte + 1 written out. XLogRead() treats this as the location where WAL begins to be read, so converts it to the WAL file name 000100FF by using XLByteToSeg. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
2010/1/27 Josh Berkus j...@agliodbs.com however, \conninfo is probably the better name. +1 Something along the lines of: Connected to localhost port 5432 as user thomb? Thom
Re: [HACKERS] C function accepting/returning cstring vs. text
On Mon, 25 Jan 2010 16:36:46 -0600 Kevin Grittner kevin.gritt...@wicourts.gov wrote: Ivan Sergio Borgonovo m...@webthatworks.it wrote: The README files might be a good place to start, then browse code. Is there a book? The more I read the source and the few info about it, the more I have questions that should have been answered by documenting the function or data structure in spite of looking for some code that use it and see if I can infer what is expecting, what should be the best context to use it in, if there are better candidates to do the same thing etc... - example of returning text (especially if size of outin) - what should I take care of for encoding (what's really inside a tsvector, text, cstring)? src/backend/utils/fmgr/README I just learned there is a return all row mode for returning set functions: There are currently two modes in which a function can return a set result: value-per-call, or materialize. In value-per-call mode, the function returns one value each time it is called, and finally reports done when it has no more values to return. In materialize mode, the function's output set is instantiated in a Tuplestore object; all the values are returned in one call. Additional modes might be added in future. There is no example of a function working in this mode. I'd guess it should be suited for quick operation and small return. But... what should be considered quick and small? When someone should use a row at a time function and a return all row function? I haven't been able to understand the difference between function returning cstring and text and if there is any need to be careful about encoding and escaping when copying from the lexeme to a buffer that will return a cstring or text. Is there any difference from function returning text and function returning cstring? Can I make direct changes to input parameters and return pointers to internal parts of these structures? Or should I always allocate my char*, make the modification there and then return my copy? Is there any operation that should take care of encoding when dealing with cstring or text? thanks -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
2010/1/27 Thom Brown thombr...@gmail.com 2010/1/27 Josh Berkus j...@agliodbs.com however, \conninfo is probably the better name. +1 Something along the lines of: Connected to localhost port 5432 as user thomb? Thom Er... ignore that. Just saw the other examples which are better ;)
Re: [HACKERS] C function accepting/returning cstring vs. text
On 27/01/2010 9:14 PM, Ivan Sergio Borgonovo wrote: On Mon, 25 Jan 2010 16:36:46 -0600 Kevin Grittnerkevin.gritt...@wicourts.gov wrote: Ivan Sergio Borgonovom...@webthatworks.it wrote: The README files might be a good place to start, then browse code. Is there a book? The more I read the source and the few info about it, the more I have questions that should have been answered by documenting the function or data structure in spite of looking for some code that use it and see if I can infer what is expecting, what should be the best context to use it in, if there are better candidates to do the same thing etc... I don't code on PostgreSQL's guts, so I'm perhaps not in the best position to speak, but: - Documentation has a cost too, particularly a maintenance cost. Outdated docs become misleading or downright false and can be much more harm than good. So a reasonable balance must be struck. I'm not saying PostgreSQL is _at_ that reasonable balance re its internal documentation, but there is such a thing as over-documenting. Writing a small book on each function means you have to maintain that, and that gets painful if code is undergoing any sort of major change. - It's easy to say should when you're not the one writing it. Personally, I try to say hey, it's cool that I have access to this system and isn't it great I even have the right to modify it to do what I want, even though the learning curve _can_ be pretty steep. Hey, you could contribute yourself - patch some documentation into those functions where you find that reading the source isn't clear enough, and they really need a see also or called from comment or the like. As it is, I'm extremely grateful for the excellent user-level/admin oriented manual and glad to see the SPI docs too. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, Jan 27, 2010 at 02:14:36PM +0100, Ivan Sergio Borgonovo wrote: I haven't been able to understand the difference between function returning cstring and text and if there is any need to be careful about encoding and escaping when copying from the lexeme to a buffer that will return a cstring or text. Well, the difference is that one is a cstring and the other is text. Seriously though, text is more useful if you want people to be able to use the result in other functions since on SQL level almost everything is text. cstring is needed for some APIs but it generally not used unless necessary. Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Please line up in a tree and maintain the heap invariant while boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] Patch: psql \whoami option
On Jan 27, 2010, at 4:01 AM, Magnus Hagander wrote: 2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. Yeah, I debated that; right now, it just ignores any output it doesn't know about and spits out the human-readable format. oooh, I could really use this. +1 to put it in 9.1-first CF. however, \conninfo is probably the better name. And what about a +1 on that name. That makes at least three, including me. :-) postgresql function version for non-psql connections? How could that function possibly know what the connection looks like from the client side? Think NAT, think proxies, think connection poolers. Yes, this doesn't seem to be a feasible thing to detect in all (many?) cases. Regards, David -- David Christensen End Point Corporation da...@endpoint.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Is there any difference from function returning text and function returning cstring? text is varlena type - it could be TOASTed, comprimated. It is better integrated to PostgreSQL world. cstring is just C zero terminated string - so good for system call, call some external libraries. Regards Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
On Jan 27, 2010, at 5:23 AM, Martin Atukunda wrote: How about using the psql prompt to convey this information? IIRC the psql prompt can be configured to show the hostname, server, port and other fields. Wouldn't this be enough? or am I missing something? Prompt customization is certainly something that could be done (and I use in my .psqlrc), but consider someone unaware of the psql prompt customization or people who are not using their own setup/account, etc. This is a command that could be useful for anyone; as experts, we tend to miss some of the holes in the current interfaces. Regards, David -- David Christensen End Point Corporation da...@endpoint.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
2010/1/27 David Christensen da...@endpoint.com: On Jan 27, 2010, at 4:01 AM, Magnus Hagander wrote: 2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. Yeah, I debated that; right now, it just ignores any output it doesn't know about and spits out the human-readable format. yeah, that's not very forwards-compatible. Someone uses it in the wrong way, and suddenly their stuff gets broken if we choose to modify it in the future. If we say we're only going ot accept two options, let's enforce that and show an error/help message if the user typos. Guessing is not what we do :-) We leave that to other databases... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
On Jan 27, 2010, at 8:08 AM, Magnus Hagander wrote: 2010/1/27 David Christensen da...@endpoint.com: On Jan 27, 2010, at 4:01 AM, Magnus Hagander wrote: 2010/1/27 Josh Berkus j...@agliodbs.com: On 1/26/10 3:24 PM, David Christensen wrote: -hackers, In the spirit of small, but hopefully useful interface improvement patches, enclosed for your review is a patch for providing psql with a \whoami command (maybe a better name is \conninfo or similar). Its purpose is to print information about the current connection, by default in a human-readable format. There is also an optional format parameter which currently accepts 'dsn' as an option to output the current connection information as a DSN. On a first note, it seems like the check for the parameter dsn isn't complete. Without testing it, it looks like it would be possible to run \whoami foobar, which should give an error. Yeah, I debated that; right now, it just ignores any output it doesn't know about and spits out the human-readable format. yeah, that's not very forwards-compatible. Someone uses it in the wrong way, and suddenly their stuff gets broken if we choose to modify it in the future. If we say we're only going ot accept two options, let's enforce that and show an error/help message if the user typos. That's a good point about forward-compatibility. In that case, I'm not sure if default is the best name for the human-readable format, but I didn't like human-readable ;-). I assume that should have an explicit spelling, and not just be the format that we get if we don't otherwise specify. Ideas, anyone? Regards, David -- David Christensen End Point Corporation da...@endpoint.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tim Bunce wrote: - Added plperl.on_perl_init GUC for DBA use (PGC_SIGHUP) SPI functions are not available when the code is run. - Added normal interpreter destruction behaviour END blocks, if any, are run then objects are destroyed, calling their DESTROY methods, if any. SPI functions will die if called at this time. So, are there still objections to applying this patch? Yes. I see I asked the wrong question. Start again. What more should be done to make all or some of it acceptable? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
2010/1/27 KaiGai Kohei kai...@ak.jp.nec.com: The attached patch is revised one based on the V3 approach. The only difference from V3 is that it also applies checks on the AT_AlterColumnType option, not only renameatt(). I think I was clear about what the next step was for this patch in my previous email, but let me try again. http://archives.postgresql.org/pgsql-hackers/2010-01/msg02407.php See also Tom's comments here: http://archives.postgresql.org/pgsql-hackers/2010-01/msg00110.php I don't believe that either Tom or I are prepared to commit a patch based on this approach, at least not unless someone makes an attempt to do it the other way and finds an even more serious problem. If you're not interested in rewriting the patch along the lines Tom suggested, then we should just mark this as Returned with Feedback and move on. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Improving the accuracy of estimate_num_groups()
2010/1/23 Tom Lane t...@sss.pgh.pa.us: I was just poking at the test case provided by Allen Johnson in bug #5294. The essence of the complaint is that the planner is choosing a sort-and-GroupAggregate plan when a HashAggregate-and-then-sort plan would be faster, because the aggregation steps are roughly the same speed either way while post-aggregation sorting is a lot faster because it has fewer rows to process. The reason the planner makes the wrong choice is that it doesn't think the aggregation will reduce the number of rows. And that can be blamed on estimate_num_groups(), which is a largely heuristic affair anyway but seems to fall down particularly badly on the number of groups in a grouped join query. In a join situation, what estimate_num_groups() does is mostly to compute the product of its estimates of the number of groups in each input relation. Of course that's often a huge overestimate. This is masked to some extent by clamping the result to be at most the estimate of the unaggregated join size, which is why we get exactly the same pre-aggregation and post-aggregation rowcount estimates in Allen's example. But we need to do better if we're to have any hope of making intelligent choices about this. The only bit of intelligence estimate_num_groups() adds for join cases is that it throws away any grouping variables that have been found to be equal to other grouping variables; that is, given select ... from ... where a.x = b.y group by a.x, b.y the estimate will be the smaller of the number of x or y values rather than their product. However, that doesn't help in the least for Allen's example, because only one of each pair of join keys appears among the grouping columns. For cases involving equated grouping columns in the same relation, we don't use that heuristic anyway; what we do is compute the product of the number of values and then reduce that by the estimated selectivity of the available restriction clauses. That seems to work reasonably well, or at least better than what is happening at the join level. So it strikes me that maybe we should delete the drop-equal-variables heuristic altogether (basically, reduce add_unique_group_var() to just lappend) and then multiply the ending number-of-groups estimate by the selectivity of the join clauses. In this way we take some account of join clauses that aren't equating one grouping column to another, whereas right now they're completely ignored. Comments? +1 for deleting heuristic part. In most cases this kind of wrong estimate occurs novices designed poor schema and usually they cannot find out the bad point they themselves made then say hey, PostgreSQL is slow. At least it seems to me the possibility that estimated number of rows at pre-aggregate and post-aggregate can be the same should be removed even though actually it can *really* produce such rows. Generally aggregate reduces rows. However, I don't see why reported queries produce different plans. Is NORMAL QUERY join case? To me, HACK QUERY also looks like join case, not same relation case. Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 01:14:16AM -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tim Bunce wrote: - Added plperl.on_perl_init GUC for DBA use (PGC_SIGHUP) SPI functions are not available when the code is run. - Added normal interpreter destruction behaviour END blocks, if any, are run then objects are destroyed, calling their DESTROY methods, if any. SPI functions will die if called at this time. So, are there still objections to applying this patch? Yes. To focus the discussion I've looked back through all the messages from you that relate to this issue so I can summarize and try to address your objections. Some I've split or presented out of order, most relate to earlier (less restricted) versions of the patch before it was split out, and naturally they are lacking some context, so I've included archive URLs. Please forgive and correct me if I misrepresent you or your intent here. Regarding the utility of plperl.on_perl_init and END: http://archives.postgresql.org/message-id/18338.1260033...@sss.pgh.pa.us The question is not about whether we think it's useful; the question is about whether it's safe. I agree. Regarding visibility of changes to plperl.on_perl_init: http://archives.postgresql.org/message-id/28618.1259952...@sss.pgh.pa.us What is to happen if the admin changes the value when the system is already up? If a GUC could be defined as PGC_BACKEND and only settable by superuser, perhaps that would be a good fit. [GucContext seems to conflate some things.] Meanwhile the _init name is meant to convey the fact that it's a before-first-use GUC, like temp_buffers. I'm happy to accept whatever you'd recommend by way of PGC_* GUC selection. Documentation can note any caveats associated with combining plperl.on_perl_init with shared_preload_libraries. http://archives.postgresql.org/message-id/4516.1263168...@sss.pgh.pa.us However, I think PGC_SIGHUP would be enough to address my basic worry, which is that people shouldn't be depending on the ability to set these things within an individual session. The patch uses PGC_SIGHUP for plperl.on_perl_init. http://archives.postgresql.org/message-id/8950.1259994...@sss.pgh.pa.us Tom, what's your objection to Shlib load time being user-visible? It's not really designed to be user-visible. Let me give you just two examples: * We call a plperl function for the first time in a session, causing plperl.so to be loaded. Later the transaction fails and is rolled back. If loading plperl.so caused some user-visible things to happen, should those be rolled back? If so, how do we get perl to play along? If not, how do we get postgres to play along? I believe that's addressed by spi functions being disabled when init code runs. * We call a plperl function for the first time in a session, causing plperl.so to be loaded. This happens in the context of a superuser calling a non-superuser security definer function, or perhaps vice versa. Whose permissions apply to whatever the on_load code tries to do? (Hint: every answer is wrong.) I think that related to on_*trusted_init not plperl.on_perl_init, and is also addressed by spi functions being disabled when init code runs. That doesn't even begin to cover the problems with allowing any of this to happen inside the postmaster. Recall that the postmaster does not have any database access. I believe that's addressed by spi functions being disabled when init code runs. Furthermore, it is a very long established reliability principle around here that the postmaster process should do as little as possible, because every thing that it does creates another opportunity to have a nonrecoverable failure. The postmaster can recover if a child crashes, but the other way round, not so much. I understand that concern. Ultimately, though, that comes down to the judgement of DBAs and the trust placed in them. They can already load arbitrary code via shared_preload_libraries. http://archives.postgresql.org/message-id/18338.1260033...@sss.pgh.pa.us I think if we do this the on_perl_init setting should probably be PGC_POSTMASTER, which would remove any issue about it changing underneath us. Yes, if the main intended usage is in combination with preloading perl at postmaster start, it would be pointless to imagine that PGC_SIGHUP is useful anyway. http://archives.postgresql.org/message-id/17793.1260031...@sss.pgh.pa.us Yeah, in the shower this morning I was thinking that not loading SPI till after the on_init code runs would alleviate the concerns about transactionality and permissions --- that would ensure that whatever on_init does affects only the Perl world and not the database world. That's included in
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 14:44:02 +0100 Martijn van Oosterhout klep...@svana.org wrote: On Wed, Jan 27, 2010 at 02:14:36PM +0100, Ivan Sergio Borgonovo wrote: I haven't been able to understand the difference between function returning cstring and text and if there is any need to be careful about encoding and escaping when copying from the lexeme to a buffer that will return a cstring or text. Well, the difference is that one is a cstring and the other is text. Seriously though, text is more useful if you want people to be able to use the result in other functions since on SQL level almost everything is text. cstring is needed for some APIs but it generally not used unless necessary. I didn't get it. Maybe I really chose the wrong function as an example (tsvectorout). What's not included in on SQL level almost everything is text? There are a lot of functions in contrib taking cstring input and returning cstring output. Are they just in the same special class of [type]in, [type]out [type]recv... functions? I've to re-read carefully http://www.postgresql.org/docs/8.4/static/xfunc-c.html since I discovered there may be explanations about text buffers etc... I discover there is a cstring_to_text function... and a text_to_cstring_buffer too... let me see if I can find something else... thanks -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 21:41:02 +0800 Craig Ringer cr...@postnewspapers.com.au wrote: I don't code on PostgreSQL's guts, so I'm perhaps not in the best position to speak, but: - Documentation has a cost too, particularly a maintenance cost. Outdated docs become misleading or downright false and can be much more harm than good. So a reasonable balance must be struck. I'm not saying PostgreSQL is _at_ that reasonable balance re its internal documentation, but there is such a thing as over-documenting. Writing a small book on each function means you have to maintain that, and that gets painful if code is undergoing any sort of major change. I'd be willing to pay for a book. - It's easy to say should when you're not the one writing it. Personally, I try to say hey, it's cool that I have access to this system and isn't it great I even have the right to modify it to do what I want, even though the learning curve _can_ be pretty steep. Well... I tend to generally make available to others everything I learn. I'd be nice a more advanced use of doxygen so it would be easier to have a map of functions. Hey, you could contribute yourself - patch some documentation into those functions where you find that reading the source isn't clear enough, and they really need a see also or called from comment or the like. Right now I've not enough knowledge to hope my notes get into the source code. Once I've a working piece of code I'll put the information I gathered in the process on my web site and if someone find them worth for a better place I'll release them with a suitable license. -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: What's not included in on SQL level almost everything is text? input and output functions always use cstring rather than text. The I/O functions are normally not used directly at the SQL level. There are a lot of functions in contrib taking cstring input and returning cstring output. Are they just in the same special class of [type]in, [type]out [type]recv... functions? Probably, but I didn't check. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit hook for CVS that disallows \t in SGML files? The process in git is pretty simple. Better write a check in the makefile. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: I just learned there is a return all row mode for returning set functions: There are currently two modes in which a function can return a set result: value-per-call, or materialize. In value-per-call mode, the function returns one value each time it is called, and finally reports done when it has no more values to return. In materialize mode, the function's output set is instantiated in a Tuplestore object; all the values are returned in one call. Additional modes might be added in future There is no example of a function working in this mode. I'd guess it should be suited for quick operation and small return. But... what should be considered quick and small? When someone should use a row at a time function and a return all row function? There are quite a few SRF functions in the code. Look for example in contrib/hstore/hstore_op.c for some fairly simple examples. SRFs are quite capable of returning huge resultsets, not just small ones. Example code for matrerialize mode can be found in the PLs among other places (e.g. plperl_return_next() ) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 11:49:46 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: There are a lot of functions in contrib taking cstring input and returning cstring output. Are they just in the same special class of [type]in, [type]out [type]recv... functions? Probably, but I didn't check. Does this nearly translate to: nothing you should care about right now and anyway just functions that won't return results to SQL? thanks -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo m...@webthatworks.it wrote: There are a lot of functions in contrib taking cstring input and returning cstring output. Are they just in the same special class of [type]in, [type]out [type]recv... functions? If you're looking in contrib subdirectories, perhaps the missing link here is the *.sql.in files. These are what are run in a database to expose functions to the SQL language. Start at those functions and see what the call tree is from there. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 10:10:01 -0500 Andrew Dunstan and...@dunslane.net wrote: There are quite a few SRF functions in the code. Look for example in contrib/hstore/hstore_op.c for some fairly simple examples. SRFs are quite capable of returning huge resultsets, not just small ones. Example code for matrerialize mode can be found in the PLs among other places (e.g. plperl_return_next() ) I'm more interested in understanding when I should use materialized mode. eg. I should be more concerned about memory or cpu cycles and what should be taken as a reference to consider memory needs large? If for example I was going to split a large TEXT into a set of record (let's say I'm processing csv that has been loaded into a text field)... I'd consider the CPU use light but the memory needs large. Would be this task suited for the materialized mode? Is there a rule of thumb to chose between one mode or the other? thanks -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: On Wed, 27 Jan 2010 10:10:01 -0500 Andrew Dunstan and...@dunslane.net wrote: There are quite a few SRF functions in the code. Look for example in contrib/hstore/hstore_op.c for some fairly simple examples. SRFs are quite capable of returning huge resultsets, not just small ones. Example code for matrerialize mode can be found in the PLs among other places (e.g. plperl_return_next() ) I'm more interested in understanding when I should use materialized mode. eg. I should be more concerned about memory or cpu cycles and what should be taken as a reference to consider memory needs large? If for example I was going to split a large TEXT into a set of record (let's say I'm processing csv that has been loaded into a text field)... I'd consider the CPU use light but the memory needs large. Would be this task suited for the materialized mode? Is there a rule of thumb to chose between one mode or the other? If you don't know your memory use will be light, use materialized mode. For small results the data will still be in memory anyway. The Tuplestore will only spill to disk if it grows beyond a certain size. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: I'm more interested in understanding when I should use materialized mode. eg. I should be more concerned about memory or cpu cycles and what should be taken as a reference to consider memory needs large? If for example I was going to split a large TEXT into a set of record (let's say I'm processing csv that has been loaded into a text field)... I'd consider the CPU use light but the memory needs large. Would be this task suited for the materialized mode? Currently, there's no difference in terms of memory needs. The backend always materializes the result of a SRF into a tuplestore anyway, if the function didn't do it itself. There has been discussion of optimizing away that materialization step, but no-one has come up with an acceptable patch for that yet. There probably isn't much difference in CPU usage either. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] make everything target
I'd like to have an everything target that would build all + html + contrib. And maybe an installcheck-everything target that would run installcheck for src, pl and contrib. Thoughts? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review: Typed Table
2010/1/26 Peter Eisentraut pete...@gmx.net: On tis, 2010-01-19 at 01:01 +0900, Hitoshi Harada wrote: I reviewed this patch today. Thank you for this very thorough and helpful review. Comments below and a new patch attached. OK, I confirmed all the issues relevant to the patch were fixed. I'm not so familiar with transaction detail, so I leave it as a known issue. I found ereport() in MergeAttributes() should be indented but except for that there's no issue. So I think I've done my review. I have a follow-up patch that I haven't been able to finish that adds ALTER TYPE stuff to do add/dropping/renaming on the type. I will submit it once we get this patch finalized and I find some time. I'll look at it when it's ready. Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
David Christensen da...@endpoint.com writes: That's a good point about forward-compatibility. In that case, I'm not sure if default is the best name for the human-readable format, but I didn't like human-readable ;-). I assume that should have an explicit spelling, and not just be the format that we get if we don't otherwise specify. Ideas, anyone? I think this patch has near zero usecase already, and more than one output format is *definitely* a waste of time. Forget the argument and just print the human readable format. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: On Wed, 27 Jan 2010 11:49:46 -0300 Alvaro Herrera alvhe...@commandprompt.com wrote: There are a lot of functions in contrib taking cstring input and returning cstring output. Are they just in the same special class of [type]in, [type]out [type]recv... functions? Probably, but I didn't check. Does this nearly translate to: nothing you should care about right now and anyway just functions that won't return results to SQL? I meant they are almost certainly in the same class as typein and typeout, but I didn't check every single one of them. As far as I can tell you are not writing an output function, but rather a debugging function of sorts. I see no reason to return cstring. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
(2010/01/27 23:29), Robert Haas wrote: 2010/1/27 KaiGai Koheikai...@ak.jp.nec.com: The attached patch is revised one based on the V3 approach. The only difference from V3 is that it also applies checks on the AT_AlterColumnType option, not only renameatt(). I think I was clear about what the next step was for this patch in my previous email, but let me try again. http://archives.postgresql.org/pgsql-hackers/2010-01/msg02407.php See also Tom's comments here: http://archives.postgresql.org/pgsql-hackers/2010-01/msg00110.php I don't believe that either Tom or I are prepared to commit a patch based on this approach, at least not unless someone makes an attempt to do it the other way and finds an even more serious problem. If you're not interested in rewriting the patch along the lines Tom suggested, then we should just mark this as Returned with Feedback and move on. The V3/V5 patch was the rewritten one based on the Tom's comment, as is. It counts the expected inhcount at the first find_all_inheritors() time at once, and it compares the pg_attribute.attinhcount. (In actually, find_all_inheritors() does not have a capability to count the number of merged from a common origin, so I newly defined the find_all_inheritors_with_inhcount().) Am I missing something? Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review: listagg aggregate
with actualised oids Regards Pavel Stehule 2010/1/26 David E. Wheeler da...@kineticode.com: On Jan 25, 2010, at 6:56 AM, Pavel Stehule wrote: actualised patch - the name is string_agg All looks fine except I'm getting this error during initdb: creating template1 database in /usr/local/pgsql-devel/data/base/1 ... FATAL: could not create unique index pg_proc_oid_index DETAIL: Key (oid)=(3031) is duplicated. child process exited with exit code 1 Would you mind re-submitting with unique OIDs? Thanks, David string_agg.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 27, 2010 at 12:46:42AM -0700, Alex Hunsaker wrote: FWIW the atexit scares me to. In what way, specifically? It runs too late, and too unpredictably, during the shutdown sequence. (In particular note that shutdown itself might be fired as an atexit callback, a move forced on us by exactly the sort of random user code that you want to add more of. It's not clear whether a Perl-added atexit would fire before or after that.) I understand concerns about interacting with the database, so the patch ensures that any use of spi functions throws an exception. That assuages my fears to only a tiny degree. SPI is not the only possible connection between perl code and the rest of the backend. Indeed, AFAICS the major *point* of these additions is to allow people to insert unknown other functionality that is likely to interact with the rest of the backend; a prospect that doesn't make me feel better about it. Specifically, how is code that starts executing at the end of a session different in risk to code that starts executing before the end of a session? If it runs before the shutdown sequence starts, we know we have a functioning backend. Once shutdown starts, it's unknown and mostly untested exactly what subsystems will still work and which won't. Injecting arbitrary user-written code into an unspecified point in that sequence is not a recipe for good results. Lastly, an atexit trigger will still fire during FATAL or PANIC aborts, which scares me even more. When the house is already afire, it's not prudent to politely let user-written perl code do whatever it wants before you get the heck out of there. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tom Lane wrote: Indeed, AFAICS the major *point* of these additions is to allow people to insert unknown other functionality that is likely to interact with the rest of the backend; a prospect that doesn't make me feel better about it. No. The major use case we've seen for END blocks is to allow a profiler to write its data out. That should have zero interaction with the rest of the backend. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Indeed, AFAICS the major *point* of these additions is to allow people to insert unknown other functionality that is likely to interact with the rest of the backend; a prospect that doesn't make me feel better about it. No. The major use case we've seen for END blocks is to allow a profiler to write its data out. That should have zero interaction with the rest of the backend. Really? We've found that gprof, for instance, doesn't exactly have zero interaction with the rest of the backend --- there's actually a couple of different bits in there to help it along, including a behavioral change during shutdown. I rather doubt that Perl profilers would turn out much different. But in any case, I don't believe for a moment that profiling is the only or even the largest use to which people would try to put this. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] testing cvs HEAD - HS/SR - missing file
Fujii Masao wrote:
On Wed, Jan 27, 2010 at 8:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
But SendRqstPtr comes from LogwrtResult.Write, surely that's correct, no?
Right. But the point is that LogwrtResult.Write might indicate 0/FF00
because it's the last byte + 1 written out. XLogRead() treats this as the
location where WAL begins to be read, so converts it to the WAL file name
000100FF by using XLByteToSeg.
Ah, I see it now, thanks. How confusing.
Your patch clearly works, but I wonder how we should logically treat
that boundary value. You're setting sentPtr to the beginning of the
logid, when LogwrtResult.Write is 0/FF00. So you're logically
thinking that the non-existent FF log segment has been sent to the
standby as soon as the previous segment has been fully sent. Another
option would be to set SendRqstPtr to the beginning of next logid, when
LogwrtResult.Write is 0/FF00.. Yet another option would be to set
startptr to the beginning of next logid, when sentPtr is 0/FF00:
***
*** 633,638
--- 633,648
* WAL record.
*/
startptr = sentPtr;
+ if (startptr.xrecoff = XLogFileSize)
+ {
+ /*
+* crossing a logid boundary, skip the non-existent
last log
+* segment in previous logical log file.
+*/
+ startptr.xlogid += 1;
+ startptr.xrecoff = 0;
+ }
+
endptr = startptr;
XLByteAdvance(endptr, MAX_SEND_SIZE);
/* round down to page boundary. */
I think I prefer that, it feels logically more correct. Setting sentPtr
beyond LogwrtResult.Write feels wrong, even though 0/FF00 and
1/ are really the same physical location. I'll commit it that way.
I also note that I broke that same thing in ReadRecord() in the
retry-logic patch I just comitted. I'll fix that too.
Thanks for the testing, Erik!
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: psql \whoami option
Magnus Hagander mag...@hagander.net writes: 2010/1/27 Martin Atukunda matl...@gmail.com: How about using the psql prompt to convey this information? I think the idea is that if you do that, it'll be there all the time, potentially crowding the space. I had the same reaction as Martin. If you want this info all the time, setting the prompt is the way to go. If you only want it occasionally, you're probably more likely to remember other ways of getting the info first (session_user, pg_stat_activity, etc etc). There may be some use-case in between where another backslash command would be helpful, but it seems a tad marginal to me. I don't object as long as it's not overdesigned, but let's keep the bells and whistles to a minimum. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 11:13:43AM -0500, Tom Lane wrote: Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 27, 2010 at 12:46:42AM -0700, Alex Hunsaker wrote: FWIW the atexit scares me to. In what way, specifically? It runs too late, and too unpredictably, during the shutdown sequence. (In particular note that shutdown itself might be fired as an atexit callback, a move forced on us by exactly the sort of random user code that you want to add more of. It's not clear whether a Perl-added atexit would fire before or after that.) man atexit says Functions so registered are called in reverse order. Since the plperl atexit is called only when a plperl SP or DO is executed it would fire before any atexit() registered during startup. The timing and predictability shouldn't be a significant concern if the plperl subsystem can't interact with the rest of the backend - which is the intent. I understand concerns about interacting with the database, so the patch ensures that any use of spi functions throws an exception. That assuages my fears to only a tiny degree. SPI is not the only possible connection between perl code and the rest of the backend. Could you give me some examples of others? Indeed, AFAICS the major *point* of these additions is to allow people to insert unknown other functionality that is likely to interact with the rest of the backend; a prospect that doesn't make me feel better about it. The major point is *not at all* to allow people to interact with the rest of the backend. I'm specifically trying to limit that. The major point is simply to allow perl code to clean itself up properly. Specifically, how is code that starts executing at the end of a session different in risk to code that starts executing before the end of a session? If it runs before the shutdown sequence starts, we know we have a functioning backend. Once shutdown starts, it's unknown and mostly untested exactly what subsystems will still work and which won't. Injecting arbitrary user-written code into an unspecified point in that sequence is not a recipe for good results. The plperl subsystem is isolated from, and can't interact with, the rest of the backend during shutdown. Can you give me examples where that's not the case? Lastly, an atexit trigger will still fire during FATAL or PANIC aborts, which scares me even more. When the house is already afire, it's not prudent to politely let user-written perl code do whatever it wants before you get the heck out of there. Again, that point rests on your underlying concern about interaction between plperl and the rest of the backend. Examples? Is there some way for plperl.c to detect a FATAL or PANIC abort? If so, or if one could be added, then we could skip the END code in those circumstances. I don't really want to add more GUCs, but perhaps controlling END block execution via a plperl.destroy_end=bool (default false) would help address your concerns. Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: Is there a book? You'll find a basic intro to this area in PostgreSQL Developer's Handbook by Geschwinde and Schonig. You're already past the level of questions they answer in there though. This whole cstring/text issue you're asking about, I figured out by reading the source code, deciphering the examples in the talk at http://www.joeconway.com/presentations/tut_oscon_2004.pdf , and then using interesting keywords there (DatumGetCString is a good one to find interesting places in the code) to find code examples and messages on that topic in the list archives. Finally, finding some commits/patches that do things similar to what you want, and dissecting how they work, is quite informative too. I agree the learning curve could be a shortened a lot, and have actually submitted a conference talk proposal in this area to try and improve that. You're a few months ahead of me having something written down to share though. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tom Lane wrote: But in any case, I don't believe for a moment that profiling is the only or even the largest use to which people would try to put this. Well, ISTR there have been requests over the years for event handlers for (among other things) session shutdown, so if you're speculating that people would use this as an end run around our lack of such things you could be right. Maybe providing for such handlers in a more general and at the same time more safe way would be an alternative. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Andrew Dunstan and...@dunslane.net writes: I see I asked the wrong question. Start again. What more should be done to make all or some of it acceptable? I think a must is to get rid of the use of atexit(). Possibly an on_proc_exit callback could be used instead, although I'm not sure how you'd handle the case of code loaded in the postmaster that would like corresponding exit-time code to happen in child processes. (OTOH, it seems likely that it's impossible to make that work correctly anyway. It certainly isn't going to work the same on EXEC_BACKEND platforms as anywhere else, and I don't particularly want to see us documenting that the feature works differently on Windows than elsewhere.) Dropping the ability to make the postmaster run any such code would go a very long way towards fixing the above, as well as assuaging other fears. The other thing that I find entirely unconvincing is Tim's idea that shutting off SPI isolates perl from the rest of the backend. I have no confidence in that, but no real idea of how to do better either :-(. If you think that shutting off SPI is sufficient, you can find counterexamples in the CVS history, for instance where we had to take special measures to prevent Perl from screwing up the locale settings. I'm afraid that on_perl_init is going to vastly expand the opportunities for that kind of unwanted side-effect; and the earlier that it runs, the more likely it's going to be that we can't recover easily. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
f...@redhat.com (Frank Ch. Eigler) writes: Tom Lane t...@sss.pgh.pa.us writes: Lastly, an atexit trigger will still fire during FATAL or PANIC aborts, which scares me even more. When the house is already afire, it's not prudent to politely let user-written perl code do whatever it wants before you get the heck out of there. Is there a reason that these panics don't use _exit(3) to bypass atexit hooks? Well, I don't really want to entirely forbid the use of atexit() --- I'm just concerned about using it to run arbitrary user-written code. There might be more limited purposes for which it's a reasonable choice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On Wed, Jan 27, 2010 at 04:52:00PM +0200, Peter Eisentraut wrote: On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit hook for CVS that disallows \t in SGML files? The process in git is pretty simple. Better write a check in the makefile. That assumes that everyone will always actually run the appropriate make before committing, which I wouldn't assume, especially for a trivial patch that's going out when someone is pressed for time. Checking in an unavoidable place seems much more foolproof. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 27, 2010 at 11:13:43AM -0500, Tom Lane wrote: (In particular note that shutdown itself might be fired as an atexit callback, a move forced on us by exactly the sort of random user code that you want to add more of. It's not clear whether a Perl-added atexit would fire before or after that.) man atexit says Functions so registered are called in reverse order. Since the plperl atexit is called only when a plperl SP or DO is executed it would fire before any atexit() registered during startup. Right, which means that it would occur either before or after on_proc_exit processing, depending on whether we got there through an exit() call or via the normal proc_exit sequence. That's just the kind of instability I don't want to have to debug. The plperl subsystem is isolated from, and can't interact with, the rest of the backend during shutdown. This is exactly the claim that I have zero confidence in. Quite frankly, the problem with Perl as an extension language is that Perl was never designed to be a subsystem: it feels free to mess around with the entire state of the process. We've been burnt multiple times by that even with the limited use we make of Perl now, and these proposed additions are going to make it a lot worse IMO. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tom Lane t...@sss.pgh.pa.us writes: [...] Lastly, an atexit trigger will still fire during FATAL or PANIC aborts, which scares me even more. When the house is already afire, it's not prudent to politely let user-written perl code do whatever it wants before you get the heck out of there. Is there a reason that these panics don't use _exit(3) to bypass atexit hooks? - FChE -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
David Fetter da...@fetter.org writes: On Wed, Jan 27, 2010 at 04:52:00PM +0200, Peter Eisentraut wrote: Better write a check in the makefile. That assumes that everyone will always actually run the appropriate make before committing, which I wouldn't assume, especially for a trivial patch that's going out when someone is pressed for time. Checking in an unavoidable place seems much more foolproof. The rule against tabs in SGML files is only a bit of anal retentivity anyway. I don't think that enforcing it is worth special hacks that we do not use anywhere else. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On Wed, Jan 27, 2010 at 12:27:08PM -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Wed, Jan 27, 2010 at 04:52:00PM +0200, Peter Eisentraut wrote: Better write a check in the makefile. That assumes that everyone will always actually run the appropriate make before committing, which I wouldn't assume, especially for a trivial patch that's going out when someone is pressed for time. Checking in an unavoidable place seems much more foolproof. The rule against tabs in SGML files is only a bit of anal retentivity anyway. I don't think that enforcing it is worth special hacks that we do not use anywhere else. Good point. What other pre-commit hooks do we need? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On Wed, 2010-01-27 at 12:27 -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Wed, Jan 27, 2010 at 04:52:00PM +0200, Peter Eisentraut wrote: Better write a check in the makefile. That assumes that everyone will always actually run the appropriate make before committing, which I wouldn't assume, especially for a trivial patch that's going out when someone is pressed for time. Checking in an unavoidable place seems much more foolproof. The rule against tabs in SGML files is only a bit of anal retentivity anyway. I disagree. SGML suffers the same as code if you are using tabs. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
David Fetter da...@fetter.org writes: On Wed, Jan 27, 2010 at 12:27:08PM -0500, Tom Lane wrote: The rule against tabs in SGML files is only a bit of anal retentivity anyway. I don't think that enforcing it is worth special hacks that we do not use anywhere else. Good point. What other pre-commit hooks do we need? Hm, aren't you one of the main cheerleaders for moving to git? Why would you want us to be installing any additional CVS dependencies right now? Adding commit hooks would just create another must-do- before-moving task. If we actually desperately *needed* some, maybe it'd be worth it, but I can't see that this kind of thing is worth it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On Wed, Jan 27, 2010 at 12:41:28PM -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Wed, Jan 27, 2010 at 12:27:08PM -0500, Tom Lane wrote: The rule against tabs in SGML files is only a bit of anal retentivity anyway. I don't think that enforcing it is worth special hacks that we do not use anywhere else. Good point. What other pre-commit hooks do we need? Hm, aren't you one of the main cheerleaders for moving to git? Yes. Why would you want us to be installing any additional CVS dependencies right now? Adding commit hooks would just create another must-do- before-moving task. Right now is not the time. If we actually desperately *needed* some, maybe it'd be worth it, but I can't see that this kind of thing is worth it. Post-git-move would be a better time to revisit this. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 17:37:23 +0200 Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Currently, there's no difference in terms of memory needs. The backend always materializes the result of a SRF into a tuplestore anyway, if the function didn't do it itself. There has been discussion of optimizing away that materialization step, but no-one has come up with an acceptable patch for that yet. There probably isn't much difference in CPU usage either. On Wed, 27 Jan 2010 10:34:10 -0500 Andrew Dunstan and...@dunslane.net wrote: If you don't know your memory use will be light, use materialized mode. For small results the data will still be in memory anyway. The Tuplestore will only spill to disk if it grows beyond a certain size. I keep on missing something. Considering all the context switching stuff, the first call initialization etc... I'd expect that what's collecting the result was going to push it downstream a bit at a time. What's actually happening is... the result get collected anyway in a Tuplestore. But then... why do we have all that logic to save the function context if anyway it is more convenient to process everything in one run? It's a pain to save the context just to save a pointer inside a structure, it would be more convenient to just process all the structure and return it as a Tuplestore in one pass. BTW thanks to Greg for pointing me once more to Joe Conway's tutorial. When I read it the first time I wasn't in a condition to take advantage of it. Now it looks more useful. -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 9:08 AM, Tom Lane wrote: This is exactly the claim that I have zero confidence in. Quite frankly, the problem with Perl as an extension language is that Perl was never designed to be a subsystem: it feels free to mess around with the entire state of the process. We've been burnt multiple times by that even with the limited use we make of Perl now, and these proposed additions are going to make it a lot worse IMO. Can you provide an example? Such concerns are impossible to address without concrete examples. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
David E. Wheeler da...@kineticode.com writes: On Jan 27, 2010, at 9:08 AM, Tom Lane wrote: This is exactly the claim that I have zero confidence in. Quite frankly, the problem with Perl as an extension language is that Perl was never designed to be a subsystem: it feels free to mess around with the entire state of the process. We've been burnt multiple times by that even with the limited use we make of Perl now, and these proposed additions are going to make it a lot worse IMO. Can you provide an example? Such concerns are impossible to address without concrete examples. Two examples that I can find in a quick review of our CVS history: perl stomping on the process's setlocale state, and perl stomping on the stdio state (Windows only). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 10:08 AM, Tom Lane wrote: Two examples that I can find in a quick review of our CVS history: perl stomping on the process's setlocale state, and perl stomping on the stdio state (Windows only). Are there links to those commits? Thanks, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make everything target
On ons, 2010-01-27 at 10:41 -0500, Andrew Dunstan wrote: I'd like to have an everything target that would build all + html + contrib. +10 And maybe an installcheck-everything target that would run installcheck for src, pl and contrib. +100 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review: Typed Table
Everyone, We could use some help. Anyone's got an idea what could be causing the behavior described below? On mån, 2010-01-25 at 21:45 +0200, Peter Eisentraut wrote: On tis, 2010-01-19 at 01:01 +0900, Hitoshi Harada wrote: * Conflict between transactions I'm not sure if this is related with the patch but I met this situation; A: regression=# create type persons_type as (name text, bdate date); A: CREATE TYPE A: regression=# begin; A: BEGIN A: regression=# drop type persons_type; A: DROP TYPE B: regression=# create table persons of persons_type; (LOCK) A: regression=# rollback; A: ROLLBACK B: CREATE TABLE B: regression=# drop table persons; B: DROP TABLE A: regression=# begin; A: BEGIN A: regression=# drop type persons_type; A: DROP TYPE B: regression=# create table persons of persons_type; (NO LOCK) B: CREATE TABLE A: regression=# commit; A: COMMIT B: regression=# select 'persons_type'::regtype; B: ERROR: type persons_type does not exist B: LINE 1: select 'persons_type'::regtype; I have at all no idea why the second create table doesn't lock. Well, if you try the same thing with CREATE FUNCTION foo() RETURNS persons_type AS $$ blah $$ LANGUAGE plpythonu; or some similar cases, there is also no lock. You will notice that (some/many?) DDL statements actually behave very poorly against concurrent other DDL. Against that background, however, the real question is why the first case *does* lock. I don't know. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clustering Docs WAS: Mammoth in Core?
Greg Smith wrote: We've got pgsql-cluster-hackers to discuss this particular area. Huh, is this a new list? It wasn't added to wwwmaster's list of lists, apparently, right? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clustering Docs WAS: Mammoth in Core?
Alvaro Herrera wrote: Greg Smith wrote: We've got pgsql-cluster-hackers to discuss this particular area. Huh, is this a new list? It wasn't added to wwwmaster's list of lists, apparently, right? The archives are at http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not listed at http://archives.postgresql.org/ for some reason. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Google Summer of Code 2010 is on! (applications due March 9)
Hi! I'm happy to facilitate this and get the details in for our application. Seems like we have lots of things that we could get students involved with, and of course, we tend to get interesting projects pitched to us that we haven't thought of before. I've attended the Mentor Summit after GSoC for the last two years, and found it to be a great resource for learning how all projects are recruiting and retaining new members. Many of these projects admire Postgres for it's stability over the years, and I've been able to communicate some of the work that we've all done on the commit fest, our review process and generally how our community works to many other projects. Those who would like to mentor and help administrate the program, please get in touch with me directly sel...@postgresql.org. I'll set up some lists to keep us in sync, and we'll report out what happens. -selena -- Forwarded message -- From: LH (Leslie Hawthorn) lho...@gmail.com Date: Tue, Jan 26, 2010 at 4:10 PM Subject: GSoC 2010 is on. To: Google Summer of Code Mentors List google-summer-of-code-mentors-l...@googlegroups.com Hello folks, If you're not following the program discussion list, you may have missed this mail with some details about timing for GSoC 2010: http://groups.google.com/group/google-summer-of-code-discuss/browse_thread/thread/d839c0b02ac15b3f Cheers, LH --- Hello everyone, Many of you were wondering if Google Summer of Code 2010 is on for 2010 and the answer is yes! We will begin accepting application from would-be mentoring organizations beginning March 8th at approximately 19:00 UTC, with applications closing on March 12th at 23:00 UTC. Students can apply between 19:00 UTC on March 29th to 19:00 UTC on April 9th. We will be updating the program website [0], including the FAQs [1], over the next few days. Not much is going to change except the timeline, though there are a few FAQs that will be added. If you have any questions in the interim please do send them to the list. Please note that questions about which project you should select, etc., will likely be responded to with depends on your skill set, so please take the time to do some research [2] about which project is right for you. While mentoring organizations for 2010 will be announced until March 18th, a great number of the projects that have participated in the past [3] will return once again in 2010. Of course, that's not a guarantee and we cannot accept every great project that applies. But if you want to get a jump start on your participation in GSoC 2010, considering lurking in the IRC channel of a past participating project and seeing what useful things you can learn about their code base and community. Check out project forums to see if it's a place you'd like to spend your time. I'll be sending a follow up message in a few minutes about what to expect in the next few days/weeks and how you can help Google Summer of Code 2010. Google Summer of Code 2010 is on! Celebrate! [0] - http://socghop.appspot.com [1] - http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2009... [2] - http://delicious.com/gsoc2009 [3] - http://socghop.appspot.com/gsoc/program/accepted_orgs/google/gsoc2009 Cheers, LH -- Leslie Hawthorn Program Manager - Open Source Google Inc. -- http://chesnok.com/daily - me http://endpoint.com - work -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clustering Docs WAS: Mammoth in Core?
Greg Smith wrote: Alvaro Herrera wrote: Greg Smith wrote: We've got pgsql-cluster-hackers to discuss this particular area. Huh, is this a new list? It wasn't added to wwwmaster's list of lists, apparently, right? The archives are at http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not listed at http://archives.postgresql.org/ for some reason. Because it wasn't added to the database. It seems we've gotten sloppy about the list creation process (though I admit it doesn't seem to be documented anywhere). If you give me a description for the list I'll add it. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review: Typed Table
Peter Eisentraut escribió: Everyone, We could use some help. Anyone's got an idea what could be causing the behavior described below? I wonder if the problem is that you're missing a recheck on the type's existence after you've grabbed the lock on it, similar to what shdepLockAndCheckObject does. Maybe the second attempt to create the table doesn't block because pg_depends contents are different? It seems very strange. On mån, 2010-01-25 at 21:45 +0200, Peter Eisentraut wrote: On tis, 2010-01-19 at 01:01 +0900, Hitoshi Harada wrote: * Conflict between transactions I'm not sure if this is related with the patch but I met this situation; A: regression=# create type persons_type as (name text, bdate date); A: CREATE TYPE A: regression=# begin; A: BEGIN A: regression=# drop type persons_type; A: DROP TYPE B: regression=# create table persons of persons_type; (LOCK) A: regression=# rollback; A: ROLLBACK B: CREATE TABLE B: regression=# drop table persons; B: DROP TABLE A: regression=# begin; A: BEGIN A: regression=# drop type persons_type; A: DROP TYPE B: regression=# create table persons of persons_type; (NO LOCK) B: CREATE TABLE A: regression=# commit; A: COMMIT B: regression=# select 'persons_type'::regtype; B: ERROR: type persons_type does not exist B: LINE 1: select 'persons_type'::regtype; I have at all no idea why the second create table doesn't lock. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On 01/27/2010 09:49 AM, Ivan Sergio Borgonovo wrote: On Wed, 27 Jan 2010 17:37:23 +0200 Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Currently, there's no difference in terms of memory needs. The backend always materializes the result of a SRF into a tuplestore anyway, if the function didn't do it itself. There has been discussion of optimizing away that materialization step, but no-one has come up with an acceptable patch for that yet. I keep on missing something. But then... why do we have all that logic to save the function context if anyway it is more convenient to process everything in one run? As already pointed out, there is currently no difference between value_per_call and materialize modes. The original intent was to have both modes eventually, with the initial one being materialize (historical note: my first SRF patch was value_per_call, but had difficult issues to be solved, and the consensus was that materialize was a simpler and safer approach for the first try at this feature). The advantage of value_per_call (if it were not materialized anyway in the backend) would be to allow pipelining, which enables very large datasets to be streamed without exhausting memory or having to wait for it all to be materialized. Implementing true value_per_call is still something on my TODO list, but obviously has not risen to a very high priority for me as it has now been an embarrassing long time since it was put there. But that said, materialize mode has proven extremely good at covering the most common use cases with acceptable performance. Joe signature.asc Description: OpenPGP digital signature
Re: [HACKERS] quoting psql varible as identifier
On Mon, Jan 25, 2010 at 7:36 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I hope, so this version is more readable and more clean. I removed some not necessary checks. This still seems overly complicated to me. I spent a few hours today working up the attached patch. Let me know your thoughts. ...Robert *** a/doc/src/sgml/ref/psql-ref.sgml --- b/doc/src/sgml/ref/psql-ref.sgml *** *** 658,664 testdb=gt; para If an unquoted argument begins with a colon (literal:/literal), it is taken as a applicationpsql/ variable and the value of the ! variable is used as the argument instead. /para para --- 658,669 para If an unquoted argument begins with a colon (literal:/literal), it is taken as a applicationpsql/ variable and the value of the ! variable is used as the argument instead. If the variable name is ! surrounded by single quotes (e.g. literal:'var'/literal), it ! will be escaped as an SQL literal and the result will be used as ! the argument. If the variable name is surrounded by double quotes, ! it will be escaped as an SQL identifier and the result will be used ! as the argument. /para para *** *** 2711,2728 bar para An additional useful feature of applicationpsql/application variables is that you can substitute (quoteinterpolate/quote) ! them into regular acronymSQL/acronym statements. The syntax for ! this is again to prepend the variable name with a colon (literal:/literal): programlisting testdb=gt; userinput\set foo 'my_table'/userinput testdb=gt; userinputSELECT * FROM :foo;/userinput /programlisting ! would then query the table literalmy_table/literal. The value of ! the variable is copied literally, so it can even contain unbalanced ! quotes or backslash commands. You must make sure that it makes sense ! where you put it. Variable interpolation will not be performed into ! quoted acronymSQL/acronym entities. /para para --- 2716,2750 para An additional useful feature of applicationpsql/application variables is that you can substitute (quoteinterpolate/quote) ! them into regular acronymSQL/acronym statements. ! applicationpsql/application provides special facilities for ! ensuring that values used as SQL literals and identifiers are ! properly escaped. The syntax for interpolating a value without ! any special escaping is again to prepend the variable name with a colon (literal:/literal): programlisting testdb=gt; userinput\set foo 'my_table'/userinput testdb=gt; userinputSELECT * FROM :foo;/userinput /programlisting ! would then query the table literalmy_table/literal. Note that this ! may be unsafe: the value of the variable is copied literally, so it can ! even contain unbalanced quotes or backslash commands. You must make sure ! that it makes sense where you put it. ! /para ! ! para ! When a value is to be used as an SQL literal or identifier, it is ! safest to arrange for it to be escaped. To escape the value of ! a variable as an SQL literal, write a colon followed by the variable ! name in single quotes. To escape the value an SQL identifier, write ! a colon followed by the variable name in double quotes. The previous ! example would be more safely written this way: ! programlisting ! testdb=gt; userinput\set foo 'my_table'/userinput ! testdb=gt; userinputSELECT * FROM :foo;/userinput ! /programlisting ! Variable interpolation will not be performed into quoted ! acronymSQL/acronym entities. /para para *** *** 2730,2769 testdb=gt; userinputSELECT * FROM :foo;/userinput copy the contents of a file into a table column. First load the file into a variable and then proceed as above: programlisting ! testdb=gt; userinput\set content `cat my_file.txt` /userinput ! testdb=gt; userinputINSERT INTO my_table VALUES (:content);/userinput ! /programlisting ! One problem with this approach is that filenamemy_file.txt/filename ! might contain single quotes. These need to be escaped so that ! they don't cause a syntax error when the second line is processed. This ! could be done with the program commandsed/command: ! programlisting ! testdb=gt; userinput\set content `sed -e s/'/''/g lt; my_file.txt` /userinput ! /programlisting ! If you are using non-standard-conforming strings then you'll also need ! to double backslashes. This is a bit tricky: ! programlisting ! testdb=gt; userinput\set content `sed -e s/'/''/g -e 's/\\//g' lt; my_file.txt` /userinput /programlisting ! Note the use of different shell quoting conventions so that neither ! the single quote marks nor the backslashes are special to the shell. ! Backslashes are still special to commandsed/command,
Re: [HACKERS] Patch: Remove gcc dependency in definition of inline functions
On Tue, Jan 19, 2010 at 4:55 PM, Kurt Harriman harri...@acm.org wrote: I'll submit an updated patch. We need this patch pretty soon if we're going to include this in 9.0, I think. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Ivan Sergio Borgonovo wrote: But then... why do we have all that logic to save the function context if anyway it is more convenient to process everything in one run? It's a pain to save the context just to save a pointer inside a structure, it would be more convenient to just process all the structure and return it as a Tuplestore in one pass. When the set-returning-function feature was written originally, years ago, the tuple at a time mode did really work tuple at a time. But it had issues and was axed out of the patch before it was committed, to keep it simple. The idea was to revisit it at some point, but it hasn't bothered anyone enough to fix it. It's basically not implemented yet. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xpath improvement suggestion
On Sun, Jan 17, 2010 at 11:33 AM, Jan Urbański wulc...@wulczer.org wrote: [ detailed review ] Arie, Are you planning to submit an updated patch? If so, please do so soon. Thanks, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
On Wed, Jan 27, 2010 at 10:17 AM, KaiGai Kohei kai...@kaigai.gr.jp wrote: (2010/01/27 23:29), Robert Haas wrote: 2010/1/27 KaiGai Koheikai...@ak.jp.nec.com: The attached patch is revised one based on the V3 approach. The only difference from V3 is that it also applies checks on the AT_AlterColumnType option, not only renameatt(). I think I was clear about what the next step was for this patch in my previous email, but let me try again. http://archives.postgresql.org/pgsql-hackers/2010-01/msg02407.php See also Tom's comments here: http://archives.postgresql.org/pgsql-hackers/2010-01/msg00110.php I don't believe that either Tom or I are prepared to commit a patch based on this approach, at least not unless someone makes an attempt to do it the other way and finds an even more serious problem. If you're not interested in rewriting the patch along the lines Tom suggested, then we should just mark this as Returned with Feedback and move on. The V3/V5 patch was the rewritten one based on the Tom's comment, as is. It counts the expected inhcount at the first find_all_inheritors() time at once, and it compares the pg_attribute.attinhcount. (In actually, find_all_inheritors() does not have a capability to count the number of merged from a common origin, so I newly defined the find_all_inheritors_with_inhcount().) Am I missing something? Err... I'm not sure. I thought I understood what the different versions of this patch were doing, but apparently I'm all confused. I'll take another look at this. Bernd (or anyone), feel free to take a look in parallel. More eyes would be helpful... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
--On 27. Januar 2010 15:42:45 -0500 Robert Haas robertmh...@gmail.com wrote: Bernd (or anyone), feel free to take a look in parallel. More eyes would be helpful... I've planned to look at this tomorrow when i'm back in office. -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] CommitFest status summary 2010-01-27
We have 20 remaining patches to deal with for the 2010-01 CommitFest. I've attempted to break down the status of these patches below. The good news is that almost half of the remaining patches are either already marked as Ready for Committer, or have already been reviewed once and will likely be marked Ready for Committer when they are re-reviewed. If you are the reviewer for one of these patches, please re-review and mark it either Waiting on Author if you find additional issues or Ready for Committer if it looks good. The somewhat-less-good news is that we still have at least four patches that have not been reviewed at all, and one of those (plpython3) has been very difficult to find a reviewer for. Some of the perl patches have not yet been reviewed either, but I'm a little less concerned about those because it seems that Andrew is working on those anyway: still, if anyone feels inclined to volunteer, I believe Andrew has said he would appreciate another pair of eyes. Overall, I would say we're doing fairly well at working through this: but most of the big patches are still ahead of us. ...Robert Ready for Committer (4) === Faster CREATE DATABASE by delaying fsync More frame options in window functions Typed tables libpq new connect function (PQconnectParams) Waiting for Re-Review (5) = Writeable CTEs Listen / Notify rewrite Prevent renaming on multiple inherited column listagg aggregate quoting psql variables (needs reverse review) Waiting for Initial Review (4) == Remove obscure permission checks in FindConversion() Provide rowcount for utility SELECTs knngist (WIP) plpython3 - no reviewer yet Waiting for Updated Patch (2) = Remove gcc dependency in definition of inline functions xpath non-nodeset result enabling Other (5) = Fix large object support in pg_dump - Tom Lane suggested a new approach, nobody has coded it rbtree - Needs both more review and some updates from the author. Add on_perl_init and proper destruction to plperl Add on_trusted_init and on_untrusted_init to plperl Package namespace and Safe init cleanup for plperl - No reviewer listed, but Andrew Dunstan is working on these. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Jan 27, 2010, at 1:00 PM, Joe Conway wrote: Implementing true value_per_call is still something on my TODO list, but obviously has not risen to a very high priority for me as it has now been an embarrassing long time since it was put there. But that said, materialize mode has proven extremely good at covering the most common use cases with acceptable performance. Hrm. I think this has been noted before, but one of the problems with VPC is that there can be a fairly significant amount of overhead involved with context setup and teardown--especially with PLs. If you're streaming millions of rows, it's no longer a small matter. I would think some extension to Tuplestore would be preferable. Where chunks of rows are placed into the Tuplestore on demand in order to minimize context setup/teardown overhead. That is, if the Tuplestore is empty and the user needs more rows, invoke the procedure again with the expectation that it will dump another chunk of rows into the container. Not a formal specification by any means, but I'm curious if anyone has considered that direction. Or along the same lines, how about a valueS-per-call mode? =) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
On Wed, 27 Jan 2010 22:06:43 +0200 Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ivan Sergio Borgonovo wrote: But then... why do we have all that logic to save the function context if anyway it is more convenient to process everything in one run? It's a pain to save the context just to save a pointer inside a structure, it would be more convenient to just process all the structure and return it as a Tuplestore in one pass. When the set-returning-function feature was written originally, years ago, the tuple at a time mode did really work tuple at a time. But it had issues and was axed out of the patch before it was committed, to keep it simple. The idea was to revisit it at some point, but it hasn't bothered anyone enough to fix it. It's basically not implemented yet. Summing it up: 1) tuple at a time theoretically should be better and future proof once someone write the missing code but the code is still not there 2) practically there is no substantial extra cost in returning tuple at a time Is 2) really true? It seems that materialized mode is much simpler. It requires a lot less code and it doesn't force you to save local variables and then restore them every time. So does it still make sense to get an idea about when the returned data set and complexity of computation really fit value_per_call or materialized mode? What could happen between function calls in value_per_call? Would still value_per_call offer a chance to postgresql/OS to better allocate CPU cycles/memory? thanks -- Ivan Sergio Borgonovo http://www.webthatworks.it -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.
On ons, 2010-01-27 at 09:04 -0800, David Fetter wrote: On Wed, Jan 27, 2010 at 04:52:00PM +0200, Peter Eisentraut wrote: On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit hook for CVS that disallows \t in SGML files? The process in git is pretty simple. Better write a check in the makefile. That assumes that everyone will always actually run the appropriate make before committing, which I wouldn't assume, especially for a trivial patch that's going out when someone is pressed for time. Even if you are pressed for time, you are supposed to check your patch. And even then, we have the community and automated build services to pick up after you. Checking in an unavoidable place seems much more foolproof. I'm going to put in an advance vote here against any kind of code-checking commit hook. Because they live in the repository metadata, they can only be seen and changed by the system administrators, they do not travel with the source code, and they are not under version control themselves. The source code and the scripts and tests to manage it should be self-consistent and not dependent on external containers. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 12:08:48PM -0500, Tom Lane wrote: Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 27, 2010 at 11:13:43AM -0500, Tom Lane wrote: (In particular note that shutdown itself might be fired as an atexit callback, a move forced on us by exactly the sort of random user code that you want to add more of. It's not clear whether a Perl-added atexit would fire before or after that.) man atexit says Functions so registered are called in reverse order. Since the plperl atexit is called only when a plperl SP or DO is executed it would fire before any atexit() registered during startup. Right, which means that it would occur either before or after on_proc_exit processing, depending on whether we got there through an exit() call or via the normal proc_exit sequence. That's just the kind of instability I don't want to have to debug. Okay. I could change the callback code to ignore calls if proc_exit_inprogress is false. So an abnormal shutdown via exit() wouldn't involve plperl at all. (Alternatively I could use use on_proc_exit() instead of atexit() to register the callback.) Would that address this call sequence instability issue? The plperl subsystem is isolated from, and can't interact with, the rest of the backend during shutdown. This is exactly the claim that I have zero confidence in. Quite frankly, the problem with Perl as an extension language is that Perl was never designed to be a subsystem: it feels free to mess around with the entire state of the process. We've been burnt multiple times by that even with the limited use we make of Perl now, and these proposed additions are going to make it a lot worse IMO. On Wed, Jan 27, 2010 at 09:53:44AM -0800, David E. Wheeler wrote: Can you provide an example? Such concerns are impossible to address without concrete examples. On Wed, Jan 27, 2010 at 01:08:56PM -0500, Tom Lane wrote: Two examples that I can find in a quick review of our CVS history: perl stomping on the process's setlocale state, and perl stomping on the stdio state (Windows only). Neither of those relate to the actions of perl source code. To address that, instead of calling perl_destruct() to perform a complete destruction I could just execute END blocks and object destructors. That would avoid executing any system-level actions. Do you have any examples of how a user could write code in a plperl END block that would interact with the rest of the backend? Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 11:28:02AM -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Indeed, AFAICS the major *point* of these additions is to allow people to insert unknown other functionality that is likely to interact with the rest of the backend; a prospect that doesn't make me feel better about it. No. The major use case we've seen for END blocks is to allow a profiler to write its data out. That should have zero interaction with the rest of the backend. Really? We've found that gprof, for instance, doesn't exactly have zero interaction with the rest of the backend --- there's actually a couple of different bits in there to help it along, including a behavioral change during shutdown. I rather doubt that Perl profilers would turn out much different. Devel::NYTProf (http://blog.timbunce.org/tag/nytprof/) has zero interaction with the rest of the backend. It works in PostgreSQL 8.4, although greatly handicapped by the lack of END blocks. http://search.cpan.org/perldoc?Devel::NYTProf::PgPLPerl But in any case, I don't believe for a moment that profiling is the only or even the largest use to which people would try to put this. Can you give any examples? Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clustering Docs WAS: Mammoth in Core?
Alvaro Herrera wrote: Greg Smith wrote: Alvaro Herrera wrote: Greg Smith wrote: We've got pgsql-cluster-hackers to discuss this particular area. Huh, is this a new list? It wasn't added to wwwmaster's list of lists, apparently, right? The archives are at http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not listed at http://archives.postgresql.org/ for some reason. Because it wasn't added to the database. It seems we've gotten sloppy about the list creation process (though I admit it doesn't seem to be documented anywhere). If you give me a description for the list I'll add it. Discussion of adding replication and clustering features to PostgreSQL, both inside the database and via integration with additional software. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 1:27 PM, Tim Bunce wrote: Okay. I could change the callback code to ignore calls if proc_exit_inprogress is false. So an abnormal shutdown via exit() wouldn't involve plperl at all. (Alternatively I could use use on_proc_exit() instead of atexit() to register the callback.) Given Tom's hesitace about atexit(), perhaps that would be best. Neither of those relate to the actions of perl source code. To address that, instead of calling perl_destruct() to perform a complete destruction I could just execute END blocks and object destructors. That would avoid executing any system-level actions. Does perl_destruct() execute system-level actions, then? If so, then it seems prudent to either audit such actions or, as you say, call destructors directly. Do you have any examples of how a user could write code in a plperl END block that would interact with the rest of the backend? I appreciate you taking the time to look at ways to address the issues Tom has raised, Tim. Good on you. There is so much benefit to this level of interaction, as shown by the success of mod_perl and other forking environments that support pre-loading code, that I think it'd be extremely valuable to get these features in 9.0. They really allow Perl to be a first-class PL in a way that it wasn't before. So if there is no way other than SPI for Perl code to interact with the backend, and system-level actions in Perl itself are disabled, it seems to me that the major issues are addressed. Am I wrong, Tom? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 4:51 PM, David E. Wheeler da...@kineticode.com wrote: Neither of those relate to the actions of perl source code. To address that, instead of calling perl_destruct() to perform a complete destruction I could just execute END blocks and object destructors. That would avoid executing any system-level actions. Does perl_destruct() execute system-level actions, then? If so, then it seems prudent to either audit such actions or, as you say, call destructors directly. What exactly do we mean by system-level actions? I mean, END blocks can execute arbitrary code ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 1:53 PM, Robert Haas wrote: What exactly do we mean by system-level actions? I mean, END blocks can execute arbitrary code Yeah. In Perl. What part of Perl can access the backend systems without SPI? And that it couldn't do at any other point in runtime? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUG?] strange behavior in ALTER TABLE ... RENAME TO on inherited columns
On Wed, Jan 27, 2010 at 3:42 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jan 27, 2010 at 10:17 AM, KaiGai Kohei kai...@kaigai.gr.jp wrote: (2010/01/27 23:29), Robert Haas wrote: 2010/1/27 KaiGai Koheikai...@ak.jp.nec.com: The attached patch is revised one based on the V3 approach. The only difference from V3 is that it also applies checks on the AT_AlterColumnType option, not only renameatt(). I think I was clear about what the next step was for this patch in my previous email, but let me try again. http://archives.postgresql.org/pgsql-hackers/2010-01/msg02407.php See also Tom's comments here: http://archives.postgresql.org/pgsql-hackers/2010-01/msg00110.php I don't believe that either Tom or I are prepared to commit a patch based on this approach, at least not unless someone makes an attempt to do it the other way and finds an even more serious problem. If you're not interested in rewriting the patch along the lines Tom suggested, then we should just mark this as Returned with Feedback and move on. The V3/V5 patch was the rewritten one based on the Tom's comment, as is. It counts the expected inhcount at the first find_all_inheritors() time at once, and it compares the pg_attribute.attinhcount. (In actually, find_all_inheritors() does not have a capability to count the number of merged from a common origin, so I newly defined the find_all_inheritors_with_inhcount().) Am I missing something? Err... I'm not sure. I thought I understood what the different versions of this patch were doing, but apparently I'm all confused. I'll take another look at this. OK, I took a look at this. It's busted: test=# create table top (a integer); CREATE TABLE test=# create table upper1 () inherits (top); CREATE TABLE test=# create table upper2 () inherits (top); CREATE TABLE test=# create table lower1 () inherits (upper1, upper2); NOTICE: merging multiple inherited definitions of column a CREATE TABLE test=# create table lower2 () inherits (upper1, upper2); NOTICE: merging multiple inherited definitions of column a CREATE TABLE test=# create table spoiler (a integer); CREATE TABLE test=# create table bottom () inherits (lower1, lower2, spoiler); NOTICE: merging multiple inherited definitions of column a NOTICE: merging multiple inherited definitions of column a CREATE TABLE test=# alter table top rename a to b; ALTER TABLE test=# select * from spoiler; ERROR: could not find inherited attribute a of relation bottom Also, there is a compiler warning due to an unused variable that should be fixed. I'll mark this Waiting on Author. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review: listagg aggregate
On Tue, Jan 26, 2010 at 1:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 26, 2010 at 12:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: But what it *produces* is a string. For comparison, the SQL-standard-specified array_agg produces arrays, but what it acts on isn't an array. This point is well-taken, but naming it string_agg() because it produces a string doesn't seem quite descriptive enough. We might someday (if we don't already) have a number of aggregates that produce an output that is a string; we can't name them all by the output type. True, but the same point could be made against array_agg, and that didn't stop the committee from choosing that name. As long as string_agg is the most obvious aggregate-to-string functionality, which ISTM it is, I think it's all right for it to have pride of place in naming. Maybe so, but personally, I'd still prefer something more descriptive. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make everything target
On Wed, Jan 27, 2010 at 10:41:16AM -0500, Andrew Dunstan wrote: I'd like to have an everything target that would build all + html + contrib. And maybe an installcheck-everything target that would run installcheck for src, pl and contrib. Thoughts? +1 on both :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Wed, Jan 27, 2010 at 01:51:47PM -0800, David E. Wheeler wrote:
On Jan 27, 2010, at 1:27 PM, Tim Bunce wrote:
Okay. I could change the callback code to ignore calls if
proc_exit_inprogress is false. So an abnormal shutdown via exit()
wouldn't involve plperl at all. (Alternatively I could use use
on_proc_exit() instead of atexit() to register the callback.)
Given Tom's hesitace about atexit(), perhaps that would be best.
I've a draft patch using !proc_exit_inprogress implemented now
(appended) and I'll test it tomorrow. That was the simplest to do first.
Once I've a reasonable way of testing the exit() and proc_exit() code
paths I'll try using on_proc_exit().
Neither of those relate to the actions of perl source code.
To address that, instead of calling perl_destruct() to perform a
complete destruction I could just execute END blocks and object
destructors. That would avoid executing any system-level actions.
Does perl_destruct() execute system-level actions, then?
Potentially: Kills threads it knows about (should be none), wait for
children (should be none), flushes all open *perl* file handles (should
be none for plperl), tears down PerlIO layers, etc. etc. In practice
none of that should affect the backend, but it's possible, especially
for the Windows port. Since none of that is needed it can be skipped.
If so, then it seems prudent to either audit such actions or, as you
say, call destructors directly.
The patch now just calls END blocks and DESTROY methods.
Do you have any examples of how a user could write code in a plperl END
block that would interact with the rest of the backend?
I appreciate you taking the time to look at ways to address the issues
Tom has raised, Tim. Good on you.
Thanks David. I appreciate the visible support!
Tom and the team set the bar high, rightly, so it's certainly been a
challenging introduction to PostgreSQL development!
Tim.
diff --git a/src/pl/plperl/plperl.c b/src/pl/plperl/plperl.c
index 8315d5a..38f2d35 100644
*** a/src/pl/plperl/plperl.c
--- b/src/pl/plperl/plperl.c
***
*** 27,32
--- 27,33
#include miscadmin.h
#include nodes/makefuncs.h
#include parser/parse_type.h
+ #include storage/ipc.h
#include utils/builtins.h
#include utils/fmgroids.h
#include utils/guc.h
*** _PG_init(void)
*** 281,287
static void
plperl_fini(void)
{
! plperl_ending = true;
plperl_destroy_interp(plperl_trusted_interp);
plperl_destroy_interp(plperl_untrusted_interp);
plperl_destroy_interp(plperl_held_interp);
--- 282,297
static void
plperl_fini(void)
{
! plperl_ending = true; /* disables use of spi_* functions */
!
! /*
!* Only perform perl cleanup if we're exiting cleanly via proc_exit().
!* If proc_exit_inprogress is false then exit() was called directly
!* (because we call atexit() very late, so get called early).
!*/
! if (!proc_exit_inprogress)
! return;
!
plperl_destroy_interp(plperl_trusted_interp);
plperl_destroy_interp(plperl_untrusted_interp);
plperl_destroy_interp(plperl_held_interp);
*** plperl_destroy_interp(PerlInterpreter **
*** 595,602
{
if (interp *interp)
{
! perl_destruct(*interp);
! perl_free(*interp);
*interp = NULL;
}
}
--- 605,640
{
if (interp *interp)
{
! /*
!* Only a very minimal destruction is performed.
!* Just END blocks and object destructors, no system-level
actions.
!* Code code here extracted from perl's perl_destruct().
!*/
!
! /* Run END blocks */
! if (PL_exit_flags PERL_EXIT_DESTRUCT_END) {
! dJMPENV;
! int x = 0;
!
! JMPENV_PUSH(x);
! PERL_UNUSED_VAR(x);
! if (PL_endav !PL_minus_c)
! call_list(PL_scopestack_ix, PL_endav);
! JMPENV_POP;
! }
! LEAVE;
! FREETMPS;
!
! PL_dirty = TRUE;
!
! /* destroy objects - call DESTROY methods */
! if (PL_sv_objcount) {
! Perl_sv_clean_objs(aTHX);
! PL_sv_objcount = 0;
! if (PL_defoutgv !SvREFCNT(PL_defoutgv))
! PL_defoutgv = NULL; /* may have been freed */
! }
!
*interp = NULL;
}
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status summary 2010-01-27
Robert Haas wrote: Some of the perl patches have not yet been reviewed either, but I'm a little less concerned about those because it seems that Andrew is working on those anyway: still, if anyone feels inclined to volunteer, I believe Andrew has said he would appreciate another pair of eyes. I have to be away from base for a couple of weeks starting Sunday in family business. That will impact my ability to deal with those remaining patches. In any case, I would still like more eyes on these last pieces, which are a bit more complex and certainly more controversial than the last few. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make everything target
Peter Eisentraut pete...@gmx.net writes: On ons, 2010-01-27 at 10:41 -0500, Andrew Dunstan wrote: I'd like to have an everything target that would build all + html + contrib. +10 And maybe an installcheck-everything target that would run installcheck for src, pl and contrib. +100 These proposals sound reasonable to me too, but is everything an appropriate target name, or is there some other/better convention? I assume we don't want to mess with the default behavior (I don't want to, anyway). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C function accepting/returning cstring vs. text
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: When the set-returning-function feature was written originally, years ago, the tuple at a time mode did really work tuple at a time. But it had issues and was axed out of the patch before it was committed, to keep it simple. The idea was to revisit it at some point, but it hasn't bothered anyone enough to fix it. It's basically not implemented yet. It depends on call context --- you're speaking of the nodeFunctionScan context, but I believe an SRF called in the select targetlist will operate in tuple-at-a-time mode if the function allows. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make everything target
On Jan 27, 2010, at 2:38 PM, Tom Lane wrote: These proposals sound reasonable to me too, but is everything an appropriate target name, or is there some other/better convention? Oooh, more bike-shedding. make theworld make toutlemonde make myday make lovenotwar Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make everything target
Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On ons, 2010-01-27 at 10:41 -0500, Andrew Dunstan wrote: I'd like to have an everything target that would build all + html + contrib. +10 And maybe an installcheck-everything target that would run installcheck for src, pl and contrib. +100 These proposals sound reasonable to me too, but is everything an appropriate target name, or is there some other/better convention? I'm not invested in the name. Suggestions welcome. I assume we don't want to mess with the default behavior (I don't want to, anyway). Me either, although it's a bit of a pity history has stuck us with the default behaviour. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
David E. Wheeler da...@kineticode.com writes: On Jan 27, 2010, at 1:53 PM, Robert Haas wrote: What exactly do we mean by system-level actions? I mean, END blocks can execute arbitrary code Yeah. In Perl. What part of Perl can access the backend systems without SPI? And that it couldn't do at any other point in runtime? You still aren't letting go of the notion that Perl could only affect the rest of the backend via SPI. The point I'm trying to impress on you is that there are any number of other possible pathways, and that Perl's historical assumption that it owns all resources of the process make those pathways a nontrivial hazard. Anything that Perl does to libc state, open file handles, etc etc carries a high risk of breaking the backend. Now it is certainly true that any such hazards can be created just from use of plperlu (we hope only plperlu, and not plperl ...) today, without any use of the proposed additional features. What is bothering me about these features is that their entire reason for existence is to encourage people to use parts of Perl that have time-extended effects on the process state. That means that (a) the probability of problems goes up substantially, and (b) our ability to fix such problems goes down substantially. Right now, the canonical approach to trying to undo anything bad Perl does is to save/restore process state around a plperl call. If we're trying to support usages in which Perl has time-extended effects on process state, that solution goes out the window, and we have to think of some other way to coexist with Perl. (Where, I note, coexist means Perl does what it damn pleases and we have to pick up the pieces --- we're not likely to get any cooperation on limiting damage from that side. Nobody even suggested that we treat stomping on setlocale state as a Perl bug, for example, rather than a fact of life that we just had to work around however we could.) So the real bottom line here is that I foresee this patch as being destabilizing and requiring us to put large amounts of time into figuring out workarounds for whatever creative things people decide to try to do with Perl. I'd feel better about it if I thought that we could get away with a policy of if it breaks it's your problem, but I do not think that will fly from a PR standpoint. It hasn't in the past. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tim Bunce tim.bu...@pobox.com writes: Okay. I could change the callback code to ignore calls if proc_exit_inprogress is false. So an abnormal shutdown via exit() wouldn't involve plperl at all. (Alternatively I could use use on_proc_exit() instead of atexit() to register the callback.) Use on_proc_exit please. I will continue to object to any attempt to hang arbitrary processing on atexit(). An advantage of on_proc_exit from your end is that it should allow you to not have to try to prevent the END blocks from using SPI, as that would still be perfectly functional when your callback gets called. (Starting a new transaction would be a good idea though, cf Async_UnlistenOnExit.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 3:11 PM, Tom Lane wrote: You still aren't letting go of the notion that Perl could only affect the rest of the backend via SPI. The point I'm trying to impress on you is that there are any number of other possible pathways, and that Perl's historical assumption that it owns all resources of the process make those pathways a nontrivial hazard. Anything that Perl does to libc state, open file handles, etc etc carries a high risk of breaking the backend. As could any other code that executes then, including C libraries installed from pgFoundry and loaded by a DBA. Now it is certainly true that any such hazards can be created just from use of plperlu (we hope only plperlu, and not plperl ...) today, without any use of the proposed additional features. What is bothering me about these features is that their entire reason for existence is to encourage people to use parts of Perl that have time-extended effects on the process state. Well, mainly it's to avoid the overhead of loading the code except at startup. That means that (a) the probability of problems goes up substantially, Why? Arbitrary code can already execute at start time. Is Perl special somehow? and (b) our ability to fix such problems goes down substantially. Why is it your problem? Right now, the canonical approach to trying to undo anything bad Perl does is to save/restore process state around a plperl call. If we're trying to support usages in which Perl has time-extended effects on process state, that solution goes out the window, and we have to think of some other way to coexist with Perl. (Where, I note, coexist means Perl does what it damn pleases and we have to pick up the pieces --- we're not likely to get any cooperation on limiting damage from that side. Nobody even suggested that we treat stomping on setlocale state as a Perl bug, for example, rather than a fact of life that we just had to work around however we could.) How is that different from any other code that gets loaded when the server starts, exactly? Do, however, feel free to report Perl bugs. Just run `perlbug`. So the real bottom line here is that I foresee this patch as being destabilizing and requiring us to put large amounts of time into figuring out workarounds for whatever creative things people decide to try to do with Perl. I'd feel better about it if I thought that we could get away with a policy of if it breaks it's your problem, but I do not think that will fly from a PR standpoint. It hasn't in the past. mod_perl has for many years. Provide lots of caveats in the documentation. Point users to it when they write in about a problem. Truth is, the vast majority of Perl modules are pretty well-behaved. I sincerely doubt you'd hear much complaint. Have the Apache guys had to take any special steps to protect httpd from mod_perl? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 27, 2010 at 11:28:02AM -0500, Tom Lane wrote: Really? We've found that gprof, for instance, doesn't exactly have zero interaction with the rest of the backend --- there's actually a couple of different bits in there to help it along, including a behavioral change during shutdown. I rather doubt that Perl profilers would turn out much different. Devel::NYTProf (http://blog.timbunce.org/tag/nytprof/) has zero interaction with the rest of the backend. I don't have to read any further than the place where it says doesn't work if you call both plperl and plperlu to realize that that's quite false. Maybe we have different definitions of what a software interaction is... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
On Jan 27, 2010, at 3:33 PM, Tom Lane wrote: I don't have to read any further than the place where it says doesn't work if you call both plperl and plperlu to realize that that's quite false. Maybe we have different definitions of what a software interaction is... I think that dates from when plperl and plperlu couldn't co-exists, which was fixed a few months ago, n'est pas? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add on_perl_init and proper destruction to plperl [PATCH]
David E. Wheeler da...@kineticode.com writes: On Jan 27, 2010, at 3:33 PM, Tom Lane wrote: I don't have to read any further than the place where it says doesn't work if you call both plperl and plperlu to realize that that's quite false. Maybe we have different definitions of what a software interaction is... I think that dates from when plperl and plperlu couldn't co-exists, which was fixed a few months ago, n'est pas? No, that was fixed years ago, at least if you have a modern Perl build that supports multiplicity at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
