Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
Christian Ullrich ch...@chrullrich.net writes: * Peter Eisentraut wrote: On 4/30/15 2:49 PM, Andrew Dunstan wrote: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. I can reproduce it. I'll look into it. I looked over the CCA animals and noticed that pademelon and gaur are apparently unaffected; pademelon and gaur do not run CCA (if they did, it would take weeks for a run to complete :-(). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
* Peter Eisentraut wrote: On 4/30/15 2:49 PM, Andrew Dunstan wrote: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. I can reproduce it. I'll look into it. I looked over the CCA animals and noticed that pademelon and gaur are apparently unaffected; what they have in common is the OS (HP-UX) and the Python version (2.5). There's nothing I can do about OS-related differences, but I thought I'd check the Python angle. With Python 2.5.6, jaguarundi locks up on the plpython tests just the same as with 3.4, and friarbird with 2.7. So that is not the culprit, either. I ran make check by hand, and noticed three tests where it seemed to hang (I gave it at least three minutes each, and the functions in the queries are simple): plpython_spiSELECT cursor_plan(); plpython_setof SELECT test_setof_as_list(1, 'list'); plpython_composite SELECT multiout_simple_setof(); These are the only plpython tests that mention SETOF at all, and the queries that hung are the first ones in their respective tests to actually build a set. Does that help? -- Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
* Tom Lane wrote: Christian Ullrich ch...@chrullrich.net writes: * Peter Eisentraut wrote: On 4/30/15 2:49 PM, Andrew Dunstan wrote: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. I can reproduce it. I'll look into it. I looked over the CCA animals and noticed that pademelon and gaur are apparently unaffected; pademelon and gaur do not run CCA (if they did, it would take weeks for a run to complete :-(). Ah. I obviously had associated the note icon with CCA. Sorry. OTOH, if I had noticed that, I might not have gone into more detail ... -- Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Transforms patch not respecting if indentation
On Mon, May 4, 2015 at 6:45 AM, Michael Paquier michael.paqu...@gmail.com
wrote:
Hi all,
Coverity complained about a small indentation issue in ruleutils.c:
+ appendStringInfoString(buf, \n TRANSFORM );
+ for (i = 0; i ntypes; i++)
+ {
+ if (i != 0)
+ appendStringInfoString(buf, , );
+ appendStringInfo(buf, FOR TYPE %s,
format_type_be(trftypes[i]));
+ }
Applied, thanks.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
[HACKERS] Transforms patch not respecting if indentation
Hi all,
Coverity complained about a small indentation issue in ruleutils.c:
+ appendStringInfoString(buf, \n TRANSFORM );
+ for (i = 0; i ntypes; i++)
+ {
+ if (i != 0)
+ appendStringInfoString(buf, , );
+ appendStringInfo(buf, FOR TYPE %s,
format_type_be(trftypes[i]));
+ }
Regards,
--
Michael
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index fea8db6..69267bd 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -2364,7 +2364,7 @@ print_function_trftypes(StringInfo buf, HeapTuple proctup)
{
if (i != 0)
appendStringInfoString(buf, , );
-appendStringInfo(buf, FOR TYPE %s, format_type_be(trftypes[i]));
+ appendStringInfo(buf, FOR TYPE %s, format_type_be(trftypes[i]));
}
}
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
On 05/01/2015 08:57 AM, Andrew Dunstan wrote: On 04/30/2015 09:09 PM, Christian Ullrich wrote: * Andrew Dunstan: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. Here's what it's been doing: query| SELECT cursor_plan(); Same here, on jaguarundi. I actually killed it intentionally this morning, hoping that whatever the problem was might have been fixed already. No such luck. I would suspect that it might have something to do with the OS, if all the other CCA animals weren't lining up nicely behind in the buildfarm status page. Yeah, this happened again this morning, so it seems to be quite reliably reproducible. I killed it and I've set friarbird to build without python for now, but this is clearly an issue that needs to be resolved. And I have confirmed that it's not an OS problem - I have reproduced it on a modern Linux instance (it's still running, in fact). So it's quite clearly a bug that needs to be fixed. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
On 04/30/2015 09:09 PM, Christian Ullrich wrote: * Andrew Dunstan: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. Here's what it's been doing: query| SELECT cursor_plan(); Same here, on jaguarundi. I actually killed it intentionally this morning, hoping that whatever the problem was might have been fixed already. No such luck. I would suspect that it might have something to do with the OS, if all the other CCA animals weren't lining up nicely behind in the buildfarm status page. I imagine it was in some sort of infinite loop. gdb says it's all in src/backend/utils/cache/plancache.c, although not the same line each time I run it. I ktrace'd it this morning, but cleverly did not keep the dump. It looked much the same to me, though, it was reading the same filenode over and over again. Yeah, this happened again this morning, so it seems to be quite reliably reproducible. I killed it and I've set friarbird to build without python for now, but this is clearly an issue that needs to be resolved. Side thought - maybe we need some sort of timeout mechanism for the buildfarm to try to stop it from hanging. There is actually some timeout code in there from back in the CVS days when occasionally CVS would hang. It could be adapted to timeout other steps. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
* Andrew Dunstan: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. Here's what it's been doing: query| SELECT cursor_plan(); Same here, on jaguarundi. I actually killed it intentionally this morning, hoping that whatever the problem was might have been fixed already. No such luck. I would suspect that it might have something to do with the OS, if all the other CCA animals weren't lining up nicely behind in the buildfarm status page. I imagine it was in some sort of infinite loop. gdb says it's all in src/backend/utils/cache/plancache.c, although not the same line each time I run it. I ktrace'd it this morning, but cleverly did not keep the dump. It looked much the same to me, though, it was reading the same filenode over and over again. -- Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS
On 4/30/15 2:49 PM, Andrew Dunstan wrote: friarbird is a FreeBSD buildfarm animal running with -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours. However, it's been stuck since Monday running the plpython regression tests. The only relevant commit seems to be the transforms feature. I can reproduce it. I'll look into it. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 6/12/13 1:20 AM, Josh Berkus wrote: Peter, All: Does anyone feel like fixing the LOAD issue with transforms? I haven't seen any activity on the patch. I plan to send in an updated patch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter, All: Does anyone feel like fixing the LOAD issue with transforms? I haven't seen any activity on the patch. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 03/13/2013 09:54 AM, Dimitri Fontaine wrote: Peter Eisentraut pete...@gmx.net writes: At run time, this will sort itself out, because all the required modules will be loaded. The problem is when you create the glue extension and haven't invoked any code from any of the dependent extension in the session. Abstractly, the possible solutions are either not to check the functions when the extension is created (possibly settable by a flag) or to somehow force a load of all dependent extensions when the new extension is created. (I say extension here even though dynamically loadable modules are attached to functions, which makes this even more confusing.) I think CREATE EXTENSION should be LOADing all distinct modules referenced from all required extensions functions. It does sound easy enough to code, and I can't imagine why we would want to not do that. Where are we with this? Will the loading issue be fixed, or should we bump this feature to 9.4? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter Eisentraut pete...@gmx.net writes: At run time, this will sort itself out, because all the required modules will be loaded. The problem is when you create the glue extension and haven't invoked any code from any of the dependent extension in the session. Abstractly, the possible solutions are either not to check the functions when the extension is created (possibly settable by a flag) or to somehow force a load of all dependent extensions when the new extension is created. (I say extension here even though dynamically loadable modules are attached to functions, which makes this even more confusing.) I think CREATE EXTENSION should be LOADing all distinct modules referenced from all required extensions functions. It does sound easy enough to code, and I can't imagine why we would want to not do that. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 2013-03-11 20:28:05 -0400, Peter Eisentraut wrote: On Mon, 2013-03-11 at 18:11 +0100, Andres Freund wrote: If we don't find a better solution, yes. Why don't we lookup type input/ouput function for parameters and return type during CREATE FUNCTION? That should solve the issue in a neater way? A function in general has no particular use for a type's input or output function. Also, a type's input/output functions are not necessarily in the same library as other things about that type that you might want or need. In theory they are not necessarily tied together, yes. In practice they nearly always are in the same shared object. The lookup for them afaics are the only reason why e.g. the plperl-hstore transform works outside of CREATE EXTENSION without explictly doing something ugly like: On 03/11/2013 09:50 AM, Andres Freund wrote: DO LANGUAGE plperlu ; SELECT NULL::hstore; beforehand. If you have a better idea than putting the above into the extension script or making a function look up its parameters, I am all ears. It sure would only be a hack. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Your error looks like youre erroring out in plperl not in hstore? Look again. Peter, is there any way for you to tackle this issue? I have no idea how to fix it, myself ... -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 2013-03-11 09:42:18 -0700, Josh Berkus wrote: Your error looks like youre erroring out in plperl not in hstore? Look again. ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key Thats a perl symbol. Peter, is there any way for you to tackle this issue? I have no idea how to fix it, myself ... If you add a: DO LANGUAGE plperlu ; SELECT NULL::hstore; to the extension script, does it work? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 03/11/2013 09:50 AM, Andres Freund wrote: DO LANGUAGE plperlu ; SELECT NULL::hstore; Aha, so that's what you meant! Yeah, it works if I reference both extensions before the CREATE EXTENSION. In that case, seems like that could be added to the extension SQL, no? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 2013-03-11 09:55:34 -0700, Josh Berkus wrote: On 03/11/2013 09:50 AM, Andres Freund wrote: DO LANGUAGE plperlu ; SELECT NULL::hstore; Aha, so that's what you meant! Yeah, it works if I reference both extensions before the CREATE EXTENSION. In that case, seems like that could be added to the extension SQL, no? If we don't find a better solution, yes. Why don't we lookup type input/ouput function for parameters and return type during CREATE FUNCTION? That should solve the issue in a neater way? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On Mon, 2013-03-11 at 18:11 +0100, Andres Freund wrote: If we don't find a better solution, yes. Why don't we lookup type input/ouput function for parameters and return type during CREATE FUNCTION? That should solve the issue in a neater way? A function in general has no particular use for a type's input or output function. Also, a type's input/output functions are not necessarily in the same library as other things about that type that you might want or need. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 3/5/13 5:52 PM, Josh Berkus wrote: More on this: the problem appears to be that the symbols for hstore are loaded only if I've just just created the extension in that database: I see. This is a problem with any kind of dynamically loadable module that uses another module. The other module is only loaded either at creation time or when a function from it is first used (or if a preload mechanism is used). At run time, this will sort itself out, because all the required modules will be loaded. The problem is when you create the glue extension and haven't invoked any code from any of the dependent extension in the session. Abstractly, the possible solutions are either not to check the functions when the extension is created (possibly settable by a flag) or to somehow force a load of all dependent extensions when the new extension is created. (I say extension here even though dynamically loadable modules are attached to functions, which makes this even more confusing.) In normal programming languages, this is normally addressed by placing explicit load/require/import statements before you do anything else. What we are doing here is more like an autoload functionality that some environments have. Those have the same problem. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter, At run time, this will sort itself out, because all the required modules will be loaded. The problem is when you create the glue extension and haven't invoked any code from any of the dependent extension in the session. Just invoking code doesn't seem to be enough. I tried just using the Hstore data type, and then loading hstore_plperl, but that still didn't work. It seems like only CREATE EXTENSION loads *all* the symbols. Abstractly, the possible solutions are either not to check the functions when the extension is created (possibly settable by a flag) or to somehow force a load of all dependent extensions when the new extension is created. The latter would be ideal, but I don't know that we currently have any mechanism for it. --Josh -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 2013-03-06 09:53:29 -0800, Josh Berkus wrote: Peter, At run time, this will sort itself out, because all the required modules will be loaded. The problem is when you create the glue extension and haven't invoked any code from any of the dependent extension in the session. Just invoking code doesn't seem to be enough. I tried just using the Hstore data type, and then loading hstore_plperl, but that still didn't work. It seems like only CREATE EXTENSION loads *all* the symbols. Your error looks like youre erroring out in plperl not in hstore? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 13-03-03 08:13 PM, Josh Berkus wrote: This (creating the extensions) works fine for me on a Ubuntu 10.x system Now if only we can work out the combinatorics issue ... The plpython2u extensions worked fine but I haven't been able to get this to work with plpython3u (python 3.1). create extension hstore_plpython3u; ERROR: could not load library /usr/local/pgsql93git/lib/hstore_plpython3.so: /usr/local/pgsql93git/lib/hstore_plpython3.so: undefined symbol: _Py_NoneStruct Peter mentioned in the submission that the unit tests don't pass with python3, it doesn't work for meoutside the tests either. Steve -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter mentioned in the submission that the unit tests don't pass with python3, it doesn't work for meoutside the tests either. Also, note that the feature is the concept of Transforms, not the individual transforms which Peter provides. The idea is to enable a new kind of extension. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 3/3/13 6:14 PM, Josh Berkus wrote:
Currently Peter is punting (as is proper in a new patch) by having a
separate extension for each combination (hstore/plperl, hstore/plpython,
ltree/plpython, etc.). This is obviously not a maintainable approach in
the long run.
There is also a {Perl, Python, Ruby, etc.} binding for {libpq, libmysql,
libpng, libyaml, libssl, libgmp, etc.}, each as a separately
downloadable and buildable package. I don't think anyone has ever
seriously considered a system by which if, say, you have Python and
libyaml installed, pyyaml magically appears. Might be nice, but maybe
not. The solution, in practice, is that you download pyyaml, and it
pulls in any required dependencies. This would work the same way.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter,
There is also a {Perl, Python, Ruby, etc.} binding for {libpq, libmysql,
libpng, libyaml, libssl, libgmp, etc.}, each as a separately
downloadable and buildable package. I don't think anyone has ever
seriously considered a system by which if, say, you have Python and
libyaml installed, pyyaml magically appears. Might be nice, but maybe
not. The solution, in practice, is that you download pyyaml, and it
pulls in any required dependencies. This would work the same way.
That would be good too, although we don't currently have that
capability; if I try to install hstore_plpython without plpython
installed, it just errors out. Aside from that, what can we reasonably
do for 9.3 to get this feature in?
Maybe we add a transforms/ subdirectory of contrib, so that it can be as
crowded as we want? Or put the transforms on PGXN for now?
I want to see this feature go in so that the community starts writing
transforms this year instead of next year.
BTW, dependancies seem to be working OK for DROP:
postgres=# drop extension plpythonu;
ERROR: cannot drop extension plpythonu because other objects depend on it
DETAIL: extension hstore_plpythonu depends on extension plpythonu
function look_up_hstore(hstore,text) depends on transform for hstore
language plpythonu
extension ltree_plpythonu depends on extension plpythonu
HINT: Use DROP ... CASCADE to drop the dependent objects too.
STATEMENT: drop extension plpythonu;
ERROR: cannot drop extension plpythonu because other objects depend on it
DETAIL: extension hstore_plpythonu depends on extension plpythonu
function look_up_hstore(hstore,text) depends on transform for hstore
language plpythonu
extension ltree_plpythonu depends on extension plpythonu
HINT: Use DROP ... CASCADE to drop the dependent objects too.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter, I'm still getting intermittent linking failures: postgres=# drop extension plperl cascade; NOTICE: drop cascades to extension hstore_plperl DROP EXTENSION postgres=# create extension plperl; CREATE EXTENSION postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs STATEMENT: create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs postgres=# There appears to be something wonky which breaks when I've been running 9.2, shut it down, and fire up 9.3. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
postgres=# create extension plperl; CREATE EXTENSION postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs STATEMENT: create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs postgres=# There appears to be something wonky which breaks when I've been running 9.2, shut it down, and fire up 9.3. More on this: the problem appears to be that the symbols for hstore are loaded only if I've just just created the extension in that database: postgres=# create database plperlh postgres-# ; CREATE DATABASE postgres=# \c plperlh; You are now connected to database plperlh as user josh. plperlh=# create extension plperl; CREATE EXTENSION plperlh=# create extension hstore; CREATE EXTENSION plperlh=# create extension hstore_plperl; CREATE EXTENSION plperlh=# plperlh=# \c postgres You are now connected to database postgres as user josh. postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key STATEMENT: create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 03/05/2013 02:52 PM, Josh Berkus wrote: plperlh=# \c postgres You are now connected to database postgres as user josh. postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key STATEMENT: create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key What happens if you set your LD_LIBRARY_PATH to reflect each installation before you start the database? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
What happens if you set your LD_LIBRARY_PATH to reflect each installation before you start the database? No change, at least not setting it to $PGHOME/lib. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter, I tried this patch out. I didn't get as far as testing the functionality because of errors. configure/make/make install/make check worked, without asserts. I believe DF found some errors when he enabled assertions. When I tried to install the actual transform extensions, though, it blew up with symbol errors: transforms=# create extension hstore transforms-# ; CREATE EXTENSION transforms=# create extension ltree transforms-# ; CREATE EXTENSION transforms=# create extension plperl transforms-# ; CREATE EXTENSION transforms=# create extension plpythonu transforms-# ; CREATE EXTENSION transforms=# create extension ltree_plpythonu; ERROR: could not load library /home/josh/pg93/lib/postgresql/ltree_plpython2.so: /home/josh/pg93/lib/postgresql/ltree_plpython2.so: undefined symbol: PyList_New STATEMENT: create extension ltree_plpythonu; transforms=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs STATEMENT: create extension hstore_plperl; This surprised me, because make check for the extensions passed fine. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Peter, all: So in addition to the bugs I encountered in getting this patch to work, we have a design issue to work out: how to load all of the transform functions. Each transform depends on an optional datatype (like hstore) and an optional external language (like plperl), which can be loaded into the database in any order. Currently Peter is punting (as is proper in a new patch) by having a separate extension for each combination (hstore/plperl, hstore/plpython, ltree/plpython, etc.). This is obviously not a maintainable approach in the long run. In an ideal world, transformations would load automatically as soon as all of their prerequisites load. For example, if you load hstore and plpythonU, then hstore_plpython should automatically load whenever the second extension is loaded. However, we currently have no mechanism for this, especially since we don't know what order the user will load the two extensions in. Realistically, this would require enhancing the extension mechanism to track transformations. Discuss? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
transforms=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs STATEMENT: create extension hstore_plperl; This surprised me, because make check for the extensions passed fine. Oh, this is on Ubuntu 12.10, not OSX. So possibly the fixes you made to fix linking on OSX broke other platforms. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 13-03-03 06:15 PM, Josh Berkus wrote: transforms=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: hstoreUniquePairs STATEMENT: create extension hstore_plperl; This surprised me, because make check for the extensions passed fine. Oh, this is on Ubuntu 12.10, not OSX. So possibly the fixes you made to fix linking on OSX broke other platforms. This (creating the extensions) works fine for me on a Ubuntu 10.x system template1=# create database test; CREATE DATABASE template1=# \c test You are now connected to database test as user ssinger. test=# create extension hstore; CREATE EXTENSION test=# create extension hstore_plpythonu; ERROR: required extension plpythonu is not installed STATEMENT: create extension hstore_plpythonu; ERROR: required extension plpythonu is not installed test=# create extension plpythonu; CREATE EXTENSION test=# create extension hstore_plpythonu; CREATE EXTENSION test=# test=# create extension plperl; CREATE EXTENSION test=# create extension hstore_plperl; CREATE EXTENSION test=# create extension plperlu; CREATE EXTENSION test=# create extension hstore_plperlu; CREATE EXTENSION test=# -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
This (creating the extensions) works fine for me on a Ubuntu 10.x system
H. So I wiped everything out and retried this with clean
directories, and it worked fine. Now I'm not sure how I got the error
in the first place.
Anyway, the feature seems to work as expected:
create function look_up_hstore (
some_hs hstore, which_key text )
returns text
language plpythonu as $f$
return some_hs[which_key]
$f$;
postgres=# create table storit ( it hstore );
CREATE TABLE
postgres=# insert into storit values ( 'f = josh,l=berkus'
),('f=jan,l=wieck'),('f=tom,l=lane');
INSERT 0 3
postgres=# select look_up_hstore(it,'f') from storit;
look_up_hstore
josh
jan
tom
(3 rows)
I'll have to think of other ways to test it out, but right now it seems
to pass muster. I was a little unclear on what Python data structure an
ltree should produce.
Now if only we can work out the combinatorics issue ...
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
Here is an updated patch for the transforms feature. The previous discussion was here: http://www.postgresql.org/message-id/1339713732.11971.79.ca...@vanquo.pezone.net Old news: At the surface, this contains: - New catalog pg_transform - CREATE/DROP TRANSFORM As proofs of concept and useful applications, I have included transforms for - PL/Perl - hstore - PL/Python - hstore - PL/Python - ltree New news: I have tried to address all issues raised during previous reviews. The regression tests for hstore_plpython and ltree_plpython don't pass with Python 3. This needs to be fixed, obviously, but it's an issue unrelated to the core functionality. As a demo that this can be used externally as well, I have written a proof-of-concept transform between plpython and the mpz type in the pgmp extension (multi-precision arithmetic extension): https://github.com/petere/pgmp/tree/transforms/plpython Note: There was a lot of churn lately in src/backend/commands/, and I had to fix merge conflicts several times, so you need source as of about right now to apply this patch cleanly. Side issue/peculiarity: I had to change some shared library linking flags for OS X. Ordinarily, when linking a dynamically loadable module on OS X, if there are unresolved symbols, it's an error. But for example, when we link hstore_plpython, we expect some symbols to be resolved in hstore.so or plpythonu.so, so we need to ignore unresolved symbols. This change basically just makes OS X behave like other platforms in this regard. There might be other portability issues on other platforms. pg-transforms-20130115.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
I haven't had the time to finish all the issues with this, but I want to keep the discussion going in the meantime and provide an updated patch. On mån, 2012-06-18 at 17:33 +0200, Andres Freund wrote: Cursory code review: * pstrndup already exists as pnstrdup (hstore_plperl.c) Fixed. * PyString_FromStringAndSize return value not decreffed? PyDict_SetItem doesn't steal references Fixed. * In plpython_to_hstore I would move the 'pairs' and some related variables in the PG_TRY block, so the reader doesn't need to check whether it should be volatile * In the same function items needs to be volatile to fit into longjmp semantics I'll recheck that later. * I don't think recording dependencies on transforms used when creating functions is a good idea as the transform might get created after the functions already exists. That seems to be a pretty confusing behaviour. We need the dependencies, because otherwise dropping a transform would break or silently alter the behavior of functions that depend on it. That sounds like my worst nightmare, thinking of some applications that would be affected by that. But your point is a good one. I think this could be addressed by prohibiting the creation of a transform that affects functions that already exist. Because the legacy behavior of PL implementations of defaulting to a string representation conversion, we would technically need a dependency on the absence of a transform object to make this airtight. In the far future, I could imagine removing this default behavior, meaning you couldn't create the function if no suitable transforms exist for all argument and return types. * I forsee the need for multiple transforms for the same type/language pair to coexist. The user would need to manually choose/call the transform in that case. This currently isn't easily possible... I thought about this briefly at the beginning, but see under worst nightmare above. Also, having a configuration setting for this or something would prevent any PL functions from being immutable. We don't allow multiple casts or multiple in/out functions either, which are related concepts. If you want different behavior, you should define a different type or different language. *) No psql backslash commands yet. Could be added. Doesn't really seem necessary to me. Not many people will need to look at this and the list of commands already is rather long. I'm going to leave this out for now. *) Permissions: Transforms don't have owners, a bit like casts. Currently, you are allowed to drop a transform if you own both the type and the language. That might be too strict, maybe own the type and have privileges on the language would be enough. Seems sensible enough to me. I have made this change. *) There is currently some syntax schizophrenia. The grammar accepts CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl, TO SQL WITH plperl_to_hstore ); but pg_dump produces CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl(hstore), TO SQL WITH plperl_to_hstore(internal) ); The SQL standard allows both. (In the same way that it allows 'DROP FUNCTION foo' without arguments, if it is not ambigious.) Precedent is that CREATE CAST requires arguments, but CREATE LANGUAGE does not. I don't find that problematic personally. I have fixed the syntax to include argument types, so the dump output and the input grammar is consistent. Other changes: - Fixed ecpg grammar to work again with this. - Changed extension naming to be more consistent. - Build additional contrib modules conditionally depending on whether --with-perl or --with-python were configured. (complaint from Jeff Janes) - Fixed Python 3. Things I still want to do: - Refactor the regression test framework for Python 3 so that contrib modules or external extensions don't have to repeat the magic in src/pl/plpython/Makefile. (Python 3 with hstore_plpython and ltree_plpython works, but the tests don't run.) - Refactor pyobject_to_string(), which is currently kind of copied and pasted from plpython, but should instead be exported by plpython in some suitable way. - Refactor shared library building so that I can have, say, hstore, hstore_plperl, and hstore_plpython in one directory, rather than in three. The reason being, if someone has a new type in a repository on github or something, I don't want them to have to make three separate projects or some crazy subdirectory structure in order to add some PL support for their type. This will require some deep Makefile.shlib hacking, but I think it's worth trying to make this simple(r). So, it's quite likely that this patch won't get finished in this commit fest. transforms-20120707.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your
Re: [HACKERS] transforms
Hi Peter, On Friday, June 15, 2012 12:42:12 AM Peter Eisentraut wrote: Here is my first patch for the transforms feature. This is a mechanism to adapt data types to procedural languages. The previous proposal was here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php At the surface, this contains: - New catalog pg_transform - CREATE/DROP TRANSFORM Cursory code review: * pstrndup already exists as pnstrdup (hstore_plperl.c) * PyString_FromStringAndSize return value not decreffed? PyDict_SetItem doesn't steal references * In plpython_to_hstore I would move the 'pairs' and some related variables in the PG_TRY block, so the reader doesn't need to check whether it should be volatile * In the same function items needs to be volatile to fit into longjmp semantics * I don't think recording dependencies on transforms used when creating functions is a good idea as the transform might get created after the functions already exists. That seems to be a pretty confusing behaviour. * I am a bit wary that some place can be used to call functions accepting INTERNAL indirectly, couldn't think of any immediately though. Will look into this a bit, but I am not experienced in the area, so ... * I forsee the need for multiple transforms for the same type/language pair to coexist. The user would need to manually choose/call the transform in that case. This currently isn't easily possible... *) No psql backslash commands yet. Could be added. Doesn't really seem necessary to me. Not many people will need to look at this and the list of commands already is rather long. *) Permissions: Transforms don't have owners, a bit like casts. Currently, you are allowed to drop a transform if you own both the type and the language. That might be too strict, maybe own the type and have privileges on the language would be enough. Seems sensible enough to me. *) If we want to offer the ability to write transforms to end users, then we need to install all the header files for the languages and the types. This actually worked quite well; including hstore.h and plperl.h for example, gave you what you needed. In other cases, some headers might need cleaning up. Also, some magic from the plperl and plpython build systems might need to be exposed, for example to find the header files. See existing modules for how this currently looks. Doesn't look to bad to me. I dont't know how this could be made easier. *) There is currently some syntax schizophrenia. The grammar accepts CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl, TO SQL WITH plperl_to_hstore ); but pg_dump produces CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl(hstore), TO SQL WITH plperl_to_hstore(internal) ); The SQL standard allows both. (In the same way that it allows 'DROP FUNCTION foo' without arguments, if it is not ambigious.) Precedent is that CREATE CAST requires arguments, but CREATE LANGUAGE does not. I don't find that problematic personally. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On Thu, Jun 14, 2012 at 3:42 PM, Peter Eisentraut pete...@gmx.net wrote: Here is my first patch for the transforms feature. This is a mechanism to adapt data types to procedural languages. The previous proposal was here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php When I apply this and go to contrib and do make check, I get: In file included from hstore_plperl.c:4:0: ../../src/pl/plperl/plperl.h:49:20: fatal error: EXTERN.h: No such file or directory Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On Sat, Jun 16, 2012 at 7:15 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Jun 14, 2012 at 3:42 PM, Peter Eisentraut pete...@gmx.net wrote: Here is my first patch for the transforms feature. This is a mechanism to adapt data types to procedural languages. The previous proposal was here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php When I apply this and go to contrib and do make check, I get: In file included from hstore_plperl.c:4:0: ../../src/pl/plperl/plperl.h:49:20: fatal error: EXTERN.h: No such file or directory Ah, that went away when I remembered to ./configure --with-perl Although the error message seem less than optimal. Aren't test usually skipped when they are missing prerequisites in the config? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] transforms
Here is my first patch for the transforms feature. This is a mechanism to adapt data types to procedural languages. The previous proposal was here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php At the surface, this contains: - New catalog pg_transform - CREATE/DROP TRANSFORM As proof of concepts and hopefully useful applications to-be, I have included transforms for - PL/Perl - hstore - PL/Python - hstore - PL/Python - ltree These, along with the documentation for the above-mentioned DDL commands and catalog can serve as explanations for reviewers. There are numerous remaining discussion points and possible work items: *) Currently, each of the above provided transforms is put into a separate subdirectory under contrib. This might be OK, but it is mainly because we cannot build more than one MODULE_big per directory. Fixing that might be more work than it's worth, but if we want to provide more of these in contrib, it might get crowded. *) Since we have separate extensions for plperl and plperlu, and also for plpython2u and plpython3u, we need one extension for adapting each of these to a given type. You can see under contrib/hstore_plperl what this looks like. Maybe this isn't fixable or worth fixing. (With the directory and packaging not finalized, I haven't included any documentation for these contrib modules yet.) *) No psql backslash commands yet. Could be added. *) Permissions: Transforms don't have owners, a bit like casts. Currently, you are allowed to drop a transform if you own both the type and the language. That might be too strict, maybe own the type and have privileges on the language would be enough. *) If we want to offer the ability to write transforms to end users, then we need to install all the header files for the languages and the types. This actually worked quite well; including hstore.h and plperl.h for example, gave you what you needed. In other cases, some headers might need cleaning up. Also, some magic from the plperl and plpython build systems might need to be exposed, for example to find the header files. See existing modules for how this currently looks. *) There is currently some syntax schizophrenia. The grammar accepts CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl, TO SQL WITH plperl_to_hstore ); but pg_dump produces CREATE TRANSFORM FOR hstore LANGUAGE plperl ( FROM SQL WITH hstore_to_plperl(hstore), TO SQL WITH plperl_to_hstore(internal) ); The SQL standard allows both. (In the same way that it allows 'DROP FUNCTION foo' without arguments, if it is not ambigious.) Precedent is that CREATE CAST requires arguments, but CREATE LANGUAGE does not. *) The issue of how to handle arguments of type internal was already discussed at http://archives.postgresql.org/pgsql-hackers/2012-05/msg00857.php and following. I have adopted the suggestion there to prohibit calling functions with arguments of type internal, but that is probably full of holes and needs to be reviewed carefully. *) I'm not very familiar with the Perl C API, so the hstore_plperl implementation is probably full of memory leaks and weirdnesses. It needs some help from a PL/Perl expert. (The PL/Python stuff is hopefully better.) *) ltree_plpython lacks the transform function from python to ltree, because of laziness and lack of clear documentation on how to construct ltree values. *) I just noticed that the grammar changes broke ECPG. I'll look into that. *) The regression tests for the core CREATE/DROP TRANSFORM functionality is in contrib/hstore_plpython, because the core has no suitable language available. It's a bit ugly, but I don't know of a better way short of implementing a fake language for regression testing only. transforms-20120615.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
