Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Wed, 2009-12-30 at 00:00 -0500, Robert Haas wrote: > I generally agree with those > ideas, although I think that trying to make the existing aggregate > interface serve this purpose will probably turn out to be trying to > make a square peg fit a round hole. I didn't think that they intended to

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Wed, Dec 30, 2009 at 12:56 AM, Daniel Farina wrote: > On Tue, Dec 29, 2009 at 9:46 PM, Robert Haas wrote: >> Sure.  If you think you can make it work using bytea, that seems >> clearly better than using an internal type, all things being equal. > > I think both are perfectly viable and can be

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 9:46 PM, Robert Haas wrote: > Sure.  If you think you can make it work using bytea, that seems > clearly better than using an internal type, all things being equal. I think both are perfectly viable and can be supported simultaneously, actually...I simply like INTERNAL bec

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Wed, Dec 30, 2009 at 12:26 AM, Daniel Farina wrote: > I think you may be right about this, so we could basically move most > things from the COPY option list into a new DDL option list...I am > more ambivalent, but it seems that would require a catalog, and so on, > which is why I did not leap

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 9:11 PM, Robert Haas wrote: > I think we should try to put an interface layer in place in the first > iteration.  I don't really see this as having much value without that. >  If we implement this strictly as a series of COPY options, we're > going to be committed to suppor

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 11:47 PM, Daniel Farina wrote: > On Tue, Dec 29, 2009 at 8:11 PM, Robert Haas wrote: >> It might be possible to do this without introducing a notion of an >> explicit object, but there are a couple of problems with that.  First, >> it requires the user to specify a lot of

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 11:44 PM, Jeff Davis wrote: > On Tue, 2009-12-29 at 23:11 -0500, Robert Haas wrote: >> I fear that to make this really useful we would need to define some >> new SQL syntax, like: >> >> CREATE [OR REPLACE] COPY TARGET name (STARTUP function_name, STREAM >> function_name, SH

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 8:11 PM, Robert Haas wrote: > It might be possible to do this without introducing a notion of an > explicit object, but there are a couple of problems with that.  First, > it requires the user to specify a lot of details on every COPY > invocation, which is awkward.  Second

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Tue, 2009-12-29 at 23:11 -0500, Robert Haas wrote: > I fear that to make this really useful we would need to define some > new SQL syntax, like: > > CREATE [OR REPLACE] COPY TARGET name (STARTUP function_name, STREAM > function_name, SHUTDOWN function_name); > DROP COPY TARGET name; > GRANT USA

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 9:56 PM, Daniel Farina wrote: > On Tue, Dec 29, 2009 at 6:48 PM, Robert Haas wrote: >> I think there's clear support for a version of COPY that returns rows >> like a SELECT statement, particularly for use with CTEs.  There seems >> to be support both for a mode that retur

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Tue, 2009-12-29 at 21:48 -0500, Robert Haas wrote: > Anyway, my specific reaction to your suggestions in the email that I > quoted is that it seems a bit baroque and that I'm not really sure > what it's useful for in practice. I'm certainly not saying it ISN'T > useful, because I can't believe

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 6:48 PM, Robert Haas wrote: > I think there's clear support for a version of COPY that returns rows > like a SELECT statement, particularly for use with CTEs.  There seems > to be support both for a mode that returns text[] or something like it > and also for a mode that re

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 9:23 PM, Daniel Farina wrote: > On Tue, Dec 29, 2009 at 5:57 PM, Robert Haas wrote: >> I am very fuzzy on where we stand with this patch.  There's a link to >> the initial version on the commitfest application, but there has been >> so much discussion since then that I thi

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 5:57 PM, Robert Haas wrote: > I am very fuzzy on where we stand with this patch.  There's a link to > the initial version on the commitfest application, but there has been > so much discussion since then that I think there are probably some > revisions to be made, though I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Sat, Dec 5, 2009 at 3:32 AM, Daniel Farina wrote: > On Mon, Nov 30, 2009 at 12:14 PM, Greg Smith wrote: >> Jeff Davis wrote: >> >> COPY target FROM FUNCTION foo() WITH RECORDS; >> >> >> In what format would the records be? > > As a not-quite aside, I think I have a better syntax suggestion.  I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-05 Thread Daniel Farina
On Mon, Nov 30, 2009 at 12:14 PM, Greg Smith wrote: > Jeff Davis wrote: > > COPY target FROM FUNCTION foo() WITH RECORDS; > > > In what format would the records be? As a not-quite aside, I think I have a better syntax suggestion. I have recently been digging around in the grammar with the change

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-30 Thread Greg Smith
Jeff Davis wrote: COPY target FROM FUNCTION foo() WITH RECORDS; In what format would the records be? What was your intended internal format for this form to process? -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQua

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Sun, 2009-11-29 at 18:53 -0800, Daniel Farina wrote: > On Sun, Nov 29, 2009 at 6:35 PM, Jeff Davis wrote: > > What if the network buffer is flushed in the middle of a line? Is that > > possible, or is there a guard against that somewhere? > > What do you mean? They both catenate onto one stre

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Daniel Farina
On Sun, Nov 29, 2009 at 6:35 PM, Jeff Davis wrote: > What if the network buffer is flushed in the middle of a line? Is that > possible, or is there a guard against that somewhere? What do you mean? They both catenate onto one stream of bytes, it shouldn't matter where the flush boundaries are...

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Thu, 2009-11-26 at 18:30 -0800, Daniel Farina wrote: > Okay, so this thread sort of wandered into how we could refactor other > elements of COPY. Do we have a good sense on what we should do to the > current patch (or at least the idea represented by it) to get it into > a committable state wit

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Fri, 2009-11-27 at 20:28 -0500, Greg Smith wrote: > In the context of the read case, I'm not as sure it's so black and > white. While the current situation does map better to a function that > produces a stream of bytes, that's not necessarily the optimal approach > for all situations. It's

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Greg Smith
Jeff Davis wrote: All I mean is that the second argument to COPY should produce/consume bytes and not records. I'm not discussing the internal implementation at all, only semantics. In other words, STDIN is not a source of records, it's a source of bytes; and likewise for STDOUT. In the conte

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Jeff Davis
On Fri, 2009-11-27 at 13:39 +, Simon Riggs wrote: > On Tue, 2009-11-24 at 22:13 -0800, Jeff Davis wrote: > > > My disagreement with the row-by-row approach is more semantics than > > performance. COPY translates records to bytes and vice-versa, and your > > original patch maintains those seman

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Simon Riggs
On Tue, 2009-11-24 at 22:13 -0800, Jeff Davis wrote: > My disagreement with the row-by-row approach is more semantics than > performance. COPY translates records to bytes and vice-versa, and your > original patch maintains those semantics. The bytes <-> records conversion is a costly one. Anythin

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Daniel Farina
On Thu, Nov 26, 2009 at 6:13 PM, David Fetter wrote: > It'd be nice to make this available to the next revision of DBI-Link > and it'll be pretty handy for our SQL/MED whenever that happens. Okay, so this thread sort of wandered into how we could refactor other elements of COPY. Do we have a goo

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread David Fetter
On Thu, Nov 26, 2009 at 10:11:12AM -0500, Andrew Dunstan wrote: > On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: > > On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: > >> On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: > >> > > >> > I worry that we're getting further away from th

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Pavel Stehule
2009/11/26 Jeff Davis : > On Thu, 2009-11-26 at 05:01 +0100, Pavel Stehule wrote: >> It working like: >> >> 1. EXECUTE SELECT 0 FROM generate_series(1,...); >> 2. STORE RESULT TO TABLE zero; >> 3. EXECUTE SELECT 1/i FROM zero; >> 4. STORE RESULT TO TABLE tmp; >> >> Problem is in seq execution. Resu

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Andrew Dunstan
On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: > On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: >> On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: >> > >> > I worry that we're getting further away from the original problem. >> Let's >> > allow functions to get the bytes of data

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Andrew Dunstan
On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: > On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: >> On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: >> > >> > I worry that we're getting further away from the original problem. >> Let's >> > allow functions to get the bytes of data

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: > On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: > > > > I worry that we're getting further away from the original problem. Let's > > allow functions to get the bytes of data from a COPY, like the original > > proposal. I am not sure CO

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Thu, 2009-11-26 at 05:01 +0100, Pavel Stehule wrote: > It working like: > > 1. EXECUTE SELECT 0 FROM generate_series(1,...); > 2. STORE RESULT TO TABLE zero; > 3. EXECUTE SELECT 1/i FROM zero; > 4. STORE RESULT TO TABLE tmp; > > Problem is in seq execution. Result is stored to destination afte

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Daniel Farina
On Wed, Nov 25, 2009 at 9:35 PM, Andrew Dunstan wrote: > On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: >> >> I worry that we're getting further away from the original problem. Let's >> allow functions to get the bytes of data from a COPY, like the original >> proposal. I am not sure COPY is

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Andrew Dunstan
On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: > > I worry that we're getting further away from the original problem. Let's > allow functions to get the bytes of data from a COPY, like the original > proposal. I am not sure COPY is the best mechanism to move records > around when INSERT ... S

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Wed, 2009-11-25 at 11:32 +0100, Pavel Stehule wrote: >> 1. >> postgres=# select count(*) from generate_series(1,100); >>   count >> ─ >>  100 >> (1 row) >> >> Time: 930,720 ms >> >> 2. >> postgres=# select count(*) from (select generate_series(1,100)

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Wed, 2009-11-25 at 11:32 +0100, Pavel Stehule wrote: > 1. > postgres=# select count(*) from generate_series(1,100); > count > ─ > 100 > (1 row) > > Time: 930,720 ms > > 2. > postgres=# select count(*) from (select generate_series(1,100)) x; > count > ─ > 1

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Tom Lane
Jeff Davis writes: > On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: >>> If SRFs use a tuplestore in that situation, it sounds like that should >>> be fixed. Why do we need to provide alternate syntax involving COPY? >> >> It isn't problem of SRF function design. It allow both mode - row

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Daniel Farina
On Tue, Nov 24, 2009 at 10:39 PM, Pavel Stehule wrote: > yes - it is two features - and should be solved independently There are some common problems though. I was thinking about this with some mind towards my existing mental model of thinking of specifying some parameters up-front and getting a

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: >> > If SRFs use a tuplestore in that situation, it sounds like that should >> > be fixed. Why do we need to provide alternate syntax involving COPY? >> >> It isn't problem of SRF function design. It allow both mode -

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Hannu Krosing
On Tue, 2009-11-24 at 21:13 -0800, Jeff Davis wrote: > On Tue, 2009-11-24 at 23:44 -0500, Tom Lane wrote: > > If you do that, then there is no possibility of ever using this feature > > except with C-coded functions, which seems to me to remove most of > > whatever use-case there was. > > It fits

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: > > If SRFs use a tuplestore in that situation, it sounds like that should > > be fixed. Why do we need to provide alternate syntax involving COPY? > > It isn't problem of SRF function design. It allow both mode - row and > tuplestor. select

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Wed, 2009-11-25 at 07:36 +0100, Pavel Stehule wrote: >> > Moving records from a function to a table can be done with: >> >  INSERT INTO mytable SELECT * FROM myfunc(); >> > And that already works fine. >> >> It works, but COPY FROM myfunc() should be significantly faste

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 07:36 +0100, Pavel Stehule wrote: > > Moving records from a function to a table can be done with: > > INSERT INTO mytable SELECT * FROM myfunc(); > > And that already works fine. > > It works, but COPY FROM myfunc() should be significantly faster. You > can skip tuple store.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 07:31 +0100, Pavel Stehule wrote: > > My disagreement with the row-by-row approach is more semantics than > > performance. COPY translates records to bytes and vice-versa, and your > > original patch maintains those semantics. > > uff, really > > COPY CSV ? CSV is like text

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Pavel Stehule : > 2009/11/25 Daniel Farina : >> On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis wrote: >>> On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an "internal" minimalize necessary changes in COPY implementation. Using a funcapi needs more work

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina : > On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis wrote: >> On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: >>> I believe so using an "internal" minimalize necessary changes in COPY >>> implementation. Using a funcapi needs more work inside COPY -  you >>> have to t

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: >> I believe so using an "internal" minimalize necessary changes in COPY >> implementation. Using a funcapi needs more work inside COPY -  you >> have to take some functionality from COPY to stream functions. >> Proba

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis wrote: > On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: >> I believe so using an "internal" minimalize necessary changes in COPY >> implementation. Using a funcapi needs more work inside COPY -  you >> have to take some functionality from COPY

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Tue, 2009-11-24 at 21:42 -0800, Daniel Farina wrote: >> You are probably right.  We could try coercing to bytea and back out >> to bytes, although it seems like a superfluous cost to force >> *everyone* to pay just to get the same bytes to a network buffer. > > Well, I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: > I believe so using an "internal" minimalize necessary changes in COPY > implementation. Using a funcapi needs more work inside COPY - you > have to take some functionality from COPY to stream functions. > Probably the most slow operations i

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 21:42 -0800, Daniel Farina wrote: > You are probably right. We could try coercing to bytea and back out > to bytes, although it seems like a superfluous cost to force > *everyone* to pay just to get the same bytes to a network buffer. Well, I suppose only performance will te

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina : > On Tue, Nov 24, 2009 at 9:35 PM, Pavel Stehule > wrote: >> 2009/11/25 Daniel Farina : >>> On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule >>> wrote: It depends on design. I don't thing so internal is necessary. It is just wrong design. >>> >>> Depends on ho

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 9:35 PM, Pavel Stehule wrote: > 2009/11/25 Daniel Farina : >> On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule >> wrote: >>> It depends on design. I don't thing so internal is necessary. It is >>> just wrong design. >> >> Depends on how lean you want to be when doing large

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina : > On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule > wrote: >> It depends on design. I don't thing so internal is necessary. It is >> just wrong design. > > Depends on how lean you want to be when doing large COPY...right now > the cost is restricted to having to call a f

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 9:13 PM, Jeff Davis wrote: > > I still don't see any reason to force it to be record by record though. > If the point is to push data from a table into a remote table, why > should the copied data be translated out of binary format into a record, > and then back into binar

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 23:44 -0500, Tom Lane wrote: > If you do that, then there is no possibility of ever using this feature > except with C-coded functions, which seems to me to remove most of > whatever use-case there was. It fits the use case involving dblink (or dblink-like modules). Maybe th

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule wrote: > It depends on design. I don't thing so internal is necessary. It is > just wrong design. Depends on how lean you want to be when doing large COPY...right now the cost is restricted to having to call a function pointer and a few branches. If

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis : > On Tue, 2009-11-24 at 14:39 +0100, Pavel Stehule wrote: >> a) good designed C API  like: >> >>    initialise_functions(fcinfo) -- std fcinfo >>    consument_process_tuple(fcinfo) -- gets standard row -- Datum >> dvalues[] + Row description >>    producent_process_tuple(fci

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Tom Lane
Jeff Davis writes: > Don't you still need the functions to accept an argument of type > internal? Otherwise, we lose the ability to copy a buffer to the dblink > connection, which was the original motivation. If you do that, then there is no possibility of ever using this feature except with C-co

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 14:39 +0100, Pavel Stehule wrote: > a) good designed C API like: > >initialise_functions(fcinfo) -- std fcinfo >consument_process_tuple(fcinfo) -- gets standard row -- Datum > dvalues[] + Row description >producent_process_tuple(fcinfo) -- returns standard row -

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 00:54 -0800, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 12:29 AM, Hannu Krosing wrote: > > COPY stdin TO udf(); > > If stdin becomes (is?) a legitimate source of records, then this patch > will Just Work. > STDIN is a source of bytes representing a set of records. Curr

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Robert Haas
On Mon, Nov 23, 2009 at 9:37 PM, Andrew Dunstan wrote: > > > Greg Smith wrote: >> >> I haven't heard anything from Andrew about ragged CVS import either.  I >> think that ultimately those features are useful, but just exceed what the >> existing code could be hacked to handle cleanly. > > The patc

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Hannu Krosing : > On Tue, 2009-11-24 at 05:00 -0800, Daniel Farina wrote: >> On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule >> wrote: > >> > then syntax should be: >> > >> > COPY table TO streamname(parameters) >> > >> > COPY table TO filestream('/tmp/foo.dta') ... >> > COPY table TO d

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Robert Haas
On Mon, Nov 23, 2009 at 8:46 PM, Greg Smith wrote: > You know how people complain about how new contributors are treated here? >  Throwing out comments like this, that come off as belittling to other > people's work, doesn't help.  All I was suggesting was that Dan wasn't > developing this in comp

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 05:00 -0800, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule > wrote: > > then syntax should be: > > > > COPY table TO streamname(parameters) > > > > COPY table TO filestream('/tmp/foo.dta') ... > > COPY table TO dblinkstream(connectionstring) ... You

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Daniel Farina : > On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule > wrote: >> 2009/11/24 Daniel Farina : >>> On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule >>> wrote: Hello I thing, so this patch is maybe good idea. I am missing better function specification. Speci

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 03:48 -0800, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 3:25 AM, Hannu Krosing wrote: > > On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: > >> On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina wrote: > >> > On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing > >> > wro

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule wrote: > 2009/11/24 Daniel Farina : >> On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule >> wrote: >>> Hello >>> >>> I thing, so this patch is maybe good idea. I am missing better >>> function specification. Specification by name isn't enough - we can

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Daniel Farina : > On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule > wrote: >> Hello >> >> I thing, so this patch is maybe good idea. I am missing better >> function specification. Specification by name isn't enough - we can >> have a overloaded functions. This syntax doesn't allow to us

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 3:25 AM, Hannu Krosing wrote: > On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: >> On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina wrote: >> > On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing >> > wrote: >> >> Can't you use existing aggregate function design ? >> >>

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina wrote: > > On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing > > wrote: > >> Can't you use existing aggregate function design ? > >> > >> CREATE AGGREGATE name ( input_data_type [ , ... ] ) ( >

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing wrote: >> Can't you use existing aggregate function design ? >> >> CREATE AGGREGATE name ( input_data_type [ , ... ] ) ( >>    SFUNC = sfunc, >>    STYPE = state_data_type >>    [ , FINALFUNC =

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing wrote: > Can't you use existing aggregate function design ? > > CREATE AGGREGATE name ( input_data_type [ , ... ] ) ( >    SFUNC = sfunc, >    STYPE = state_data_type >    [ , FINALFUNC = ffunc ] >    [ , INITCOND = initial_condition ] >    [ , SORTOP

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 02:37 -0800, Daniel Farina wrote: > On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule > wrote: > > Hello > > > > I thing, so this patch is maybe good idea. I am missing better > > function specification. Specification by name isn't enough - we can > > have a overloaded function

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule wrote: > Hello > > I thing, so this patch is maybe good idea. I am missing better > function specification. Specification by name isn't enough - we can > have a overloaded functions. This syntax doesn't allow to use explicit > cast - from my personal

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
Hello I thing, so this patch is maybe good idea. I am missing better function specification. Specification by name isn't enough - we can have a overloaded functions. This syntax doesn't allow to use explicit cast - from my personal view, the syntax is ugly - with type specification we don't need t

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 12:38 AM, Hannu Krosing wrote: > COPY func(rowtype) FROM stdin; I didn't consider rowtype...I did consider a type list, such as: COPY func(typea, typeb, typec) FROM ... Which would then operate just like a table, but be useless for the data-cleaning case, and would not a

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 12:29 AM, Hannu Krosing wrote: > COPY stdin TO udf(); If stdin becomes (is?) a legitimate source of records, then this patch will Just Work. The patch is already quite useful in the COPY (SELECT ...) TO FUNCTION ... scenario. > COPY udf() FROM stdin; This is unaddressed

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Mon, 2009-11-23 at 16:25 -0800, Daniel Farina wrote: > On Mon, Nov 23, 2009 at 4:20 PM, Tom Lane wrote: > > pgsql-hackers had some preliminary discussions a couple months back > > on refactoring COPY to allow things like this --- see the thread > > starting here: > > http://archives.postgresql.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Mon, 2009-11-23 at 18:31 -0500, Greg Smith wrote: > Robert Haas wrote: > > I'm fuzzy on what problem this is attempting to solve... as mentioned > > in the above guidelines, it's usually good to start with some design > > discussions before writing/submitting code. > This has been through some

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Andrew Dunstan
Greg Smith wrote: I haven't heard anything from Andrew about ragged CVS import either. I think that ultimately those features are useful, but just exceed what the existing code could be hacked to handle cleanly. The patch is attached for your edification/amusement. I have backpatched it t

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Greg Smith
Tom Lane wrote: Those discussions don't have a lot of credibility if they didn't take place on the public mailing lists. You know how people complain about how new contributors are treated here? Throwing out comments like this, that come off as belittling to other people's work, doesn't hel

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 4:20 PM, Tom Lane wrote: > pgsql-hackers had some preliminary discussions a couple months back > on refactoring COPY to allow things like this --- see the thread > starting here: > http://archives.postgresql.org/pgsql-hackers/2009-09/msg00486.php > While I don't think we ar

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Tom Lane
Greg Smith writes: > Robert Haas wrote: >> I'm fuzzy on what problem this is attempting to solve... as mentioned >> in the above guidelines, it's usually good to start with some design >> discussions before writing/submitting code. > This has been through some heavy design discussions with a few

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 4:03 PM, Daniel Farina wrote: > Yes.  Take a look at the tests introduced to core PostgeSQL (see patch > 2), where instead of returning a text[] I return just a single text of > the verbatim output of the copy.  You could imagine making that an SRF > instead.  It would have

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 3:46 PM, Andrew Dunstan wrote: > I recently found myself trying to push data through dblink() and ended up > writing code to make a call to the target to call a function which called > back to the source to select the data and insert it. The speedup was > massive, so I'll b

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Andrew Dunstan
Greg Smith wrote: Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy design discussions with a few PG hacker

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 2:16 PM, Robert Haas wrote: > On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina wrote: >> Signed-off-by: Daniel Farina > > Thanks for the patch.  You may want to take a look at this: > > http://wiki.postgresql.org/wiki/Submitting_a_Patch > > I'm fuzzy on what problem this is

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Greg Smith
Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy design discussions with a few PG hackers you know and some y

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Robert Haas
On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina wrote: > Signed-off-by: Daniel Farina Thanks for the patch. You may want to take a look at this: http://wiki.postgresql.org/wiki/Submitting_a_Patch I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it

[HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
Signed-off-by: Daniel Farina --- contrib/dblink/expected/dblink.out | 272 contrib/dblink/sql/dblink.sql | 112 +++ 2 files changed, 384 insertions(+), 0 deletions(-) diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expecte