Re: [HACKERS] exposing COPY API

2011-02-09 Thread Shigeru HANADA
On Tue, 8 Feb 2011 08:49:36 -0500 Robert Haas robertmh...@gmail.com wrote: On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: I'll submit revised file_fdw patch after removing IsForeignTable() catalog lookup along Heikki's proposal. So I'm a bit confused. I

Re: [HACKERS] exposing COPY API

2011-02-09 Thread Robert Haas
On Wed, Feb 9, 2011 at 7:38 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: On Tue, 8 Feb 2011 08:49:36 -0500 Robert Haas robertmh...@gmail.com wrote: On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: I'll submit revised file_fdw patch after removing

Re: [HACKERS] exposing COPY API

2011-02-09 Thread Andrew Dunstan
On 02/09/2011 12:26 PM, Robert Haas wrote: On Wed, Feb 9, 2011 at 7:38 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: On Tue, 8 Feb 2011 08:49:36 -0500 Robert Haasrobertmh...@gmail.com wrote: On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: I'll submit

Re: [HACKERS] exposing COPY API

2011-02-09 Thread Robert Haas
On Wed, Feb 9, 2011 at 12:45 PM, Andrew Dunstan and...@dunslane.net wrote: Itagaki-san published a patch for this about about 12 hours ago in the file_fdw thread that looks pretty committable to me. OK, excellent. This whole API thing is a breakout from file_fdw, because the original

Re: [HACKERS] exposing COPY API

2011-02-08 Thread Itagaki Takahiro
On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan and...@dunslane.net wrote: Here is a patch against the latest revision of file_fdw to exercise this API. It includes some regression tests, and I think apart from one or two small details plus a requirement for documentation, is complete. The patch

Re: [HACKERS] exposing COPY API

2011-02-08 Thread Andrew Dunstan
On 02/08/2011 03:49 AM, Itagaki Takahiro wrote: On Tue, Feb 8, 2011 at 09:38, Andrew Dunstanand...@dunslane.net wrote: Here is a patch against the latest revision of file_fdw to exercise this API. It includes some regression tests, and I think apart from one or two small details plus a

Re: [HACKERS] exposing COPY API

2011-02-08 Thread Shigeru HANADA
On Tue, 8 Feb 2011 17:49:09 +0900 Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan and...@dunslane.net wrote: Here is a patch against the latest revision of file_fdw to exercise this API. It includes some regression tests, and I think apart from

Re: [HACKERS] exposing COPY API

2011-02-08 Thread Robert Haas
On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA han...@metrosystems.co.jp wrote: On Tue, 8 Feb 2011 17:49:09 +0900 Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan and...@dunslane.net wrote: Here is a patch against the latest revision of file_fdw

Re: [HACKERS] exposing COPY API

2011-02-07 Thread Andrew Dunstan
On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It

Re: [HACKERS] exposing COPY API

2011-02-07 Thread Andrew Dunstan
On 02/07/2011 11:34 AM, Andrew Dunstan wrote: On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char

Re: [HACKERS] exposing COPY API

2011-02-04 Thread Itagaki Takahiro
Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It just returns separated fields in the next line.

Re: [HACKERS] exposing COPY API

2011-02-04 Thread Andrew Dunstan
On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Itagaki Takahiro
On Fri, Feb 4, 2011 at 09:48, Andrew Dunstan and...@dunslane.net wrote: I'd like to be able to add a callback function to construct the values for the tuple. So it would become something like:   typedef void (*copy_make_values) (CopyState cstate, NumFieldsRead int); You can do nothing

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Andrew Dunstan
Of course, I want this so I could construct a text array from the read in data, but I could also imagine a foreign data wrapper wanting to mangle the data before handing it to postgres, say by filling in a field or hashing it. Could you explain the actual use-cases and examples? I think we

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Itagaki Takahiro
On Fri, Feb 4, 2011 at 11:32, Andrew Dunstan and...@dunslane.net wrote: Umm, where? I can't find this in the documentation http://pgbulkload.projects.postgresql.org/pg_bulkload.html Here: http://pgbulkload.projects.postgresql.org/pg_bulkload.html#filter The object, as I have explained

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Andrew Dunstan
On 02/03/2011 09:43 PM, Itagaki Takahiro wrote: On Fri, Feb 4, 2011 at 11:32, Andrew Dunstanand...@dunslane.net wrote: Umm, where? I can't find this in the documentation http://pgbulkload.projects.postgresql.org/pg_bulkload.html Here:

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Itagaki Takahiro
On Fri, Feb 4, 2011 at 12:17, Andrew Dunstan and...@dunslane.net wrote: http://pgbulkload.projects.postgresql.org/pg_bulkload.html#filter AFAICT, this doesn't support ragged tables with too many columns, which is my prime use case. If it supported variadic arguments in filter functions it

Re: [HACKERS] exposing COPY API

2011-02-03 Thread Andrew Dunstan
On 02/03/2011 10:43 PM, Itagaki Takahiro wrote: Also, a FDW allows the COPY to be used as a FROM target, giving it great flexibility. AFAICT this does not. BTW, which do you want to improve, FDW or COPY FROM? If FDW, the better API would be raw version of NextCopyFrom(). For example: