Hi
2016-04-05 10:45 GMT+02:00 Pavel Stehule <pavel.steh...@gmail.com>:
> Hi
>
> here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY
> formats for COPY statements.
>
> The RAW with text formats means unescaped data, but with correct encoding
> - input/output is realised with input/output function. RAW binary means
> content produced/received by sending/received functions.
>
> Now both directions (input/output) working well
>
> Some examples of expected usage:
>
> copy (select xmlelement(name foo, 'hello')) to stdout (format raw_binary,
> encoding 'latin2');
>
> create table avatars(id serial, picture bytea);
> \copy avatars(picture) from ~/images/foo.jpg (format raw_binary);
> select lastval();
>
> create table doc(id serial, txt text);
> \copy doc(txt) from ~/files/aaa.txt (format raw_text, encoding 'latin2');
> select lastval();
>
> Regards
>
> Pavel
>
>
I am sending fresh version of COPY RAW patch.
There is new regress client test requested by Tom.
Note: I though about another solution based on binary parameters and binary
result support in psql. Somelike:
INSERT INTO foo(a) VALUES($1)
\gpush filename
SELECT a FROM foo
\gpop filename
but, it is less intuitive, and doesn't work with stdin/stdout - so it is
significant week against COPY based solution for scripting from shell. More
\g***** solution is still possible if will be requested in future.
Regards
Pavel
[pavel@nemesis ~]$ cat avatar.gif | psql -Xq -At -c "copy xx(b) from stdin
(format raw_text)" -c "select lastval()" postgres
313
commit d62c1ff8dee2324ce1fe7765c2d015e68f5f923a
Author: Pavel Stehule <pavel.steh...@gooddata.com>
Date: Sat Jul 16 10:35:25 2016 +0200
with regress tests
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 6285dd0..4c6cacb 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3226,8 +3226,9 @@ int PQfformat(const PGresult *res,
<para>
Format code zero indicates textual data representation, while format
- code one indicates binary representation. (Other codes are reserved
- for future definition.)
+ code one indicates binary representation. Format code two indicates
+ raw_text representation and format code three indicates raw_binary
+ representation (Other codes are reserved for future definition.)
</para>
</listitem>
</varlistentry>
@@ -3557,6 +3558,26 @@ typedef struct
</para>
<variablelist>
+ <varlistentry id="libpq-pqcopyformat">
+ <term>
+ <function>PQcopyFormat</function>
+ <indexterm>
+ <primary>PQcopyFormat</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Format code zero indicates textual data representation, format one
+ indicates binary representation, format two indicates raw
+ representation.
+<synopsis>
+int PQcopyFormat(PGresult *res);
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-pqcmdstatus">
<term>
<function>PQcmdStatus</function>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9c96d8f..adcff46 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3239,6 +3239,7 @@ CopyInResponse (B)
characters, etc).
1 indicates the overall copy format is binary (similar
to DataRow format).
+ 2 indicates the overall copy format is raw.
See <xref linkend="sql-copy">
for more information.
</para>
@@ -3262,8 +3263,9 @@ CopyInResponse (B)
<listitem>
<para>
The format codes to be used for each column.
- Each must presently be zero (text) or one (binary).
- All must be zero if the overall copy format is textual.
+ Each must be zero (text), one (binary), two (raw_text)
+ or three (raw_binary). All must be zero if the overall
+ copy format is textual.
</para>
</listitem>
</varlistentry>
@@ -3313,7 +3315,8 @@ CopyOutResponse (B)
is textual (rows separated by newlines, columns
separated by separator characters, etc). 1 indicates
the overall copy format is binary (similar to DataRow
- format). See <xref linkend="sql-copy"> for more information.
+ format). 2 indicates raw_text or raw_binary format.
+ See <xref linkend="sql-copy"> for more information.
</para>
</listitem>
</varlistentry>
@@ -3335,8 +3338,9 @@ CopyOutResponse (B)
<listitem>
<para>
The format codes to be used for each column.
- Each must presently be zero (text) or one (binary).
- All must be zero if the overall copy format is textual.
+ Each must be zero (text), one (binary), two (raw_text)
+ or three (raw_binary). All must be zero if the overall
+ copy format is textual.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 07e2f45..4e339e4 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -197,7 +197,9 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
Selects the data format to be read or written:
<literal>text</>,
<literal>csv</> (Comma Separated Values),
- or <literal>binary</>.
+ <literal>binary</>,
+ <literal>raw_text</>
+ or <literal>raw_binary</>.
The default is <literal>text</>.
</para>
</listitem>
@@ -888,6 +890,44 @@ OIDs to be shown as null if that ever proves desirable.
</para>
</refsect3>
</refsect2>
+
+ <refsect2>
+ <title>Raw_text/raw_binary Format</title>
+
+ <para>
+ The <literal>raw_text</literal> format option causes all data to be
+ stored/read as one text value. This format doesn't use any metadata
+ - only raw data are exported or imported.
+ </para>
+
+ <para>
+ The <literal>raw_binary</literal> format option causes all data to be
+ stored/read as binary format rather than as text. It shares format
+ for data with <literal>binary</literal> format. This format doesn't
+ use any metadata - only row data in network byte order are exported
+ or imported.
+ </para>
+
+ <para>
+ Because this format doesn't support any delimiter, only one value
+ can be exported or imported. NULL values are not allowed.
+ </para>
+ <para>
+ The <literal>raw_binary</literal> format can be used for export or import
+ bytea values.
+<programlisting>
+COPY images(data) FROM '/usr1/proj/img/01.jpg' (FORMAT raw_binary);
+</programlisting>
+ It can be used successfully for export XML in different encoding
+ or import valid XML document with any supported encoding:
+<screen><![CDATA[
+SET client_encoding TO latin2;
+
+COPY (SELECT xmlelement(NAME data, 'Hello')) TO stdout (FORMAT raw_binary);
+<?xml version="1.0" encoding="LATIN2"?><data>Hello</data>
+]]></screen>
+ </para>
+ </refsect2>
</refsect1>
<refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f45b330..c63d052 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -110,6 +110,7 @@ typedef struct CopyStateData
char *filename; /* filename, or NULL for STDIN/STDOUT */
bool is_program; /* is 'filename' a program to popen? */
bool binary; /* binary format? */
+ bool raw; /* raw mode? */
bool oids; /* include OIDs? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
@@ -342,12 +343,27 @@ SendCopyBegin(CopyState cstate)
/* new way */
StringInfoData buf;
int natts = list_length(cstate->attnumlist);
- int16 format = (cstate->binary ? 1 : 0);
+ int16 format;
+ int mode;
int i;
pq_beginmessage(&buf, 'H');
- pq_sendbyte(&buf, format); /* overall format */
+
+ if (cstate->raw)
+ mode = 2;
+ else if (cstate->binary)
+ mode = 1;
+ else
+ mode = 0;
+
+ pq_sendbyte(&buf, mode); /* overall mode */
pq_sendint(&buf, natts, 2);
+
+ if (!cstate->raw)
+ format = cstate->binary ? 1 : 0;
+ else
+ format = cstate->binary ? 3 : 2;
+
for (i = 0; i < natts; i++)
pq_sendint(&buf, format, 2); /* per-column formats */
pq_endmessage(&buf);
@@ -356,10 +372,10 @@ SendCopyBegin(CopyState cstate)
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
/* old way */
- if (cstate->binary)
+ if (cstate->binary || cstate->raw)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY BINARY is not supported to stdout or from stdin")));
+ errmsg("COPY BINARY or COPY RAW_TEXT/RAW_BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
@@ -368,10 +384,10 @@ SendCopyBegin(CopyState cstate)
else
{
/* very old way */
- if (cstate->binary)
+ if (cstate->binary || cstate->raw)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY BINARY is not supported to stdout or from stdin")));
+ errmsg("COPY BINARY or COPY RAW_TEXT/RAW_BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
@@ -387,12 +403,27 @@ ReceiveCopyBegin(CopyState cstate)
/* new way */
StringInfoData buf;
int natts = list_length(cstate->attnumlist);
- int16 format = (cstate->binary ? 1 : 0);
+ int16 format;
+ int mode;
int i;
pq_beginmessage(&buf, 'G');
- pq_sendbyte(&buf, format); /* overall format */
+
+ if (cstate->raw)
+ mode = 2;
+ else if (cstate->binary)
+ mode = 1;
+ else
+ mode = 0;
+
+ pq_sendbyte(&buf, mode); /* overall format */
pq_sendint(&buf, natts, 2);
+
+ if (!cstate->raw)
+ format = cstate->binary ? 1 : 0;
+ else
+ format = cstate->binary ? 3 : 2;
+
for (i = 0; i < natts; i++)
pq_sendint(&buf, format, 2); /* per-column formats */
pq_endmessage(&buf);
@@ -402,10 +433,10 @@ ReceiveCopyBegin(CopyState cstate)
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
/* old way */
- if (cstate->binary)
+ if (cstate->binary || cstate->raw)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY BINARY is not supported to stdout or from stdin")));
+ errmsg("COPY BINARY or COPY RAW_TEXT/RAW_BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
/* any error in old protocol will make us lose sync */
pq_startmsgread();
@@ -414,10 +445,10 @@ ReceiveCopyBegin(CopyState cstate)
else
{
/* very old way */
- if (cstate->binary)
+ if (cstate->binary || cstate->raw)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY BINARY is not supported to stdout or from stdin")));
+ errmsg("COPY BINARY or COPY RAW_TEXT/RAW_BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
/* any error in old protocol will make us lose sync */
pq_startmsgread();
@@ -482,7 +513,7 @@ CopySendEndOfRow(CopyState cstate)
switch (cstate->copy_dest)
{
case COPY_FILE:
- if (!cstate->binary)
+ if (!cstate->binary && !cstate->raw)
{
/* Default line termination depends on platform */
#ifndef WIN32
@@ -526,6 +557,9 @@ CopySendEndOfRow(CopyState cstate)
}
break;
case COPY_OLD_FE:
+ /* This old protocol doesn't allow RAW_TEXT/RAW_BINARY */
+ Assert(!cstate->raw);
+
/* The FE/BE protocol uses \n as newline for all platforms */
if (!cstate->binary)
CopySendChar(cstate, '\n');
@@ -540,7 +574,7 @@ CopySendEndOfRow(CopyState cstate)
break;
case COPY_NEW_FE:
/* The FE/BE protocol uses \n as newline for all platforms */
- if (!cstate->binary)
+ if (!cstate->binary && !cstate->raw)
CopySendChar(cstate, '\n');
/* Dump the accumulated row as one CopyData message */
@@ -766,6 +800,38 @@ CopyLoadRawBuf(CopyState cstate)
return (inbytes > 0);
}
+/*
+ * CopyLoadallRawBuf loads all content into raw_buf.
+ *
+ * This routine is used in raw_text/raw_binary mode. If original RAW_BUF_SIZE is not
+ * enough, then the buffer is enlarged.
+ */
+static void
+CopyLoadallRawBuf(CopyState cstate)
+{
+ int nbytes = 0;
+ int inbytes;
+ Size raw_buf_size = RAW_BUF_SIZE;
+
+ do
+ {
+ /* hold enough space for one data packet */
+ if ((raw_buf_size - nbytes - 1) < 8 * 1024)
+ {
+ raw_buf_size += RAW_BUF_SIZE;
+ cstate->raw_buf = repalloc(cstate->raw_buf, raw_buf_size);
+ }
+
+ inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, 1, raw_buf_size - nbytes - 1);
+ nbytes += inbytes;
+ }
+ while (inbytes > 0);
+
+ cstate->raw_buf[nbytes] = '\0';
+ cstate->raw_buf_index = 0;
+ cstate->raw_buf_len = nbytes;
+}
+
/*
* DoCopy executes the SQL COPY statement
@@ -1013,6 +1079,13 @@ ProcessCopyOptions(CopyState cstate,
cstate->csv_mode = true;
else if (strcmp(fmt, "binary") == 0)
cstate->binary = true;
+ else if (strcmp(fmt, "raw_text") == 0)
+ cstate->raw = true;
+ else if (strcmp(fmt, "raw_binary") == 0)
+ {
+ cstate->binary = true;
+ cstate->raw = true;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -1162,16 +1235,21 @@ ProcessCopyOptions(CopyState cstate,
* Check for incompatible options (must do these two before inserting
* defaults)
*/
- if (cstate->binary && cstate->delim)
+ if ((cstate->binary || cstate->raw) && cstate->delim)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DELIMITER in BINARY mode")));
- if (cstate->binary && cstate->null_print)
+ if ((cstate->binary || cstate->raw) && cstate->null_print)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify NULL in BINARY mode")));
+ if (cstate->raw && cstate->oids)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify OIDS in RAW_TEXT/RAW_BINARY mode")));
+
/* Set defaults for omitted options */
if (!cstate->delim)
cstate->delim = cstate->csv_mode ? "," : "\t";
@@ -1608,6 +1686,12 @@ BeginCopy(bool is_from,
}
}
+ /* No more columns are allowed in RAW mode */
+ if (cstate->raw && list_length(cstate->attnumlist) > 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("Single column result/target is required in RAW_TEXT/RAW_BINARY mode")));
+
/* Use client encoding when ENCODING option is not specified. */
if (cstate->file_encoding < 0)
cstate->file_encoding = pg_get_client_encoding();
@@ -1899,7 +1983,7 @@ CopyTo(CopyState cstate)
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
- if (cstate->binary)
+ if (cstate->binary && !cstate->raw)
{
/* Generate header for a binary copy */
int32 tmp;
@@ -1931,6 +2015,9 @@ CopyTo(CopyState cstate)
{
bool hdr_delim = false;
+ /* raw_text/raw_binary mode is not allowed here */
+ Assert(!cstate->raw);
+
foreach(cur, cstate->attnumlist)
{
int attnum = lfirst_int(cur);
@@ -1967,6 +2054,10 @@ CopyTo(CopyState cstate)
{
CHECK_FOR_INTERRUPTS();
+ /* stop quickly in raw_text/raw_binary when more rows is detected */
+ if (cstate->raw && processed > 0)
+ break;
+
/* Deconstruct the tuple ... faster than repeated heap_getattr */
heap_deform_tuple(tuple, tupDesc, values, nulls);
@@ -1983,11 +2074,25 @@ CopyTo(CopyState cstate)
else
{
/* run the plan --- the dest receiver will send tuples */
- ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ ExecutorRun(cstate->queryDesc, ForwardScanDirection, cstate->raw ? 2L : 0L);
processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
- if (cstate->binary)
+ /* raw_text/raw_binary requires exactly one row */
+ if (cstate->raw)
+ {
+ if (processed > 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_ROWS),
+ errmsg("single row result is required by RAW_TEXT/RAW_BINARY mode")));
+
+ if (processed == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_NO_DATA_FOUND),
+ errmsg("single row result is required by RAW_TEXT/RAW_BINARY mode")));
+ }
+
+ if (cstate->binary && !cstate->raw)
{
/* Generate trailer for a binary copy */
CopySendInt16(cstate, -1);
@@ -2015,28 +2120,31 @@ CopyOneRowTo(CopyState cstate, Oid tupleOid, Datum *values, bool *nulls)
MemoryContextReset(cstate->rowcontext);
oldcontext = MemoryContextSwitchTo(cstate->rowcontext);
- if (cstate->binary)
+ if (!cstate->raw)
{
- /* Binary per-tuple header */
- CopySendInt16(cstate, list_length(cstate->attnumlist));
- /* Send OID if wanted --- note attnumlist doesn't include it */
- if (cstate->oids)
+ if (cstate->binary)
{
- /* Hack --- assume Oid is same size as int32 */
- CopySendInt32(cstate, sizeof(int32));
- CopySendInt32(cstate, tupleOid);
+ /* Binary per-tuple header */
+ CopySendInt16(cstate, list_length(cstate->attnumlist));
+ /* Send OID if wanted --- note attnumlist doesn't include it */
+ if (cstate->oids)
+ {
+ /* Hack --- assume Oid is same size as int32 */
+ CopySendInt32(cstate, sizeof(int32));
+ CopySendInt32(cstate, tupleOid);
+ }
}
- }
- else
- {
- /* Text format has no per-tuple header, but send OID if wanted */
- /* Assume digits don't need any quoting or encoding conversion */
- if (cstate->oids)
+ else
{
- string = DatumGetCString(DirectFunctionCall1(oidout,
- ObjectIdGetDatum(tupleOid)));
- CopySendString(cstate, string);
- need_delim = true;
+ /* Text format has no per-tuple header, but send OID if wanted */
+ /* Assume digits don't need any quoting or encoding conversion */
+ if (cstate->oids)
+ {
+ string = DatumGetCString(DirectFunctionCall1(oidout,
+ ObjectIdGetDatum(tupleOid)));
+ CopySendString(cstate, string);
+ need_delim = true;
+ }
}
}
@@ -2055,6 +2163,11 @@ CopyOneRowTo(CopyState cstate, Oid tupleOid, Datum *values, bool *nulls)
if (isnull)
{
+ if (cstate->raw)
+ ereport(ERROR,
+ (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED),
+ errmsg("NULL value is not allowed in RAW_TEXT/RAW_BINARY mode")));
+
if (!cstate->binary)
CopySendString(cstate, cstate->null_print_client);
else
@@ -2062,7 +2175,72 @@ CopyOneRowTo(CopyState cstate, Oid tupleOid, Datum *values, bool *nulls)
}
else
{
- if (!cstate->binary)
+ if (cstate->raw)
+ {
+ const void *content;
+ int size;
+
+ if (!cstate->binary)
+ {
+ string = OutputFunctionCall(&out_functions[attnum - 1],
+ value);
+
+ /* We would to transcode, but without escaping */
+ if (cstate->need_transcoding)
+ content = pg_server_to_any(string, strlen(string), cstate->file_encoding);
+ else
+ content = string;
+
+ size = strlen((const char *) content);
+ }
+ else
+ {
+ bytea *outputbytes;
+
+ /*
+ * Some binary output functions depends can depends on client encoding.
+ * The binary output of xml is good example. Set client_encoding
+ * temporaly before out function execution.
+ */
+ if (cstate->need_transcoding)
+ {
+ int old_server_encoding = pg_get_client_encoding();
+ volatile bool reset_encoding = false;
+
+ PG_TRY();
+ {
+ /* We don't expect an error, because encoding was checked before */
+ if (PrepareClientEncoding(cstate->file_encoding) < 0)
+ elog(ERROR, "PrepareClientEncoding(%d) failed", cstate->file_encoding);
+
+ SetClientEncoding(cstate->file_encoding);
+ reset_encoding = true;
+
+ outputbytes = SendFunctionCall(&out_functions[attnum - 1],
+ value);
+ SetClientEncoding(old_server_encoding);
+ }
+ PG_CATCH();
+ {
+ if (reset_encoding)
+ SetClientEncoding(old_server_encoding);
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+ }
+ else
+ {
+ outputbytes = SendFunctionCall(&out_functions[attnum - 1],
+ value);
+ }
+ content = VARDATA(outputbytes);
+ size = VARSIZE(outputbytes) - VARHDRSZ;
+ }
+
+ /* Send only content in RAW_TEXT/RAW_BINARY mode */
+ CopySendData(cstate, content, size);
+ }
+ else if (!cstate->binary)
{
string = OutputFunctionCall(&out_functions[attnum - 1],
value);
@@ -2811,65 +2989,69 @@ BeginCopyFrom(Relation rel,
}
}
- if (!cstate->binary)
+ /* The raw mode hasn't any header information */
+ if (!cstate->raw)
{
- /* must rely on user to tell us... */
- cstate->file_has_oids = cstate->oids;
- }
- else
- {
- /* Read and verify binary header */
- char readSig[11];
- int32 tmp;
-
- /* Signature */
- if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
- memcmp(readSig, BinarySignature, 11) != 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("COPY file signature not recognized")));
- /* Flags field */
- if (!CopyGetInt32(cstate, &tmp))
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("invalid COPY file header (missing flags)")));
- cstate->file_has_oids = (tmp & (1 << 16)) != 0;
- tmp &= ~(1 << 16);
- if ((tmp >> 16) != 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("unrecognized critical flags in COPY file header")));
- /* Header extension length */
- if (!CopyGetInt32(cstate, &tmp) ||
- tmp < 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("invalid COPY file header (missing length)")));
- /* Skip extension header, if present */
- while (tmp-- > 0)
+ if (!cstate->binary)
+ {
+ /* must rely on user to tell us... */
+ cstate->file_has_oids = cstate->oids;
+ }
+ else
{
- if (CopyGetData(cstate, readSig, 1, 1) != 1)
+ /* Read and verify binary header */
+ char readSig[11];
+ int32 tmp;
+
+ /* Signature */
+ if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
+ memcmp(readSig, BinarySignature, 11) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("COPY file signature not recognized")));
+ /* Flags field */
+ if (!CopyGetInt32(cstate, &tmp))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("invalid COPY file header (wrong length)")));
+ errmsg("invalid COPY file header (missing flags)")));
+ cstate->file_has_oids = (tmp & (1 << 16)) != 0;
+ tmp &= ~(1 << 16);
+ if ((tmp >> 16) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("unrecognized critical flags in COPY file header")));
+ /* Header extension length */
+ if (!CopyGetInt32(cstate, &tmp) ||
+ tmp < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("invalid COPY file header (missing length)")));
+ /* Skip extension header, if present */
+ while (tmp-- > 0)
+ {
+ if (CopyGetData(cstate, readSig, 1, 1) != 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("invalid COPY file header (wrong length)")));
+ }
}
- }
- if (cstate->file_has_oids && cstate->binary)
- {
- getTypeBinaryInputInfo(OIDOID,
- &in_func_oid, &cstate->oid_typioparam);
- fmgr_info(in_func_oid, &cstate->oid_in_function);
- }
+ if (cstate->file_has_oids && cstate->binary)
+ {
+ getTypeBinaryInputInfo(OIDOID,
+ &in_func_oid, &cstate->oid_typioparam);
+ fmgr_info(in_func_oid, &cstate->oid_in_function);
+ }
- /* create workspace for CopyReadAttributes results */
- if (!cstate->binary)
- {
- AttrNumber attr_count = list_length(cstate->attnumlist);
- int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+ /* create workspace for CopyReadAttributes results */
+ if (!cstate->binary)
+ {
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
- cstate->max_fields = nfields;
- cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
+ cstate->max_fields = nfields;
+ cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
+ }
}
MemoryContextSwitchTo(oldcontext);
@@ -2968,7 +3150,54 @@ NextCopyFrom(CopyState cstate, ExprContext *econtext,
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
- if (!cstate->binary)
+ if (cstate->raw)
+ {
+ int m = linitial_int(cstate->attnumlist) - 1;
+
+ /* All content was read in first cycle */
+ if (++cstate->cur_lineno > 1)
+ return false;
+
+ CopyLoadallRawBuf(cstate);
+
+ cstate->cur_attname = NameStr(attr[m]->attname);
+
+ if (!cstate->binary)
+ {
+ char *cvt;
+
+ cvt = pg_any_to_server(cstate->raw_buf,
+ cstate->raw_buf_len,
+ cstate->file_encoding);
+
+ values[m] = InputFunctionCall(&in_functions[m],
+ cvt,
+ typioparams[m],
+ attr[m]->atttypmod);
+ }
+ else
+ {
+ cstate->attribute_buf.data = cstate->raw_buf;
+ cstate->attribute_buf.len = cstate->raw_buf_len;
+ cstate->attribute_buf.cursor = 0;
+ cstate->raw_buf = NULL;
+
+ /* Call the column type's binary input converter */
+ values[m] = ReceiveFunctionCall(&in_functions[m], &cstate->attribute_buf,
+ typioparams[m], attr[m]->atttypmod);
+
+ /* Trouble if it didn't eat the whole buffer */
+ if (cstate->attribute_buf.cursor != cstate->attribute_buf.len)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
+ errmsg("incorrect binary data format")));
+ }
+
+ nulls[m] = false;
+
+ cstate->cur_attname = NULL;
+ }
+ else if (!cstate->binary)
{
char **field_strings;
ListCell *cur;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 8469d9f..2c8d6f5 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1974,8 +1974,8 @@ psql_completion(const char *text, int start, int end)
/* Handle COPY [BINARY] <sth> FROM|TO filename */
else if (Matches4("COPY|\\copy", MatchAny, "FROM|TO", MatchAny) ||
Matches5("COPY", "BINARY", MatchAny, "FROM|TO", MatchAny))
- COMPLETE_WITH_LIST6("BINARY", "OIDS", "DELIMITER", "NULL", "CSV",
- "ENCODING");
+ COMPLETE_WITH_LIST8("BINARY", "RAW_TEXT", "RAW_BINARY", "OIDS",
+ "DELIMITER", "NULL", "CSV", "ENCODING");
/* Handle COPY [BINARY] <sth> FROM|TO filename CSV */
else if (Matches5("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "CSV") ||
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..a2754f1 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,4 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQcopyFormat 172
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 2621767..09967e9 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -155,6 +155,7 @@ PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status)
result->resultStatus = status;
result->cmdStatus[0] = '\0';
result->binary = 0;
+ result->raw = 0;
result->events = NULL;
result->nEvents = 0;
result->errMsg = NULL;
@@ -256,8 +257,10 @@ PQsetResultAttrs(PGresult *res, int numAttributes, PGresAttDesc *attDescs)
if (!res->attDescs[i].name)
return FALSE;
- if (res->attDescs[i].format == 0)
+ if (res->attDescs[i].format == 0 || res->attDescs[i].format == 2)
res->binary = 0;
+ if (res->attDescs[i].format == 2 || res->attDescs[i].format == 3)
+ res->raw = 1;
}
return TRUE;
@@ -2932,6 +2935,21 @@ PQcmdStatus(PGresult *res)
}
/*
+ * PQcopyFormat
+ *
+ * Returns a info about copy mode:
+ * -1 signalize a error, 0 = text mode, 1 = binary mode, 2 = raw mode
+ */
+int
+PQcopyFormat(const PGresult *res)
+{
+ if (res->raw)
+ return 2;
+ else
+ return res->binary;
+}
+
+/*
* PQoidStatus -
* if the last command was an INSERT, return the oid string
* if not, return ""
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 0b8c62f..1783844 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -1486,6 +1486,10 @@ getCopyStart(PGconn *conn, ExecStatusType copytype)
*/
format = (int) ((int16) format);
result->attDescs[i].format = format;
+
+ /* when any field uses raw format, then COPY RAW_* was used */
+ if (format == 2 || format == 3)
+ result->raw = true;
}
/* Success! */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 9ca0756..7984666 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -479,6 +479,7 @@ extern Oid PQftype(const PGresult *res, int field_num);
extern int PQfsize(const PGresult *res, int field_num);
extern int PQfmod(const PGresult *res, int field_num);
extern char *PQcmdStatus(PGresult *res);
+extern int PQcopyFormat(const PGresult *res);
extern char *PQoidStatus(const PGresult *res); /* old and ugly */
extern Oid PQoidValue(const PGresult *res); /* new and improved */
extern char *PQcmdTuples(PGresult *res);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1183323..8fc4b04 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -180,6 +180,7 @@ struct pg_result
char cmdStatus[CMDSTATUS_LEN]; /* cmd status from the query */
int binary; /* binary tuple values if binary == 1,
* otherwise text */
+ int raw; /* raw mode for COPY */
/*
* These fields are copied from the originating PGconn, so that operations
diff --git a/src/interfaces/libpq/test/Makefile b/src/interfaces/libpq/test/Makefile
index ab41dc3..f5f1511 100644
--- a/src/interfaces/libpq/test/Makefile
+++ b/src/interfaces/libpq/test/Makefile
@@ -9,14 +9,18 @@ endif
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = uri-regress
+PROGS = uri-regress copy-raw-regress
all: $(PROGS)
installcheck: all
SRCDIR='$(top_srcdir)' SUBDIR='$(subdir)' \
$(PERL) $(top_srcdir)/$(subdir)/regress.pl
+ SRCDIR='$(top_srcdir)' SUBDIR='$(subdir)' \
+ $(PERL) $(top_srcdir)/$(subdir)/copy-raw-regress.pl
clean distclean maintainer-clean:
rm -f $(PROGS)
rm -f regress.out regress.diff
+ rm -f copy-raw-regress.out
+ rm -f uri-regress.o copy-raw-regress.o
\ No newline at end of file
diff --git a/src/interfaces/libpq/test/copy-raw-expected.out b/src/interfaces/libpq/test/copy-raw-expected.out
new file mode 100644
index 0000000..6bbef51
--- /dev/null
+++ b/src/interfaces/libpq/test/copy-raw-expected.out
@@ -0,0 +1,33 @@
+trying
+copy-raw-regress: buffer is empty
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 0)
+copy-raw-regress: buffer is empty
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 0)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 1)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 1)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 10)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 10)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 1000)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 1000)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 10000)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 10000)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 100000)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 100000)
+copy-raw-regress: the test COPY TO was successful (a, raw_text, 1000000)
+copy-raw-regress: the test COPY TO was successful (b, raw_binary, 1000000)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 0)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 0)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 1)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 1)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 10)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 10)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 1000)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 1000)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 10000)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 10000)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 100000)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 100000)
+copy-raw-regress: the test COPY FROM was successful (a, raw_text, 1000000)
+copy-raw-regress: the test COPY FROM was successful (b, raw_binary, 1000000)
+COPY RAW tests done
+
diff --git a/src/interfaces/libpq/test/copy-raw-regress.c b/src/interfaces/libpq/test/copy-raw-regress.c
new file mode 100644
index 0000000..2cdb331
--- /dev/null
+++ b/src/interfaces/libpq/test/copy-raw-regress.c
@@ -0,0 +1,597 @@
+/*
+ * copy-raw-regress.c
+ * A test program for COPY API
+ *
+ * Portions Copyright (c) 2012-2016, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/interfaces/libpq/test/copy-raw-regress.c
+ */
+
+#include "postgres_fe.h"
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+#include "catalog/pg_type.h"
+
+char *text_message = "The PostgreSQL Database, ";
+char *binary_message = "\360\237\230\203\040\360\237\230\202\040";
+
+const char *prgname = "copy-raw-regress";
+
+/*
+ * Hide possible notice from DROP TABLE IF EXIT statement
+ *
+ */
+static void
+terseNoticeProcessor(void *arg, const char *message)
+{
+ /* do nothing */
+}
+
+/*
+ * Prepare textual and binary data with requested size.
+ *
+ */
+static void
+PrepareTestValues(char *values[], int size)
+{
+ char *tbuffer;
+ char *bbuffer;
+ char *p;
+ int nchars;
+ int l;
+
+ tbuffer = pg_malloc(size + 1);
+ bbuffer = pg_malloc(size + 1);
+
+ p = tbuffer;
+ nchars = size;
+ l = strlen(text_message);
+ while (nchars > 0)
+ {
+ if (l < nchars)
+ {
+ strcpy(p, text_message);
+ p += l;
+ nchars -= l;
+ }
+ else
+ {
+ strncpy(p, text_message, nchars);
+ nchars = 0;
+ }
+ }
+
+ tbuffer[size] = '\0';
+ values[0] = tbuffer;
+
+ p = bbuffer;
+ nchars = size;
+ l = 10;
+ while (nchars > 0)
+ {
+ if (l < nchars)
+ {
+ memcpy(p, binary_message, l);
+ p += l;
+ nchars -= l;
+ }
+ else
+ {
+ memset(p, 0, nchars);
+ nchars = 0;
+ }
+ }
+
+ bbuffer[size] = '\0';
+ values[1] = bbuffer;
+}
+
+/*
+ * Store textual and binary data in table copy_raw. Clean this table before.
+ *
+ */
+static bool
+InsertData(PGconn *conn, char *values[], int size)
+{
+ PGresult *res;
+ Oid types[] = {TEXTOID, BYTEAOID};
+ int formats[] = {0, 1};
+ int lens[2];
+
+ lens[0] = 0; /* text format is used, value is ignored */
+ lens[1] = size; /* binary mode, value should be correct */
+
+ res = PQexec(conn, "TRUNCATE TABLE copy_raw");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not to truncate table \"copy_raw\": %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ res = PQexecParams(conn, "INSERT INTO copy_raw(id, a,b) VALUES(1, $1, $2)",
+ 2,
+ types,
+ (const char * const*) values,
+ lens,
+ formats,
+ 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not to insert to table \"copy_raw\": %s\n",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+ return true;
+}
+
+/*
+ * COPY TO test - insert data to server, get it with COPY API and compare result
+ *
+ */
+static bool
+CopyToTest(PGconn *conn, int colnum, const char *method, int size)
+{
+ char *colnames[] = {"a", "b"};
+ char *values[2];
+ PGresult *res;
+ PQExpBufferData query;
+ int len;
+ char *buffer;
+
+ PrepareTestValues(values, size);
+ if (!InsertData(conn, values, size))
+ return false;
+
+ initPQExpBuffer(&query);
+ appendPQExpBuffer(&query,
+ "COPY (SELECT %s FROM copy_raw WHERE id = 1) TO stdout (FORMAT %s)",
+ colnames[colnum],
+ method);
+
+ res = PQexec(conn, query.data);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "%s: could not to execute \"%s\": %s\n",
+ prgname,
+ query.data,
+ PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+
+ PQclear(res);
+
+ len = PQgetCopyData(conn, &buffer, false);
+ if (len == -2)
+ {
+ fprintf(stderr, "%s: cannot to get data: %s\n",
+ prgname, PQerrorMessage(conn));
+ return false;
+ }
+
+ if (len == -1)
+ {
+ fprintf(stderr, "%s: buffer is empty\n",
+ prgname);
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not to execute \"%s\": %s\n",
+ prgname,
+ query.data,
+ PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ len = 0;
+ }
+
+ if (len != size)
+ {
+ fprintf(stderr, "%s: the length of data (%d) is different than expected (%d)\n",
+ prgname, len, size);
+ return false;
+ }
+
+ if (len > 0 && memcmp(values[colnum], buffer, len) != 0)
+ {
+ fprintf(stderr, "%s: the content of loaded data is different than expected\n",
+ prgname);
+ return false;
+ }
+
+ fprintf(stderr, "%s: the test COPY TO was successful (%s, %s, %d)\n",
+ prgname,
+ colnames[colnum],
+ method,
+ size);
+
+ PQfreemem(buffer);
+ termPQExpBuffer(&query);
+
+ pg_free(values[0]);
+ pg_free(values[1]);
+
+ return true;
+}
+
+static PGconn *
+PrepareConnection(char *connectionstr)
+{
+ PQconninfoOption *opts;
+ PQconninfoOption *opt;
+ char *errmsg = NULL;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ int nopts = 0;
+ int i = 0;
+ PGconn *conn;
+
+ opts = PQconninfoParse(connectionstr, &errmsg);
+ if (opts == NULL)
+ {
+ fprintf(stderr, "%s: %s\n", prgname, errmsg);
+ return NULL;
+ }
+
+ for (opt = opts; opt->keyword != NULL; opt++)
+ {
+ if (opt->val != NULL && opt->val[0] != '\0')
+ nopts++;
+ }
+
+ keywords = pg_malloc0((nopts + 1) * sizeof(*keywords));
+ values = pg_malloc0((nopts + 1) * sizeof(*values));
+
+ for (opt = opts; opt->keyword != NULL; opt++)
+ {
+ if (opt->val != NULL && opt->val[0] != '\0')
+ {
+ keywords[i] = opt->keyword;
+ values[i] = opt->val;
+ i++;
+ }
+ }
+
+ conn = PQconnectdbParams(keywords, values, false);
+ if (!conn)
+ {
+ fprintf(stderr, "%s: could not connect to server\n",
+ prgname);
+ return NULL;
+ }
+
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "%s: could not connect to server: %s",
+ prgname, PQerrorMessage(conn));
+ PQfinish(conn);
+ return NULL;
+ }
+
+ pg_free(values);
+ pg_free(keywords);
+ PQconninfoFree(opts);
+
+ return conn;
+}
+
+static bool
+CreateTable(PGconn *conn)
+{
+ PGresult *res;
+ PQnoticeProcessor prev_notproc;
+
+ prev_notproc = PQsetNoticeProcessor(conn, terseNoticeProcessor, NULL);
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS copy_raw");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not drop table \"copy_raw\": %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ PQsetNoticeProcessor(conn, prev_notproc, NULL);
+
+ res = PQexec(conn, "CREATE TABLE copy_raw(id serial, a text, b bytea)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not create table table \"copy_raw\": %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ return true;
+}
+
+static bool
+DropTable(PGconn *conn)
+{
+ PGresult *res;
+
+ res = PQexec(conn, "DROP TABLE copy_raw");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not drop table \"copy_raw\": %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ return true;
+}
+
+/*
+ * returns result of lastval()
+ *
+ * returns false on error
+ */
+static bool
+get_lastval(PGconn *conn, int *lastval)
+{
+ PGresult *res;
+ char *val;
+
+ res = PQexec(conn, "SELECT lastval()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "%s: could not to call lastval function: %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+
+ val = PQgetvalue(res, 0, 0);
+ Assert(val != NULL);
+
+ *lastval = atoi(val);
+
+ PQclear(res);
+
+ return true;
+}
+
+
+/*
+ * does COPY FROM RAW
+ *
+ */
+static bool
+DoCopyFromBuffer(PGconn *conn, int colnum, const char *method, const char *buffer, int size, int *id)
+{
+ PQExpBufferData query;
+ PGresult *res;
+ char *colnames[] = {"a", "b"};
+ int copy_data_res;
+ const char *errormsg = NULL;
+
+ res = PQexec(conn, "TRUNCATE TABLE copy_raw");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not to truncate table \"copy_raw\": %s",
+ prgname, PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+ PQclear(res);
+
+ initPQExpBuffer(&query);
+ appendPQExpBuffer(&query,
+ "COPY copy_raw(%s) FROM stdin (FORMAT %s)",
+ colnames[colnum],
+ method);
+
+ res = PQexec(conn, query.data);
+
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "%s: could not to execute \"%s\": %s\n",
+ prgname,
+ query.data,
+ PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+
+ PQclear(res);
+
+ copy_data_res = PQputCopyData(conn, buffer, size);
+ if (copy_data_res == -1)
+ {
+ fprintf(stderr, "%s: could not to send data: %s\n",
+ prgname, PQerrorMessage(conn));
+ return false;
+ }
+
+ copy_data_res = PQputCopyEnd(conn, errormsg);
+ if (errormsg != NULL)
+ {
+ fprintf(stderr, "%s: could not to send data: %s\n",
+ prgname, errormsg);
+ return false;
+ }
+ if (copy_data_res == -1)
+ {
+ fprintf(stderr, "%s: could not to send data: %s\n",
+ prgname, PQerrorMessage(conn));
+ return false;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "%s: could not to execute \"%s\": %s\n",
+ prgname,
+ query.data,
+ PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+
+ PQclear(res);
+ termPQExpBuffer(&query);
+
+ if (!get_lastval(conn, id))
+ return false;
+
+ return true;
+}
+
+/*
+ * Send data to server with COPY TO RAW, then get data back with SELECT BINARY
+ * and compare data.
+ */
+static bool
+CopyFromTest(PGconn *conn, int colnum, const char *method, int size)
+{
+ char *colnames[] = {"a", "b"};
+ char *values[2];
+ PGresult *res;
+ int id;
+ PQExpBufferData query;
+ char *buffer;
+ int len;
+
+ PrepareTestValues(values, size);
+
+ if (!DoCopyFromBuffer(conn, colnum, method, (const char *) values[colnum], size, &id))
+ return false;
+
+ initPQExpBuffer(&query);
+ appendPQExpBuffer(&query,
+ "SELECT %s FROM copy_raw WHERE id = %d",
+ colnames[colnum],
+ id);
+
+ res = PQexecParams(conn, query.data,
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 1);
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "%s: could not to execute \"%s\": %s\n",
+ prgname,
+ query.data,
+ PQerrorMessage(conn));
+ PQclear(res);
+ return false;
+ }
+
+ termPQExpBuffer(&query);
+
+ buffer = PQgetvalue(res, 0, 0);
+ len = PQgetlength(res, 0, 0);
+
+ if (len != size)
+ {
+ fprintf(stderr, "%s: the length of data (%d) is different than expected (%d)\n",
+ prgname, len, size);
+ PQclear(res);
+ return false;
+ }
+
+ if (len > 0 && memcmp(values[colnum], buffer, len) != 0)
+ {
+ fprintf(stderr, "%s: the content of stored data is different than expected\n",
+ prgname);
+ PQclear(res);
+ return false;
+ }
+
+ PQclear(res);
+
+ pg_free(values[0]);
+ pg_free(values[1]);
+
+ fprintf(stderr, "%s: the test COPY FROM was successful (%s, %s, %d)\n",
+ prgname,
+ colnames[colnum],
+ method,
+ size);
+
+ return true;
+}
+
+int
+main(int argc, char *argv[])
+{
+ PGconn *conn;
+ int sizes[] = {0, 1, 10, 1000, 10000, 100000, 1000000};
+ const char *method[] = {"raw_text", "raw_binary"};
+ int i;
+ int j;
+
+ if (argc != 2)
+ {
+ fprintf(stderr, "%s: usage %s connection string\n",
+ prgname, prgname);
+ exit(1);
+ }
+
+ conn = PrepareConnection(argv[1]);
+ if (conn == NULL)
+ exit(1);
+
+ if (!CreateTable(conn))
+ {
+ PQfinish(conn);
+ exit(1);
+ }
+
+ for (i = 0; i < 7; i++)
+ {
+ for (j = 0; j < 2; j++)
+ {
+ if (!CopyToTest(conn, j, method[j], sizes[i]))
+ {
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+ }
+
+ for (i = 0; i < 7; i++)
+ {
+ for (j = 0; j < 2; j++)
+ {
+ if (!CopyFromTest(conn, j, method[j], sizes[i]))
+ {
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+ }
+
+ if (!DropTable(conn))
+ {
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQfinish(conn);
+
+ fprintf(stderr, "COPY RAW tests done\n");
+
+ exit(0);
+}
diff --git a/src/interfaces/libpq/test/copy-raw-regress.pl b/src/interfaces/libpq/test/copy-raw-regress.pl
new file mode 100644
index 0000000..7b1714a
--- /dev/null
+++ b/src/interfaces/libpq/test/copy-raw-regress.pl
@@ -0,0 +1,48 @@
+#!/usr/bin/perl -w
+
+use strict;
+
+# use of SRCDIR/SUBDIR is required for supporting VPath builds
+my $srcdir = $ENV{'SRCDIR'} or die 'SRCDIR environment variable is not set';
+my $subdir = $ENV{'SUBDIR'} or die 'SUBDIR environment variable is not set';
+
+my $expected_out = "$srcdir/$subdir/copy-raw-expected.out";
+
+# the output file should land in the build_dir of VPath, or just in
+# the current dir, if VPath isn't used
+my $regress_out = "copy-raw-regress.out";
+
+# save STDOUT/ERR and redirect both to regress.out
+open(OLDOUT, ">&", \*STDOUT) or die "can't dup STDOUT: $!";
+open(OLDERR, ">&", \*STDERR) or die "can't dup STDERR: $!";
+
+open(STDOUT, ">", $regress_out)
+ or die "can't open $regress_out for writing: $!";
+open(STDERR, ">&", \*STDOUT) or die "can't dup STDOUT: $!";
+
+print "trying\n";
+system("./copy-raw-regress \"postgresql://\"");
+print "\n";
+
+# restore STDOUT/ERR so we can print the outcome to the user
+open(STDERR, ">&", \*OLDERR) or die; # can't complain as STDERR is still duped
+open(STDOUT, ">&", \*OLDOUT) or die "can't restore STDOUT: $!";
+
+my $diff_status = system(
+ "diff -c \"$srcdir/$subdir/$regress_out\" $expected_out >regress.diff");
+
+print "=" x 70, "\n";
+if ($diff_status == 0)
+{
+ print "All tests passed\n";
+ exit 0;
+}
+else
+{
+ print <<EOF;
+FAILED: the test result differs from the expected output
+
+Review the difference in "$subdir/regress.diff"
+EOF
+ exit 1;
+}
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f6260a..3d36dd3 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -466,3 +466,17 @@ DROP FUNCTION truncate_in_subxact();
DROP TABLE x, y;
DROP FUNCTION fn_x_before();
DROP FUNCTION fn_x_after();
+CREATE TABLE x(a bytea);
+INSERT INTO x VALUES('\x41484f4a0a');
+INSERT INTO x VALUES('\x41484f4a0a');
+-- should to fail
+COPY (SELECT a,a FROM x LIMIT 1) TO STDOUT (FORMAT raw_binary);
+ERROR: Single column result/target is required in RAW_TEXT/RAW_BINARY mode
+COPY (SELECT a FROM x) TO STDOUT (FORMAT raw_binary);
+AHOJ
+AHOJ
+ERROR: single row result is required by RAW_TEXT/RAW_BINARY mode
+-- should be ok
+COPY (SELECT a FROM x LIMIT 1) TO STDOUT (FORMAT raw_binary);
+AHOJ
+DROP TABLE x;
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.source
index cb13606..085ae36 100644
--- a/src/test/regress/input/copy.source
+++ b/src/test/regress/input/copy.source
@@ -133,3 +133,82 @@ this is just a line full of junk that would error out if parsed
\.
copy copytest3 to stdout csv header;
+
+-- copy raw
+CREATE TABLE x(a bytea);
+INSERT INTO x VALUES('\x41484f4a0a');
+SELECT length(a) FROM x;
+
+INSERT INTO x VALUES('\x41484f4a0a');
+
+-- should to fail
+COPY (SELECT a,a FROM x LIMIT 1) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+COPY (SELECT a FROM x) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+
+-- should be ok
+COPY (SELECT a FROM x LIMIT 1) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+TRUNCATE x;
+COPY x FROM '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+SELECT length(a) FROM x;
+COPY x TO stdout (FORMAT raw_binary);
+
+TRUNCATE x;
+
+\COPY x FROM '@abs_builddir@/results/raw.data' (FORMAT raw_binary)
+SELECT length(a) FROM x;
+COPY x TO stdout (FORMAT raw_binary);
+
+\COPY x TO '@abs_builddir@/results/raw2.data' (FORMAT raw_binary)
+TRUNCATE x;
+
+\COPY x FROM '@abs_builddir@/results/raw2.data' (FORMAT raw_binary)
+SELECT length(a) FROM x;
+COPY x TO stdout (FORMAT raw_binary);
+
+-- test big file
+TRUNCATE x;
+-- use different mechanism for load to bytea
+\lo_import @abs_builddir@/data/hash.data
+\set lo_oid :LASTOID
+INSERT INTO x VALUES(lo_get(:lo_oid));
+\lo_unlink :lo_oid
+
+COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary)
+
+SELECT md5(a), length(a) FROM x;
+
+TRUNCATE x;
+COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary);
+COPY x TO '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+\COPY x TO '@abs_builddir@/results/hash3.data' (FORMAT raw_binary)
+
+-- read again
+COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/results/hash3.data' (FORMAT raw_binary)
+-- cross
+COPY x FROM '@abs_builddir@/results/hash3.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary)
+
+SELECT md5(a), length(a) FROM x;
+
+DROP TABLE x;
+
+-- insert into multicolumn table
+CREATE TABLE x(id serial, a bytea, b bytea);
+
+-- should fail, too much columns
+COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+
+-- should work
+COPY x(a) FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+COPY x(b) FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+SELECT id, md5(a), md5(b) FROM x;
+
+-- test raw_text
+COPY (SELECT a FROM x WHERE id = 1) TO '@abs_builddir@/results/hash4.data' (FORMAT raw_text);
+COPY x(a) FROM '@abs_builddir@/results/hash4.data' (FORMAT raw_text);
+SELECT id, md5(a) FROM x WHERE id = lastval();
+
+DROP TABLE x;
+
diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.source
index b7e372d..e34bbab 100644
--- a/src/test/regress/output/copy.source
+++ b/src/test/regress/output/copy.source
@@ -95,3 +95,114 @@ copy copytest3 to stdout csv header;
c1,"col with , comma","col with "" quote"
1,a,1
2,b,2
+-- copy raw
+CREATE TABLE x(a bytea);
+INSERT INTO x VALUES('\x41484f4a0a');
+SELECT length(a) FROM x;
+ length
+--------
+ 5
+(1 row)
+
+INSERT INTO x VALUES('\x41484f4a0a');
+-- should to fail
+COPY (SELECT a,a FROM x LIMIT 1) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+ERROR: Single column result/target is required in RAW_TEXT/RAW_BINARY mode
+COPY (SELECT a FROM x) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+ERROR: single row result is required by RAW_TEXT/RAW_BINARY mode
+-- should be ok
+COPY (SELECT a FROM x LIMIT 1) TO '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+TRUNCATE x;
+COPY x FROM '@abs_builddir@/results/raw.data' (FORMAT raw_binary);
+SELECT length(a) FROM x;
+ length
+--------
+ 5
+(1 row)
+
+COPY x TO stdout (FORMAT raw_binary);
+AHOJ
+TRUNCATE x;
+\COPY x FROM '@abs_builddir@/results/raw.data' (FORMAT raw_binary)
+SELECT length(a) FROM x;
+ length
+--------
+ 5
+(1 row)
+
+COPY x TO stdout (FORMAT raw_binary);
+AHOJ
+\COPY x TO '@abs_builddir@/results/raw2.data' (FORMAT raw_binary)
+TRUNCATE x;
+\COPY x FROM '@abs_builddir@/results/raw2.data' (FORMAT raw_binary)
+SELECT length(a) FROM x;
+ length
+--------
+ 5
+(1 row)
+
+COPY x TO stdout (FORMAT raw_binary);
+AHOJ
+-- test big file
+TRUNCATE x;
+-- use different mechanism for load to bytea
+\lo_import @abs_builddir@/data/hash.data
+\set lo_oid :LASTOID
+INSERT INTO x VALUES(lo_get(:lo_oid));
+\lo_unlink :lo_oid
+COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary)
+SELECT md5(a), length(a) FROM x;
+ md5 | length
+----------------------------------+--------
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+(3 rows)
+
+TRUNCATE x;
+COPY x FROM '@abs_builddir@/data/hash.data' (FORMAT raw_binary);
+COPY x TO '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+\COPY x TO '@abs_builddir@/results/hash3.data' (FORMAT raw_binary)
+-- read again
+COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/results/hash3.data' (FORMAT raw_binary)
+-- cross
+COPY x FROM '@abs_builddir@/results/hash3.data' (FORMAT raw_binary);
+\COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary)
+SELECT md5(a), length(a) FROM x;
+ md5 | length
+----------------------------------+--------
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+ e446fe6ea5a347e69670633412c7f8cb | 153749
+(5 rows)
+
+DROP TABLE x;
+-- insert into multicolumn table
+CREATE TABLE x(id serial, a bytea, b bytea);
+-- should fail, too much columns
+COPY x FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+ERROR: Single column result/target is required in RAW_TEXT/RAW_BINARY mode
+-- should work
+COPY x(a) FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+COPY x(b) FROM '@abs_builddir@/results/hash2.data' (FORMAT raw_binary);
+SELECT id, md5(a), md5(b) FROM x;
+ id | md5 | md5
+----+----------------------------------+----------------------------------
+ 1 | e446fe6ea5a347e69670633412c7f8cb |
+ 2 | | e446fe6ea5a347e69670633412c7f8cb
+(2 rows)
+
+-- test raw_text
+COPY (SELECT a FROM x WHERE id = 1) TO '@abs_builddir@/results/hash4.data' (FORMAT raw_text);
+COPY x(a) FROM '@abs_builddir@/results/hash4.data' (FORMAT raw_text);
+SELECT id, md5(a) FROM x WHERE id = lastval();
+ id | md5
+----+----------------------------------
+ 3 | e446fe6ea5a347e69670633412c7f8cb
+(1 row)
+
+DROP TABLE x;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index 39a9deb..7e22ee4 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -333,3 +333,16 @@ DROP FUNCTION truncate_in_subxact();
DROP TABLE x, y;
DROP FUNCTION fn_x_before();
DROP FUNCTION fn_x_after();
+
+CREATE TABLE x(a bytea);
+INSERT INTO x VALUES('\x41484f4a0a');
+INSERT INTO x VALUES('\x41484f4a0a');
+
+-- should to fail
+COPY (SELECT a,a FROM x LIMIT 1) TO STDOUT (FORMAT raw_binary);
+COPY (SELECT a FROM x) TO STDOUT (FORMAT raw_binary);
+
+-- should be ok
+COPY (SELECT a FROM x LIMIT 1) TO STDOUT (FORMAT raw_binary);
+
+DROP TABLE x;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
