Re: [HACKERS] Why the asprintf patch is still breaking the buildfarm
On 22/10/2013 09:58, Tom Lane wrote: So I returned from vacation only to find that the buildfarm has a bad case of acne. All the Windows members are red or pink, and have been for awhile. Sigh. After some research I believe that I understand the reason for the CHECK failures, at least: 1. src/port/asprintf.c exhibits a truly touching faith that vsnprintf will report exactly the number of bytes that would have been required, even if the buffer is not that large. While this is what is specified in recent versions of the POSIX standard, older platforms have much sketchier behavior. Just to be pedantic, this is required by C99. > [...] Regards Manlio Perillo -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposed TODO: add support for "any" for PL/PythonU and PL/Perl
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Il 14/05/2013 20:24, Josh Berkus ha scritto: > Hackers, > > I'd like to add the following todo items to the TODO list: > > PL/Python: > * add support for anyelement and anyarray to PL/Python > * add support for VARIADIC "ANY" to PL/Python > > PL/Perl: > * add support for anyelement and anyarray to PL/Perl > * add support for VARIADIC "ANY" to PL/Perl > > The reason for this is that both Python and Perl are loosely typed > languages, I don't know about Perl, but, to be precise, Python is not a loosely typed language. It is a strongly typed dynamic language. > [...] Regards Manlio Perillo -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlGTq1sACgkQscQJ24LbaUQdogCeNd1Boj/w2UaQIvDuiSU41Lrn UnoAn3vuehDS2+woqVyahwktm5sYCtGG =mwuk -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [RFC] ideas for a new Python DBAPI driver (was Re: [HACKERS] libpq test suite)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Il 15/02/2013 02:45, Andrew McNamara ha scritto:
>> For my Python DBAPI2 PostgreSQL driver I plan the following optimizations:
>
> I suggest you have a look at my Python ocpgdb driver:
>
> http://code.google.com/p/ocpgdb/
>
Thanks, I did not know it.
> It uses the v3 binary protocol exclusively (to avoid the usual escaping
> security issues). A number of gotchyas were discovered along the way -
> in particular, you must be a lot more careful about types (as you note
> in a later reply).
Note that this query:
curs.execute('SELECT * FROM foo WHERE bah > %s', '2006-1-1')
is IMHO incorrect, as per DBAPI 2.0:
http://www.python.org/dev/peps/pep-0249/#type-objects-and-constructors
> There were also some issues with the v3 protocol,
> most of which have been fixed now.
>
I hope the issues are discussed in the commit log messages; I'm creating
a git local mirror of the svn repository.
> ocpgdb does not implement everything, just the bits I needed. That said,
> other people/projects are using it in production, and it's proven to be
> fast and stable.
>
Regards Manlio
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlEemTYACgkQscQJ24LbaUTujgCfZhrNTsqy/PvRJ4qwLVqy8QVT
lNwAnjsJooEv/vss32RNMKEISOdZ16F1
=nKO5
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [RFC] ideas for a new Python DBAPI driver (was Re: [HACKERS] libpq test suite)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Il 14/02/2013 20:01, Peter Eisentraut ha scritto:
> On 2/14/13 9:23 AM, Manlio Perillo wrote:
>> 1) always use PQsendQueryParams functions.
>>
>>This will avoid having to escape parameters, as it is done in
>>psycopg2
>>(IMHO it still use simple query protocol for compatibility purpose)
>
> I think the reason this doesn't work is that in order to prepare a query
> you need to know the parameter types, but you don't know that in Python,
> or at least with the way the DB-API works.
Hint: .setinputsizes.
In my implementation, prepared queries are **only** used if user calls
setinputsizes; if setinputsizes is not called, preparing a query can
cause performance loss, so it is better to not prepare one.
> For example, if you write
>
> cur.execute("SELECT * FROM tbl WHERE a = %s AND b = %s", (val1, val2))
>
> what types will you pass to PQsendQueryParams?
>
1) if setinputsizes is used, use the type specified here.
2) if setinputsizes is not used, query the driver's type catalog, to
get the postgresql type oid associated with a Python object; e.g.:
pg_type = connection.get_type_by_object(val1)
buf, format = pg_type.output_function(val1)
param_oid = pg_type.oid
> You can make some obvious heuristics, but this sort of thing can get
> complicated pretty quickly.
A non trivial case if when val is a list, that should be mapped to a
PostgreSQL array.
However, you can always set the oid to 0, and let PostgreSQL deduce the
type, as it is done in psycopg2. If user called setinputsizes, we are
happy.
Regards Manlio
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlEdO54ACgkQscQJ24LbaURFWACgkG3dkDYUA3tHooiXpGDf8hm0
Fz0AnjhFDwJU/XSqtgPyHwuHw9+GiRlv
=m68+
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [RFC] ideas for a new Python DBAPI driver (was Re: [HACKERS] libpq test suite)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Il 14/02/2013 18:18, Jonathan Rogers ha scritto: > A number of the described features sound quite useful. Is it not > practical to extend an existing library such as psycopg2? I suspect there are compatibility issues. > What method > will you use to call libpq functions? As you are no doubt aware, > psycopg2 uses the traditional CPython API but there is a fork which uses > ctypes instead, enabling it to work with CPython and Pypy. > I'm implementing a prototype version, in pure Python with ctypes. When the prototype is ready, I will implement a CPython extension module (since the purpose of writing yet another driver is to make it more efficient than the current best driver). I will also discuss about porting some of the features to psycopg2 (but first I need to have a working driver). > Manlio Perillo wrote: > >> For my Python DBAPI2 PostgreSQL driver I plan the following optimizations: > >> 2) when the driver detects a Python string is being sent to the >>database, use binary format. > > What exactly do you mean by "Python string"? > A Python string object. The libpq interface will implement the functions pg_str_encode(string object) -> bytes pg_str_decode(bytes, result object) -> string object as described in http://wiki.postgresql.org/wiki/Driver_development but with some differences. The pg_str_encode function will return a tuple with the raw bytes and the "suggested" parameter format. As an example, pg_str_encode() will return the byte string and 1 (since it is a binary string, and I want to avoid to use PQescapeBytea function). For the DBAPI interface, I plan to implement a generic API to map PostgreSQL types to Python types (different from the one implemented in psycopg2); something like: pgtype = connection.create_type_map( , , , , , ) but I have yet to decide how to detect the param format to use. Maybe there will be only one output function, that will decide the best format to be used. Regards Manlio -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEdIYQACgkQscQJ24LbaUSfBACfWR7eD6pdyipC3/fljUiuelx6 GV4An3agVt4tx0E/JDUvO0iM8/BiZT1o =xckI -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[RFC] ideas for a new Python DBAPI driver (was Re: [HACKERS] libpq test suite)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Il 14/02/2013 14:06, Albe Laurenz ha scritto: > Manlio Perillo wrote: >> Sorry for the question, but where can I find the libpq test suite? >> I can not find it in the PostgreSQL sources; it seems that there are >> only some examples, in src/test/examples. > > The regression tests are in src/interfaces/libpq/test > and currently contain only URL parsing tests. > Ok, thanks. Since I'm not sure if I should add a new test here, I'll use the test suite of my project, since it contains an (almost) 1:1 wrapper around libpq. >> I'm planning to add some new features to libpq: >> >> * make PQsendPrepare send a "Describe Portal" protocol message >> * add support for setting per column binary result format > > I suggested exactly that here: > http://www.postgresql.org/message-id/d960cb61b694cf459dcfb4b0128514c208a4e...@exadv11.host.magwien.gv.at > and met resistance: > - one can use libpqtypes > - I couldn't find a convincing use case > - it clutters up the API > For my Python DBAPI2 PostgreSQL driver I plan the following optimizations: 1) always use PQsendQueryParams functions. This will avoid having to escape parameters, as it is done in psycopg2 (IMHO it still use simple query protocol for compatibility purpose) 2) when the driver detects a Python string is being sent to the database, use binary format. As a special case, this will avoid having to use PQescapeByteaConn when sending binary string (e.g. byte strings in Python 3.x) 3) enable use of prepared statements, but only if the user requested it, using setinputsizes function (used to set the Oids of the parameters) 4) when using a prepared statement, check the Oids of the result tuple. In order to make this efficient, I proposed a patch to send a Describe Portal message in PQsendPrepare function. When the driver detects that one of the result column is a string type, set the result format for that column to binary. As a special case, this will avoid having to use PQunescapeBytea when receiving a bytea data. This is currently impossible, using libpq API. 5) when returning the result set of a query, after a call to cursor.fetchall(), do not convert all the data to Python objects. This will be done only "on request". This should optimize memory usage, as reported in: http://wiki.postgresql.org/wiki/Python_PostgreSQL_Driver_TODO 6) make available the use of PQsetSingleRowMode, to optimize large result set (as an option to the connection.cursor method) 7) as a generalization of PQsetSingleRowMode, expose in libpq API some of protocol internal portal API. One possible idea is to add a PQsetRowSize function, that will set the size of the result set, to be used in the Execute protocol message (currently libpq always set it to 0, to get the entire result set, and it does not support the Portal Suspended message) This will avoid having to use named cursor, as it is done in psycopg. I'll try to make a patch to check if this is feasible, can be done efficiently, and the new API has a minimal impact on existing API Note that I will have to code these features, in order to check they will work as I expect. > [...] >> >> [1] A new Python PostgreSQL driver, implemented following >> http://wiki.postgresql.org/wiki/Driver_development >> and with many optimization (compared to psycopg2) enabled by the >> use of the extended query protocol > > I think that you'll need to explain in more detail why > your proposed additions would be necessary for your project. > Especially since many good drivers have been written against > libpq as it is. > > Yours, > Laurenz Albe > Thanks Manlio Perillo -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEc81MACgkQscQJ24LbaURO9ACfctOREoaAtMDm06Sg+qv5jesj iW0An1CVAOaHzYaSn+P1AIJvXpI7nVT0 =rK4j -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] libpq test suite
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi. Sorry for the question, but where can I find the libpq test suite? I can not find it in the PostgreSQL sources; it seems that there are only some examples, in src/test/examples. I'm planning to add some new features to libpq: * make PQsendPrepare send a "Describe Portal" protocol message * add support for setting per column binary result format * add direct support for portals (I think I need these for a project I'm working on [1]), How can I check if the new code does not break existing usage? [1] A new Python PostgreSQL driver, implemented following http://wiki.postgresql.org/wiki/Driver_development and with many optimization (compared to psycopg2) enabled by the use of the extended query protocol Thanks Manlio Perillo -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEc0iAACgkQscQJ24LbaURiVQCfaANOEaIJHdq/ZoQYx1Hu8wZr On8An202Fqc928K9NuDRen/0B06sN/RS =Qngt -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] send Describe Portal message in PQsendPrepare
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Il 11/02/2013 22:39, Tom Lane ha scritto: > Manlio Perillo writes: >> What is the reason why PQsendPrepare function does not send a >> Describe Portal message? > > That would add a round trip, no? > Well, no. An extra round trip is required with current implementation, since I need to call PQsendPrepare, wait for server response, call PQsendDescribePrepared, wait for server response. >> Sending a Describe Portal message, make it possible for PQsendPrepare >> function to *return* a PGresult with more useful informations, instead >> of just the result status. > > That's *definitely* wrong, because the entire point of the PQsend > functions is they don't wait for a server response. > The PQsendQueryParams sends the following protocol messages: * Parse * Bind * Describe Portal * Execute * Sync and of course does not wait for each response, since this is done in the state machine. I noted that PQsendQueryParams sends a Describe Portal message, and I found it strange that the same is not done by PQsendPrepare. I wrote the patch to check if this was done due to some technical reason, but this seems to not be the case. > regards, tom lane Regards Manlio Perillo -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEaLvkACgkQscQJ24LbaUSBgACgjASGXyTl+rpHWGAGk5nm7Fnj T68Anin9iEfbLw75ObHJxU6yfIazEZDS =ZmPu -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] send Describe Portal message in PQsendPrepare
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi.
What is the reason why PQsendPrepare function does not send a
Describe Portal message?
Just as a proof of concept, I wrote a very simple patch, attached, and
it *seems* to work.
Sending a Describe Portal message, make it possible for PQsendPrepare
function to *return* a PGresult with more useful informations, instead
of just the result status.
Thanks Manlio Perillo
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlEZT8IACgkQscQJ24LbaUT0VwCcCGKGD6CYrb53B/z+1SdB8vX5
cWkAnjOkSxCVFRo32qzIegY1UhddxBcI
=D+Qj
-END PGP SIGNATURE-
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 77124ef..f9cee86 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1265,6 +1265,13 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
+ /* construct the Describe Portal message */
+ if (pqPutMsgStart('D', false, conn) < 0 ||
+ pqPutc('S', conn) < 0 ||
+ pqPuts(stmtName, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
/* construct the Sync message */
if (pqPutMsgStart('S', false, conn) < 0 ||
pqPutMsgEnd(conn) < 0)
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index c605bcd..cca9f90 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -231,19 +231,6 @@ pqParseInput3(PGconn *conn)
conn->asyncStatus = PGASYNC_READY;
break;
case '1': /* Parse Complete */
- /* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
- {
- if (conn->result == NULL)
- {
- conn->result = PQmakeEmptyPGresult(conn,
- PGRES_COMMAND_OK);
- if (!conn->result)
-return;
- }
- conn->asyncStatus = PGASYNC_READY;
- }
- break;
case '2': /* Bind Complete */
case '3': /* Close Complete */
/* Nothing to do for these message types */
@@ -266,7 +253,8 @@ pqParseInput3(PGconn *conn)
break;
case 'T': /* Row Description */
if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ conn->queryclass == PGQUERY_DESCRIBE ||
+ conn->queryclass == PGQUERY_PREPARE)
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -299,7 +287,8 @@ pqParseInput3(PGconn *conn)
* instead of TUPLES_OK. Otherwise we can just ignore
* this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->queryclass == PGQUERY_DESCRIBE ||
+ conn->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -455,7 +444,8 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->queryclass == PGQUERY_DESCRIBE ||
+ conn->queryclass == PGQUERY_PREPARE)
{
if (conn->result)
result = conn->result;
@@ -562,7 +552,8 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->queryclass == PGQUERY_DESCRIBE ||
+ conn->queryclass == PGQUERY_PREPARE)
{
conn->asyncStatus = PGASYNC_READY;
return 0;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
