Re: [HACKERS] Vote totals for SET in aborted transaction
Marc G. Fournier writes: > Myself, I wonder why Oracle went the route they went ... does anyone have > access to a Sybase / Informix system, to confirm how they do it? Is > Oracle the 'odd man out', or are we going to be that? *Adding* something > (ie. DROP TABLE rollbacks) that nobody appears to have is one thing ... > but changing the behaviour is a totally different .. FWIW, Ingres also doesn't rollback SET. However all its SET functionality is the sort of stuff you wouldn't assume to rollback: auto-commit connection journaling logging session work locations maxidle You cannot do something sane like modify the date output through SET. Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [INTERFACES] ECPG: FETCH ALL|n FROM cursor - Memory allocation?
Okay, lets see if i've got this right... If I allocate the memory before the FETCH then I (naturally) free it. However If I NULL initialise the pointer then libecpg will allocate the memory and I must NOT free it - libecpg will free it automatically... Yeah? I think this highlights the need for some documentation on this aspect. Regards, Lee Kindness. Christof Petig writes: > Michael Meskes wrote: > > On Thu, Apr 25, 2002 at 12:42:00PM +0100, Lee Kindness wrote: > >>Should the input pointers be NULL initialised? How should the memory > >>be freed? > > > > > > A simple free() will do. You also can free all automatically > > allocated memory from the most recent executed statement by calling > > ECPGfree_auto_mem(). But this is not documented and will never be. > > > > The correct way is to free(array1) and free(array2) while libecpg will > > free the internal structures when the next statement is executed. > > Never, never mix these two! ECPGfree_auto_mem will free even memory > which has already been free'd by the user, perhaps we should get rid of > this method (any allocated memory regions are stored in a list, if you > never call ECPGfree_auto_mem, this list grows and grows). > > Christof > ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Redhat 7.3 time manipulation bug
Thomas Lockhart writes: > Right. IBM. > > Most likely they have fixed it by now ... > Nope, though I don't know for sure. Anyone here have a recent AIX > machine to test? Well on AIX 4.3.3 the output from Lamar's earlier test program is: The system thinks 11/30/1969 is a timestamp of -1 and tm_isdst is left at -1... I could boot the machine into 5.0 too, but going by the AIX 5L manpages it still returns -1: Note: The mktime subroutine cannot convert time values before 00:00:00 UTC, January 1, 1970 and after 03:14:07 UTC, January 19, 2038. And getting an Irix 5.3 box up and running would be a chore! Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [BUGS] Bug #640: ECPG: inserting float numbers
Bruce, the attached source reproduces this on 7.2, I don't have a
later version at hand to test if it's been fixed:
createdb floattest
echo "CREATE TABLE tab1(col1 FLOAT);" | psql floattest
ecpg insert-float.pgc
gcc insert-float.c -lecpg -lpq
./a.out floattest
results in:
col1: -0.06
*!*!* Error -400: 'ERROR: parser: parse error at or near "a"' in line 21.
and in epcgdebug:
[29189]: ECPGexecute line 21: QUERY: insert into tab1 ( col1 ) values (
-6.002122251e-06A ) on connection floattest
[29189]: ECPGexecute line 21: Error: ERROR: parser: parse error at or near "a"
[29189]: raising sqlcode -400 in line 21, ''ERROR: parser: parse error at or near
"a"' in line 21.'.
Regards, Lee Kindness.
Bruce Momjian writes:
> Has this been addressed? Can you supply a reproducable example?
> Edward Pilipczuk wrote:
> > On Monday, 22 April 2002 18:41, you wrote:
> > > Edward ([EMAIL PROTECTED]) reports a bug with a severity of 1
> > > ECPG: inserting float numbers
> > > Inserting records with single precision real variables having small value
> > > (range 1.0e-6 or less) frequently results in errors in ECPG translations
> > > putting into resulted sql statement unexpected characters => see fragments
> > > of sample code and ECPGdebug log where after value of rate variable the
> > > unexpected character '^A' appears
> > >
> > > Sample Code
> > > [ snip ]
#include
EXEC SQL INCLUDE sqlca;
int main(int argc, char **argv)
{
EXEC SQL BEGIN DECLARE SECTION;
char *db = argv[1];
float col1;
EXEC SQL END DECLARE SECTION;
FILE *f;
if( (f = fopen("ecpgdebug", "w" )) != NULL )
ECPGdebug(1, f);
EXEC SQL CONNECT TO :db;
EXEC SQL BEGIN;
col1 = -6e-06;
printf("col1: %f\n", col1);
EXEC SQL INSERT INTO tab1(col1) VALUES (:col1);
if( sqlca.sqlcode < 0 )
{
fprintf(stdout, "*!*!* Error %ld: %s\n", sqlca.sqlcode, sqlca.sqlerrm.sqlerrmc);
EXEC SQL ABORT;
EXEC SQL DISCONNECT;
return( 1 );
}
else
{
EXEC SQL COMMIT;
EXEC SQL DISCONNECT;
return( 0 );
}
}
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] [BUGS] Bug #640: ECPG: inserting float numbers
Bruce, after checking the libecpg source i'm fairly sure the problem is due to the malloc buffer that the float is being sprintf'd into being too small... It is always allocated 20 bytes but with a %.14g printf specifier -6e-06 results in 20 characters: -6.0e-06 and the NULL goes... bang! I guess the '-' wasn't factored in and 21 bytes would be enough. Patch against current CVS (but untested): Index: src/interfaces/ecpg/lib/execute.c === RCS file: /projects/cvsroot/pgsql/src/interfaces/ecpg/lib/execute.c,v retrieving revision 1.36 diff -r1.36 execute.c 703c703 < if (!(mallocedval = ECPGalloc(var->arrsize * 20, stmt->lineno))) --- > if (!(mallocedval = ECPGalloc(var->arrsize * 21, >stmt->lineno))) 723c723 < if (!(mallocedval = ECPGalloc(var->arrsize * 20, stmt->lineno))) --- > if (!(mallocedval = ECPGalloc(var->arrsize * 21, >stmt->lineno))) Lee. Bruce Momjian writes: > > OK, I have reproduced the problem on my machine: > > #$ ./a.out floattest > col1: -0.06 > *!*!* Error -220: No such connection NULL in line 21. > > Wow, how did that "A" get into the query string: > > insert into tab1 ( col1 ) values ( -6.002122251e-06A ) > > Quite strange. Michael, any ideas? > > Lee Kindness wrote: > Content-Description: message body text > > > Bruce, the attached source reproduces this on 7.2, I don't have a > > later version at hand to test if it's been fixed: > > > > createdb floattest > > echo "CREATE TABLE tab1(col1 FLOAT);" | psql floattest > > ecpg insert-float.pgc > > gcc insert-float.c -lecpg -lpq > > ./a.out floattest > > > > results in: > > > > col1: -0.06 > > *!*!* Error -400: 'ERROR: parser: parse error at or near "a"' in line 21. > > > > and in epcgdebug: > > > > [29189]: ECPGexecute line 21: QUERY: insert into tab1 ( col1 ) values ( >-6.002122251e-06A ) on connection floattest > > [29189]: ECPGexecute line 21: Error: ERROR: parser: parse error at or near "a" > > [29189]: raising sqlcode -400 in line 21, ''ERROR: parser: parse error at or >near "a"' in line 21.'. > > > > Regards, Lee Kindness. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] [BUGS] Bug #640: ECPG: inserting float numbers
Bruce, this error and the one in your earlier post are not indicative of the bug, but rather of the connection failing - looking at the created ecpgdebug file should confirm this. I have since compiled 7.3 with the patch locally and cannot recreate the bug (after messing around with the HBA cfg file - I was getting the same error as you). My command line (with 7.3 sitting in /database/pgsql-test on port 5433 and LD_LIBRARY_PATH setup): /database/pgsql-test/bin/ecpg insert-float.pgc gcc insert-float.c -I/database/pgsql-test/include -L/database/pgsql-test/lib -lecpg -lpq ./a.out floattest@localhost:5433 Regards, Lee Kindness. Bruce Momjian writes: > I am now getting this error: > #$ ./a.out floattest > col1: -0.06 > *!*!* Error -220: No such connection NULL in line 21. > I will wait for Michael to comment on this. > > ------- > > Lee Kindness wrote: > > Lee Kindness writes: > > > and the NULL goes... bang! I guess the '-' wasn't factored in and 21 > > > bytes would be enough. Patch against current CVS (but untested): > > > > Ooops, a context diff is below... > > > > Index: src/interfaces/ecpg/lib/execute.c > > === > > RCS file: /projects/cvsroot/pgsql/src/interfaces/ecpg/lib/execute.c,v > > retrieving revision 1.36 > > diff -c -r1.36 execute.c > > *** src/interfaces/ecpg/lib/execute.c 2002/01/13 08:52:08 1.36 > > --- src/interfaces/ecpg/lib/execute.c 2002/06/11 11:45:35 > > *** > > *** 700,706 > >break; > > #endif /* HAVE_LONG_LONG_INT_64 */ > >case ECPGt_float: > > ! if (!(mallocedval = ECPGalloc(var->arrsize * 20, >stmt->lineno))) > >return false; > > > >if (var->arrsize > 1) > > --- 700,706 > >break; > > #endif /* HAVE_LONG_LONG_INT_64 */ > >case ECPGt_float: > > ! if (!(mallocedval = ECPGalloc(var->arrsize * 21, >stmt->lineno))) > >return false; > > > >if (var->arrsize > 1) > > *** > > *** 720,726 > >break; > > > >case ECPGt_double: > > ! if (!(mallocedval = ECPGalloc(var->arrsize * 20, >stmt->lineno))) > >return false; > > > >if (var->arrsize > 1) > > --- 720,726 > >break; > > > >case ECPGt_double: > > ! if (!(mallocedval = ECPGalloc(var->arrsize * 21, >stmt->lineno))) > >return false; > > > >if (var->arrsize > 1) ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Open 7.3 items
Vince Vielhaber writes: > [ 'user@' patch ] > whim. Then again as long as 7.2.1 is stable enough for me there's > no reason to upgrade 'cuze I damn sure ain't going back and changing > all sorts of programs and scripts that have global users. Having read bits and pieces of this thread, can those in favour confirm that this would be an effect of this patch? If so I fail to see the usefulness of this and indeed it would be very harmful to existing installations! All use of PostgreSQL utilities in scripts for our product always do a '-U sprint' to use a global user, this aids our internal development and makes installation notes for clients easier... Also what effect would adding significance to '@' in the context of usernames have, if any, on the current use of it as a database/host separator (in ECPG, certainly would be useful in the utilities too)? Thanks, Lee. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Open 7.3 items
I'd have thought that if a matching user couldn't be found in the specified database then it would default to searching through the global users? Would be more intuitive... Lee. Bruce Momjian writes: > Sample run: > $ psql -U postgres test > psql: FATAL: user "postgres@test" does not exist > > $ psql -U postgres@ test > Welcome to psql 7.3devel, the PostgreSQL interactive terminal. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] 7.3 gotchas for applications and client libraries
Tom, do you think there is millage in adding functions (at least to contrib) to PostgreSQL to avoid some of the common tasks applications look into pg_* for? For example I recently audited our code here for pg_* access, and managed to create two plpgsql functions to replace all occurrences. They were relatively simple queries to check if a table existed and to check if a column existed, functions for 7.2.x: \echo creating function: column_exists CREATE OR REPLACE FUNCTION column_exists(NAME, NAME) RETURNS BOOLEAN AS ' DECLARE tab ALIAS FOR $1; col ALIAS FOR $2; rec RECORD; BEGIN SELECT INTO rec * FROM pg_class c, pg_attribute a WHERE c.relname = tab AND c.oid = a.attrelid AND a.attnum > 0 AND a.attname = col; IF NOT FOUND THEN RETURN false; ELSE RETURN true; END IF; END; ' LANGUAGE 'plpgsql'; \echo creating function: table_exists CREATE OR REPLACE FUNCTION table_exists(NAME) RETURNS BOOLEAN AS ' DECLARE tab ALIAS FOR $1; rec RECORD; BEGIN SELECT INTO rec * FROM pg_class c WHERE c.relname = tab; IF NOT FOUND THEN RETURN false; ELSE RETURN true; END IF; END; ' LANGUAGE 'plpgsql'; Obviously these need attention when our application targets 7.3 (and thanks for the heads-up), but all changes are localised. Surely these must be fairly common tests and maybe better added to the database server so applications are less dependant on internal catalogues? Any desire for me to polish these two functions up for contrib in 7.3? Actually the Cookbook at http://www.brasileiro.net/postgres/ has similar function which will need attention for 7.3 too, is the eventual plan for this to be folded into the core release? Thanks, Lee. Tom Lane writes: > Bruce suggested that we need a porting guide to help people look for > application and client-library code that will be broken by the changes > in PG 7.3. Here is a first cut at documenting the issues. > Comments welcome --- in particular, what have I missed? > [snip ] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Location of database files?
Is there any way to determine the location of files in a database without being the postgres user? Essentially i'm after the setting of PGDATA so i can then show disk status (df) for that partition. The pg_database catalogue has 'datpath': If the database is stored at an alternative location then this records the location. It's either an environment variable name or an absolute path, depending how it was entered. so I'm really looking for the default location... I could knock together a C function to do this (and indeed another to return the usage stats too), but would like to check first there's no simple way already! Regards, Lee Kindness. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Where to post a new PostgreSQL utility?
Dave Page writes: > Just because I'm curious, is *all* new stuff going to Gborg, and is the > existing /contrib going to migrated there? I'm curious too... If that is to happen then the profile of gborg would need to be massively increased. Currenly the only real link on the 'net to gborg (by searching through http://www.google.com/search?q=link:gborg.postgresql.org) is from the User Lounge (link "Related projects") on the PostgreSQL site. I'd have thought a link on the main left-hand list would be more apt. And gbord needs a search facility if people are going to be able to find anything... Lee. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] COPY syntax
Bruce Momjian writes: > Peter Eisentraut wrote: > > Bruce Momjian writes: > > > > COPY table TO STDOUT WITH BINARY OIDS; > > > > Shouldn't the "binary", being an adjective, be attached to something? > > > Uh, it is attached to WITH? > > Attached to a noun phrase, like "mode" or "output". Note that all the > > other things the typically follow WITH in any command are nouns. > Should we add an optional MODE after BINARY? Are you serious? You'd like to mess up the COPY syntax even further for a purely grammatical reason! A good few months ago I put formward an idea to change (well migrate really) to "COPY TABLE" rather than "COPY" - this would allow a well designed and thoughtout syntax for the new version while retaining old compatibility. egards, Lee Kindness. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] ecpg "problem" ...
Marc,
Marc G. Fournier writes:
> if (ic_flag == 1) {
> /*only select those non-IC/Spyder nodes that has full update set*/
> EXEC SQL DECLARE full_dyn_node CURSOR FOR
> SELECT node_name FROM NODE
> WHERE dynamic_community = 'f' AND ic_flag='n' AND machine_type!=22
> AND node_id != 0 AND NODE_NAME != :nodename;
> }
> else{
> EXEC SQL DECLARE full_dyn_node CURSOR FOR
> SELECT node_name FROM NODE
> WHERE dynamic_community = 'f'
> AND node_id != 0 AND NODE_NAME != :nodename; (line#493)
> }
>
> the above code generates the following error:
>
> The compiler complains:
> ../subapi.pgc:493: ERROR: cursor full_dyn_node already defined
>
> since its envelop'd in an if/else clause, shouldn't it work?
Unfortuantely no, you can only ever have one "EXEC SQL DECLARE" for a
given cursor name due to ecpg/ESQL simple parsing. What you would do
in a situation like this is something like:
if( ic_flag == 1 )
/* only select those non-IC/Spyder nodes that has full update set */
sprintf(stmt, "SELECT node_name FROM NODE WHERE dynamic_community = 'f' AND
ic_flag = 'n' AND machine_type != 22 AND node_id != 0 AND NODE_NAME != %s", nodename);
else
sprintf(stmt, "SELECT node_name FROM NODE WHERE dynamic_community = 'f' AND
node_id != 0 AND NODE_NAME != %s", nodename);
EXEC SQL PREPARE s_statement FROM :stmt;
EXEC SQL DECLARE full_dyn_node CURSOR FOR s_statement;
Regards, Lee.
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [mail] Re: [HACKERS] Native Win32 sources
Al, to be honest I don't think the Windows native would save hassle, rather it'd probably cause more! No disrespect to those doing the version, read on for reasoning... Yes, you get a beta of a Windows native version just now, yes it probably will not be that long till the source is a available... But how long till it's part of a cosha PostgreSQL release? Version 7.4... Could be up to six months... Do you want to run pre-release versions in the meantime? Don't think so, not in a production environment! So, the real way to save hassle is probably a cheap commodity PC with Linux installed... Or settle for the existing, non-native, Windows version. By the way, just to open Office documents? Have you tried OpenOffice? Regards, Lee Kindness. Al Sutton writes: > Is there a rough date for when they'll be available? > > I have a development team at work who currently have an M$-Windows box and a > Linux box each in order to allow them to read M$-Office documents sent to us > and develop against PostgreSQL (which we use in production). > > I know I could have a shared Linux box with multiple databases and have them > bind to that, but one of the important aspects of our application is > response time, and you can't accurately measure response times for code > changes on a shared system. > > Having a Win32 native version would save a lot of hassles for me. > > Al. > > - Original Message - > From: "Bruce Momjian" <[EMAIL PROTECTED]> > > > Ulrich Neumann wrote: > > > Hello, > > > > > > i've read that there are 2 different native ports for Windows > > > somewhere. > > > > > > I've searched for them but didn't found them. Is there anyone who can > > > point me to a link or send me a copy of the sources? > > > > Oh, you are probably asking about the sources. They are not publically > > available yet. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] System Tables
See: http://www.us.postgresql.org/users-lounge/docs/7.2/postgres/catalogs.html for PostgreSQL 7.2.x, for 7.3 see: http://developer.postgresql.org/docs/postgres/catalogs.html Lee. Steve Jackson writes: > Hi > > Has anyone of you a good pointer to a description of where in the system > tables I may find what informations? I try to code a generic procedure > which gets information (like field type, field length, foreign keys...) > about tables and fields from a system table. > > Thank you for your help in advance ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] 7.4 Wishlist
Christopher Kings-Lynne writes: > Just out of interest, if someone was going to pay you to hack on Postgres > for 6 months, what would you like to code for 7.4? Well judging by the hoards on Slashdot, it would appear that replication is the hot enhancement... Slashdot | PostgreSQL 7.3 Released http://developers.slashdot.org/article.pl?sid=02/11/30/1815200 Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.3 gotchas for applications and client libraries
Tom/Hackers,
Going back a bit, but relevant with 7.3's release...
Tom Lane writes on 03 Sep 2002:
> Lee Kindness <[EMAIL PROTECTED]> writes:
> >
> > [ original post was regarding the mileage in adding utility
> > functions to PostgreSQL to cut-out common catalog lookups, thus
> > making apps less fragile to catalog changes ]
> >
> > CREATE OR REPLACE FUNCTION column_exists(NAME, NAME) RETURNS BOOLEAN AS '
> > CREATE OR REPLACE FUNCTION table_exists(NAME) RETURNS BOOLEAN AS'
> >
> > Obviously these need attention when our application targets 7.3 (and
> > thanks for the heads-up), but all changes are localised.
>
> They are? What will your policy be about schema names --- won't you
> have to touch every caller to add a schema name parameter?
As it turns out, no. And thinking about i'm sure this is right
approach too, assuming:
CREATE SCHEMA a;
CREATE SCHEMA b;
CREATE TABLE a.foo(f1 INT, f2 TEXT);
CREATE TABLE b.foo(f1 TEXT, f2 NUMERIC(10,1));
then:
SELECT column_exists('foo', 'f1');
should return 'f', however:
SELECT column_exists('a.foo', 'f1');
should return 't', likewise with:
SET SEARCH_PATH TO "a","public";
SELECT column_exists('foo', 'f1');
I can't see any use in a separate parameter - the user will want the
current - in scope - table, or explicitly specify the schema with the
table name.
> I'm not averse to trying to push logic over to the backend, but I think
> the space of application requirements is wide enough that designing
> general-purpose functions will be quite difficult.
On the whole I'd agree, but I think determining if a table/column
exists has quite a high usage... More so with things like
current_database() added to 7.3. Anyway, for reference here are
column_exists(table, column) and table_exists(table) functions for
PostgreSQL 7.3, changes from 7.3 version maked by ' -- PG7.3':
\echo creating function: column_exists
CREATE OR REPLACE FUNCTION column_exists(NAME, NAME) RETURNS BOOLEAN AS '
DECLARE
tab ALIAS FOR $1;
col ALIAS FOR $2;
rec RECORD;
BEGIN
SELECT INTO rec *
FROM pg_class c, pg_attribute a
WHERE c.relname = tab
AND pg_table_is_visible(c.oid) -- PG7.3
AND c.oid = a.attrelid
AND a.attnum > 0
AND a.attname = col;
IF NOT FOUND THEN
RETURN false;
ELSE
RETURN true;
END IF;
END;
' LANGUAGE 'plpgsql';
\echo creating function: table_exists
CREATE OR REPLACE FUNCTION table_exists(NAME) RETURNS BOOLEAN AS '
DECLARE
tab ALIAS FOR $1;
rec RECORD;
BEGIN
SELECT INTO rec *
FROM pg_class c
WHERE c.relname = tab;
AND pg_table_is_visible(c.oid) -- PG7.3
IF NOT FOUND THEN
RETURN false;
ELSE
RETURN true;
END IF;
END;
' LANGUAGE 'plpgsql';
Of course, thanks for the original email in this thread:
http://www.ca.postgresql.org/docs/momjian/upgrade_tips_7.3
Thanks, Lee Kindness.
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] [GENERAL] 7.3 -> pg_atoi: zero-length string
Was it necessary? No idea, you're welcome to search through the
pgsql-hackers archives to determine the reasoning behind the change. I
believe the change was made by Bruce Momjian (going by the release
notes). I only remember reading the discussion in passing.
This is also one of the reasons for beta releases - to allow people to
test against the new version and pick up these sort of things. Either
this gives them/you time to make changes, or to lobby to get the old
behaviour back.
If I were in your situation i'd probably hack back the old behaviour
to 7.3, compile and run that while changes were made. Or stick with
7.2.x until changes were made to your applications, have you got a
'big carrot' for going with 7.3?
Lee.
Ben-Nes Michael writes:
> That's indeed very nice but I don't see the logic in it.
>
> If I want to upgrade I need to go on all my projects and change thousands of
> lines.
>
> And that's not all :(
> I have other applications like phprojekt that was not developed by me and
> became useless now as I cant insert.
>
> Was this step so necessary ?
>
> - Original Message -
> From: "Lee Kindness" <[EMAIL PROTECTED]>
> To: "Ben-Nes Michael" <[EMAIL PROTECTED]>
> Cc: "postgresql" <[EMAIL PROTECTED]>; "Lee Kindness"
> <[EMAIL PROTECTED]>
> Sent: Tuesday, December 03, 2002 12:37 PM
> Subject: [GENERAL] 7.3 -> pg_atoi: zero-length string
>
> > Hi, one of the changes in 7.3 was to disallow '' being implicitly
> > converted to 0. In your example below image_order is clearly not a
> > text/char column, so what are you trying to set it too? If you want it
> > be 0 then explicitly use 0, if you want it to be undefined then use
> > NULL.
> >
> > Lee.
> >
> > Ben-Nes Michael writes:
> > > Hi
> > >
> > > After I upgraded 7.2.3 to 7.3 I started to get the following errors:
> > > pg_atoi: zero-length string
> > >
> > > its seems that i get it when not all field have content:
> > >
> > > this is one example that generate the error:
> > > insert into images (section_id, pic_date, image_order) values ('8',
> > > '2002-12-03', '')
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [GENERAL] 7.3 -> pg_atoi: zero-length string
I don't think many will be caught-out, since it's an overwhelmingly
daft thing to do anyway!
If you've got a numeric type column then assign numeric values to, not
strings. Or explicitly convert.
I'm sure all occurrences of this "in the wild" are due to sloppy
SQL...
For the record:
lkind@coulin:~% sql -uingres iidbdb
INGRES TERMINAL MONITOR Copyright (c) 1981, 1997 Computer Associates Intl, Inc.
OpenIngres SPARC SOLARIS Version OI 2.0/9712 (su4.us5/00) login
Tue Dec 3 16:17:27 2002
continue
* create table test(f1 int);\g
Executing . . .
continue
* insert into test(f1) values('');\g
Executing . . .
E_US0B61 line 1, You cannot assign a value of type 'varchar' to a column
of type 'integer'. Explicitly convert the value to the required type.
(Tue Dec 3 16:18:22 2002)
continue
* select int4('');\g
Executing . . .
col1
0
(1 row)
continue
Lee.
Ben-Nes Michael writes:
> Then Why not set it to NULL
>
> Seems logic as there is nothing between ''
>
> What is the solution of other dbs ( oracle, db2 .. ) to:
> insert into table (num) value (''); ?
>
> Who knows how many application will suffer becouse of this.
>
> - Original Message -
> From: "Bruce Momjian" <[EMAIL PROTECTED]>
> To: "Lee Kindness" <[EMAIL PROTECTED]>
> Cc: "Ben-Nes Michael" <[EMAIL PROTECTED]>; "postgresql"
> <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
> Sent: Tuesday, December 03, 2002 6:08 PM
> Subject: Re: [GENERAL] 7.3 -> pg_atoi: zero-length string
>
> > The change was made to tighten up the code to catch errors sooner.
> > There isn't much logic to making '' be 0, and no one could make a case
> > for keeping such a mapping.
> >
> > --
> -
> >
> > Lee Kindness wrote:
> > > Was it necessary? No idea, you're welcome to search through the
> > > pgsql-hackers archives to determine the reasoning behind the change. I
> > > believe the change was made by Bruce Momjian (going by the release
> > > notes). I only remember reading the discussion in passing.
> > >
> > > This is also one of the reasons for beta releases - to allow people to
> > > test against the new version and pick up these sort of things. Either
> > > this gives them/you time to make changes, or to lobby to get the old
> > > behaviour back.
> > >
> > > If I were in your situation i'd probably hack back the old behaviour
> > > to 7.3, compile and run that while changes were made. Or stick with
> > > 7.2.x until changes were made to your applications, have you got a
> > > 'big carrot' for going with 7.3?
> > >
> > > Lee.
> > >
> > > Ben-Nes Michael writes:
> > > > That's indeed very nice but I don't see the logic in it.
> > > >
> > > > If I want to upgrade I need to go on all my projects and change
> thousands of
> > > > lines.
> > > >
> > > > And that's not all :(
> > > > I have other applications like phprojekt that was not developed by me
> and
> > > > became useless now as I cant insert.
> > > >
> > > > Was this step so necessary ?
> > > >
> > > > - Original Message -
> > > > From: "Lee Kindness" <[EMAIL PROTECTED]>
> > > > To: "Ben-Nes Michael" <[EMAIL PROTECTED]>
> > > > Cc: "postgresql" <[EMAIL PROTECTED]>; "Lee Kindness"
> > > > <[EMAIL PROTECTED]>
> > > > Sent: Tuesday, December 03, 2002 12:37 PM
> > > > Subject: [GENERAL] 7.3 -> pg_atoi: zero-length string
> > > >
> > > > > Hi, one of the changes in 7.3 was to disallow '' being implicitly
> > > > > converted to 0. In your example below image_order is clearly not a
> > > > > text/char column, so what are you trying to set it too? If you want
> it
> > > > > be 0 then explicitly use 0, if you want it to be undefined then use
> > > > > NULL.
> > > > >
> > > > > Lee.
> > > > >
> > > > > Ben-Nes Michael writes:
> > > > > > Hi
> > > > > >
> > > > > > After I upgraded 7.2.3 to 7.3 I started to get the following
> errors:
> > > > > > pg_atoi: zero-length string
> > > > > >
> > > > > > its seems that i get it when not all field have content:
> > > > > >
> > > > > > this is one example that generate the error:
> > > > > > insert into images (section_id, pic_date, image_order) values
> ('8',
> > > > > > '2002-12-03', '')
> > >
> >
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PQnotifies() in 7.3 broken?
Perhaps the .so name should have been updated for PostgreSQL 7.3? For example in 7.2 libpq is: /usr/lib/libpq.so -> libpq.so.2.2 /usr/lib/libpq.so.2 -> libpq.so.2.2 /usr/lib/libpq.so.2.0 -> libpq.so.2 /usr/lib/libpq.so.2.2 and PostgreSQL 7.3: /usr/lib/libpq.so -> libpq.so.2.2 /usr/lib/libpq.so.2 -> libpq.so.2.2 /usr/lib/libpq.so.2.0 -> libpq.so.2 /usr/lib/libpq.so.2.2 the same. This would seem to imply binary compatibility? Lee. Tom Lane writes: > "Jeroen T. Vermeulen" <[EMAIL PROTECTED]> writes: > > When receiving a trigger notification under 7.3, the structure returned > > by PQnotifies() appears to be bogus. In a test I ran, its be_pid was > > consistently zero and its relname pointed into never-neverland. > We changed the PQnotifies result structure in 7.3 (to insulate clients > from the value of NAMEDATALEN). I think you are compiling your app with > a 7.3 libpq header and then running it with 7.2 libpq code, or possibly > vice versa. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] big text field -> message type 0x44
Tom Lane writes: > Okay, so it seems -D_REENTRANT is the appropriate fix. > > We could either add that to the template/solaris file, or just add a > note to FAQ_Solaris advising that it be added to the configure switches > if people intend to use libpq in threaded programs. Is there any > cost or downside to just adding it always in template/solaris? However, _REENTRANT is not a Solarisism... On all (recent) UNIX systems it toggles on correct handling for thread specific instances of historically global variables (eg errno). It should be considered for all platforms if libpq is intended to be used from threaded programs. You'll probably find Tomas's code breaks on Linux too... Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PQnotifies() in 7.3 broken?
Bruce Momjian writes: > Tom Lane wrote: > > Lee Kindness <[EMAIL PROTECTED]> writes: > > > Perhaps the .so name should have been updated for PostgreSQL 7.3? > > > > It should have been. If it wasn't, that was a serious oversight. > > Not sure if we should change it in 7.3.1 or not, though; it may be > > too late for that. Any thoughts out there? > Seems I did forget. I always update the minor for a major release, but > when development starts, and I seem to have forgotten for 7.3. Sorry. Personally I think automatically updating the version numbers is as bad as not doing it at all - it misses the point. The version numbers in shared library filenames denote binary compatibilty, if the /public/ API changes then the version number really must be incremented. If the version increments without any associated API changes then it's just a PITA for developers and products linking to the PostgreSQL libraries! It forces recompilation when there is not really a need. Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] big text field -> message type 0x44
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > Tom Lane writes: > >>> Okay, so it seems -D_REENTRANT is the appropriate fix. > > However, _REENTRANT is not a Solarisism... On all (recent) UNIX > > systems it toggles on correct handling for thread specific instances > > of historically global variables (eg errno). It should be considered > > for all platforms if libpq is intended to be used from threaded > > programs. > Now that I think about it, what that macro is probably really doing is > switching the code from looking at a static "errno" variable to looking > at a per-thread variable. So in fact -D_REENTRANT would be correct if > you intended to link with a thread-aware libc, and wrong if you intended > to link with a non-aware libc. (Is there such a thing as a non-threaded > implementation of libc on the platforms where -D_REENTRANT does > anything?) If this analysis is right then I think we should *not* > force _REENTRANT; it will have to be up to users to choose the mechanism > they want to use in their programs. I think in the long-term the libraries are going to have to be looked at in detail to ensure they work as would be expected from multithreaded programs. I cannot see any harm in adding -D_REENTRANT to CFLAGS even though some platforms supersede it with -lthread or something (becaue they still define _REENTRANT behind the scenes). I remember in the past reading in detail the issues involved with making shared libraries work as expected from threads. However I no-longer has access to that book, but think it was "Multithreaded Programming with Pthreads"... Again, something i'd like to look at later this month. Workwise the threaded code we had which used embedded SQL calls in C fell into heaps when moved from Ingres to PostgreSQL. And Ingres's ESQL/C is real crap for threading and we employeed loads of mutexes... So, ... Lee. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Let's create a release team
Dan Langille writes: > On 10 Dec 2002 at 9:34, Tom Lane wrote: > > Anyone running cvsup would have a complete copy of the source CVS, I > > believe. It would be more troubling to reconstruct the mailing list > > archives; I'm not sure that those are mirrored anywhere > Do you mean the repository, or the source. The repository is the ,v > files The source isn't. Most developers would have the source, > but not necessarily the repo. See: http://www.cvsup.org/ It mirrors the repository and some of the PostgreSQL developers use this... Lee. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] [INTERFACES] Patch for DBD::Pg pg_relcheck problem
Ian Barwick writes: > On Tuesday 10 December 2002 00:47, Tom Lane wrote: > > In the next protocol version update (hopefully 7.4) I would like to see > > the basic version string (eg, "7.3.1" or "7.4devel") delivered to the > > client automatically during connection startup and then available from a > > libpq inquiry function. This would eliminate the need to call version() > > explicitly and to know that you must skip "PostgreSQL " in its output. > Something along the lines of > char *PQversion(const PGconn *conn) ? Probably: int PQversion(const PGconn *conn) would be better, and easier to parse? For example the value returned for 7.3.1 would be 7003001; for 7.4 7004000; for 101.10.2 101010002. This allows simple numerical tests... Lee. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PQnotifies() in 7.3 broken?
Making something binary incompatible IS an API change! In the case in question an externally visible structure definition was changed - clearly a change of API... Bruce Momjian writes: > Tom Lane wrote: > > Bruce Momjian <[EMAIL PROTECTED]> writes: > > > So if a recompile fixes it, increment minor, else major. > > > > Wrong --- if you need a recompile then it's not binary-compatible, so > > it should be a major version bump. > > But the previous poster said only API changes were reasons to bump the > major, right? ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Library Versions (was: PQnotifies() in 7.3 broken?)
Guys, can I take this chance to summarise the thread and when the major and minor versions should be updated, perhaps could be added to the developers FAQ if everyone is in agreement? Major Version = The major version number should be updated whenever the source of the library changes to make it binary incompatible. Such changes include, but limited to: 1. Removing a public function or structure (or typedef, enum, ...) 2. Modifying a public functions arguments. 3. Removing a field from a public structure. 3. Adding a field to a public structure, unless steps have been previously taken to shield users from such a change, for example by such structures only ever being allocated/instantiated by a library function which would give the new field a suitable default value. Adding a new function would NOT force an increase in the major version number. When the major version is increased all applications which link to the library MUST be recompiled - this is not desirable. When the major version is updated the minor version gets reset. Minor Version = The minor version number should be updated whenever the functionality of the library has changed, typically and change in source code between releases would mean an increase in the minor version number so long as it does not require a major version increase. Thanks, Lee. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Library Versions (was: PQnotifies() in 7.3 broken?)
Guys,
Some further comments on bumbing the major version number which aren't
so cut-n-dry...
Lee Kindness writes:
> The major version number should be updated whenever the source of the
> library changes to make it binary incompatible. Such changes include,
> but limited to:
>
> 1. Removing a public function or structure (or typedef, enum, ...)
>
> 2. Modifying a public functions arguments.
>
> 3. Removing a field from a public structure.
>
> 3. Adding a field to a public structure, unless steps have been
> previously taken to shield users from such a change, for example by
> such structures only ever being allocated/instantiated by a library
> function which would give the new field a suitable default value.
For #2 steps could be taken to maintain binary compatibility across
minor PostgreSQL releases (e.g. the 7.2 series, the 7.3 series, the
7.4/8.0 series). Consider the following function
void print_stuff(int arg1, int arg2)
{
printf("stuff: %d %d\n", arg1, arg2);
}
If we wanted to add a third argument:
void print_stuff(int arg1, int arg2, int arg3)
{
printf("stuff: %d %d %d\n", arg1, arg2, arg3);
}
Then doing it like this:
void print_stuff2(int arg1, int arg2, int arg3)
{
printf("stuff: %d %d %d\n", arg1, arg2, arg3);
}
void print_stuff(int arg1, int arg2)
{
print_stuff(arg1, arg2, 0);
}
would maintain binary compatibility. Obviously this would add a fair
bit of cruft if used extensively, but considering the changes between
minor versions would probably be worthwhile to avoid bumping library
major version. Naturally in the next major version print_stuff() would
assume the functionality and arguments of print_stuff2().
Lee.
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PQnotifies() in 7.3 broken?
Jeroen T. Vermeulen writes: > On Mon, Dec 16, 2002 at 05:41:06PM +0100, Jeroen T. Vermeulen wrote: > > Speaking of which, what if user relies on sizeof(PGnotify::relname)? > ^ > code Yes, a change in the size of relname makes this binary incompatible and the user code changes may not be just a simple recompile - It all depends on what it being used for! Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Table Timemachine!
Guys, I've been asked by a colleague about methods to keep track of 'previous' contents of a table - i.e. changes made and a way of getting back to a previous state. Now, I know INSERT/UPDATE/DELETE triggers to maintain an accompanying table is a way to do this. But, I have a nagging feeling that somebody has at one point put together such a feature in a generic fashion... I thought there was a 'time machine' module in contrib! Searches through the mailing lists have not turned up any specific pointers. Aanyone else remember this, and remember where? Or am I misguided and deluded! Thanks, Lee. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] [GENERAL] Table Timemachine!
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > I have a nagging feeling that somebody has at one point put together > > such a feature in a generic fashion... I thought there was a 'time > > machine' module in contrib! > contrib/spi/timetravel.* > I haven't ever tried it ... It's really quite good. Could do with an update - it uses abstime internally and the docs refer to 'currabstime' which is no longer around. Nice. Lee. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Bison 1.875 RPMs
Guys, for your convenience i've put online a source RPM for Bison 1.875 along with binary RPMs for Redhat 7.2, 7.3 and 8.0. Hunting around the net i didn't find any existing Bison >= 1.50 RPMs, so this should be useful for those compiling PostgreSQL (ECPG in particular) from the CVS source: http://services.csl.co.uk/postgresql/bison-1.875-1PGDG.src.rpm http://services.csl.co.uk/postgresql/redhat72/bison-1.875-1PGDG.i386.rpm http://services.csl.co.uk/postgresql/redhat73/bison-1.875-1PGDG.i386.rpm http://services.csl.co.uk/postgresql/redhat80/bison-1.875-1PGDG.i386.rpm The PGDG tag seemed appropriate. >From version 1.50 onward Bison starting installing a 'yacc' script - i've disabled the creation of this in these RPMs so upgrades can be done simply without conflicting with the 'byacc' package (guess this is Bison based too)... Thanks, Lee. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] PostgreSQL libraries - PThread Support, but not use...
On a slightly related note to the other threads thread [sic] going on... Over the Christmas/New Year break i've been looking into making the PostgreSQL client libraries (in particular libpq and ecpg) thread-safe - that is they can safely be used by a program which itself is using mutliple threads. I'm largely there with ecpg (globals are protected, a sqlca for each thread), but of course it relies on libpq which needs work to share a connection between thrreads. Relying on thread experience from a few years back on Solaris I had assumed I could just use the pthread_* routines in the shared libraries and they would reference the weak symbols in libc, rather than explicitly pull in libpthread. If the application using the library linked to libpthreads then the 'real' thread routines would be activated, single threaded apps wouldn't need link to libpthread... However there seems to be no standard to which pthread_* routines are available as weak symbols in libc (Linux and Solaris differ). So a couple of questions to decide the course of this work: 1. It's looking likely that the libraries will need to link to libpthread, and as a result anything linking to the libraries will need to link to libpthread too. Will this be accepted in a patch? A similar problem has cropt up with the perl integration recently too (i.e. the Perl developers have decided to link in libpthread). 2. Is their any mileage in using an abstraction layer - ACE, npr? A Bit OTT for what i'm doing, but... 3. Am I wasting my time? I think making the PostgreSQL libraries thread-safe/aware is very worthwhile, but a lot of hurdles... Thanks, Lee. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] PostgreSQL libraries - PThread Support, but not use...
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > On a slightly related note to the other threads thread [sic] going > > on... Over the Christmas/New Year break i've been looking into making > > the PostgreSQL client libraries (in particular libpq and ecpg) > > thread-safe - that is they can safely be used by a program which > > itself is using mutliple threads. I'm largely there with ecpg (globals > > are protected, a sqlca for each thread), but of course it relies on > > libpq which needs work to share a connection between thrreads. > > AFAIK, libpq is thread-safe already, it's just not thread-aware. > What you'd presumably want is a wrapper layer that adds a mutex to > prevent multiple threads from manipulating a PGconn at the same time. > Couldn't this be done without touching libpq at all? Certainly, it could. I've not fully investigated the current failings of libpq WRT to threading yet. But it's certainly a bit more than I stated above. I don't know where the statement that libpq is thread safe originated from (I see it's on the website). but at a quick glance I believe that krb4, krb5, PQoidStatus(), PQsetClientEncoding(), winsock_strerror() need more though investigation and non-thread-safe fuctions are also being used (at a glance gethostbyname() and strtok()). > > 1. It's looking likely that the libraries will need to link to > > libpthread, and as a result anything linking to the libraries will > > need to link to libpthread too. Will this be accepted in a patch? > If the patch requires pthread and not any other form of thread support, > probably not. See nearby threading thread ;-) > In general I'd be unhappy with doing anything to libpq that forces > non-threaded clients to start depending on libpthread (or other thread > libraries). Thus the idea of a wrapper seems more appealing to me. Okay, but what about ecpg? Thread-local sqlca instances would be a real benefit, guess Michael and Christof are the guys to talk to? I suppose the real problem is the needed baggage with this - the autoconf macros will be a nightmare! Thanks, Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] PostgreSQL libraries - PThread Support, but not use...
Tom Lane writes: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > We have definatly had requests for improved thread-safeness for libpq > > and ecpg in the past, so whatever you can do would be a help. We say > > libpq is thread-safe, but specifically mention the non-threadsafe calls > > in the libpq documentation, or at least we should. > We do: > http://www.ca.postgresql.org/users-lounge/docs/7.3/postgres/libpq-threading.html > But Lee's point about depending on possibly-unsafe libc routines is a > good one. I don't think anyone's gone through the code with an eye to > that. Right, so a reasonable angle for me to take is to go through the libpq source looking for potential problem areas and use of "known bad" functions. I can add autoconf checks for the replacement *_r() functions, and use these in place of the traditional ones where available. If any function is found to be not thread-safe and cannot be made so using standard library calls then it needs to be documented as such both in the source and the aforementioned documentation. This approach avoids any thread library dependencies and documents the current state of play WRT thread safety (i.e it's a good, and needed, basis for any later work). ECPG is a separate issue, and best handled as such (it will need thread calls). I'll post a patch for it at a later date so the changes are available to anyone who wants to play with ECPG and threads. Ta, Lee. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] PostgreSQL libraries - PThread Support, but not use...
Bruce Momjian writes: > Lee Kindness wrote: > > Right, so a reasonable angle for me to take is to go through the libpq > > source looking for potential problem areas and use of "known bad" > > functions. I can add autoconf checks for the replacement *_r() > > functions, and use these in place of the traditional ones where > > available. > I am a little confused by the *_r functions. Are they for all > functions? BSD/OS doesn't have them, but all our libc functions are > threadsafe except for things like strtok, where they recommend strsep, > and gethostbyname, where they would suggest getaddrinfo, I guess. Some functions in the C library (and other common system libraries) are defined in such a way to make their implementation non-reentrant. Normally this is due to return values being supplied in a static buffer which is overwritten by the subsequent call, or the calls relying on static data between calls. A list such functions would include: asctime crypt ctime drand48 ecvt encrypt erand48 fcvt fgetgrent fgetpwent fgetspent getaliasbyname getaliasent getdate getgrent getgrgid getgrnam gethostbyaddr gethostbyname gethostbyname2 gethostent getlogin getnetbyaddr getnetbyname getnetent getnetgrent getprotobyname getprotobynumber getprotoent getpwent getpwnam getpwuid getservbyname getservbyport getservent getspent getspnam getutent getutid getutline gmtime hcreate hdestroy hsearch initstate jrand48 lcong48 localtime lrand48 mrand48 nrand48 ptsname qecvt qfcvt rand random readdir readdir64 seed48 setkey setstate sgetspent srand48 srandom strerror strtok tmpnam ttyname to one degree or another. The important ones to watch for are: ctime, localtime, asctime, gmtime, readdir, strtok and tmpnam. Now these functions are often augmented by a _r partner which fixes their API to allow their implementations to be reentrant. After a quick grep libpq could be using crypt, gethostbyname, random, strerror, encrypt, getpwuid, rand and strtok. As you rightly note, ins ome cases the correct fix is to use alternative functions and not the _r versions - this avoids lots of ifdefs! L. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [PATCHES] [HACKERS] PostgreSQL libraries - PThread Support, but not use...
Tom, Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > + #define _THREAD_SAFE > > + #define _REENTRANT > > + #define _POSIX_PTHREAD_SEMANTICS > What is this stuff, and why isn't it wrapped in any sort of > platform-specific test? If it's needed, why is it in only one .c > file? It's actually in libpq-int.h too... The correct way for this is to compile with the compilers specific thread flags, however the downside to this has already been discussed. Depending on the system one, or a combination of those flags will turn on some magic such as errno being a function call rather than a global variable. This is needed to make the library thread safe. On a second look libpq-int.h isn't the best place for this (hence it also appears in one of the C files), it needs to be done in each C file before any of the system headers are included - a libpq-threads.h header? Would this be ok? Do do things 100% right we'd need to detect compiler thread flags and compile with them... > Also, haven't you broken SOCK_STRERROR for the Windows case? Sorry, I seem to have forgotton to update the prototype in win32.h to match the updated function. Updated diff attached (and online). Lee. diffs-libpq.txt Description: Binary data ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] PostgreSQL libraries - PThread Support, but not use...
Guys, just posted patches for libpq to address thread-safe issues. Now uses strerror_r(), gethostbyname_r(), getpwuid_r() and crypt_r() where available. Passes all tests on Linux and Solaris. Any comments let me know - my first major(ish) outing in the PostgreSQL source, in particular I'm not use in the configure stuff is 100% right... Patches also at: http://services.csl.co.uk/postgresql/diffs-20030109-configure.txt.gz http://services.csl.co.uk/postgresql/diffs-20030109-libpq.txt.gz Thanks, Lee. Lee Kindness writes: > Tom Lane writes: > > Bruce Momjian <[EMAIL PROTECTED]> writes: > > > We have definatly had requests for improved thread-safeness for libpq > > > and ecpg in the past, so whatever you can do would be a help. We say > > > libpq is thread-safe, but specifically mention the non-threadsafe calls > > > in the libpq documentation, or at least we should. > > We do: > > http://www.ca.postgresql.org/users-lounge/docs/7.3/postgres/libpq-threading.html > > But Lee's point about depending on possibly-unsafe libc routines is a > > good one. I don't think anyone's gone through the code with an eye to > > that. > > Right, so a reasonable angle for me to take is to go through the libpq > source looking for potential problem areas and use of "known bad" > functions. I can add autoconf checks for the replacement *_r() > functions, and use these in place of the traditional ones where > available. > > If any function is found to be not thread-safe and cannot be made so > using standard library calls then it needs to be documented as such > both in the source and the aforementioned documentation. > > This approach avoids any thread library dependencies and documents the > current state of play WRT thread safety (i.e it's a good, and needed, > basis for any later work). > > ECPG is a separate issue, and best handled as such (it will need > thread calls). I'll post a patch for it at a later date so the changes > are available to anyone who wants to play with ECPG and threads. > > Ta, Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] [HACKERS] PostgreSQL libraries - PThread Support, but
Ok guys, I propose that the new libpq diff and 2 source files which i'll soon send to pgsql-patches is applied to the source. This diff is a cleaned up version of the previous version with the wrapper functions moved out into their own file and more comments added. Also the use of crypt_r() has been removed (not worth the effort), the cpp defines have been renamed to be consistent with each other and Tom's concerns with loose #defines has been partly addressed. This diff does not include any configure changes. I plan to tackle this separately ASAP, and hopefully produce something more acceptable. I will add checks for appropriate compiler thread flags (for compiling libpq, and alow the removal of #defines in libpq-reentrant.h), and link flags & libs (for a planned threaded libpq test program and renentrant ecpg library). If a thread environment is found then check for the reentrant functions will be done. Looking at various open source projects configure.in files there seems to be little commonality in the thread test macros (telp gratefully accepted!), I currently think that something like the approach used by glib is most suitable (switch on OS). All sound acceptable? Thanks, Lee. Peter Eisentraut writes: > Lee Kindness writes: > > > Patches attached to make libpq thread-safe, now uses strerror_r(), > > gethostbyname_r(), getpwuid_r() and crypt_r() where available. Where > > strtok() was previously used strchr() is now used. > > AC_TRY_RUN tests are prohibited. Also, try to factor out some of these > huge tests into separate macros and put them into config/c-library.m4. > And it would be nice if there was some documentation about what was > checked for. If you just want to check whether gethostbyname_r() has 5 or > 6 arguments you can do that in half the space. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] ECPG, threading and pooling
Michael, Michael Meskes writes: > On Thu, Jan 23, 2003 at 02:40:33PM +0530, Shridhar >Daithankar<[EMAIL PROTECTED]> wrote: > > I would like to use ECPG as it is relatively easy to code. However my > > application is multithreaded and also uses connecion pools. > > I'm afraid it needs some work to be thread-safe. sqlca is defined > statically. No big deal it seems to implement a thread safe version but > I haven't yet found the time. I've spent a bit of time on making ecpg thread safe over Christmas, while it's not finished i'm sure the attached patch is at least useful and a step in the right direction. Lee. ecpg-threadsafe.patch.gz Description: Binary data ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] ECPG, threading and pooling
Michael Meskes writes: > On Mon, Jan 27, 2003 at 11:52:18AM +0000, Lee Kindness wrote: > > I've spent a bit of time on making ecpg thread safe over Christmas, > > while it's not finished i'm sure the attached patch is at least useful > > and a step in the right direction. > Thanks a lot. I have no experience in multithreaded software development > so I cannot completely check your patch but it surely looks good. Shall > I commit it, or will you? I think we should get it into CVS for all to > test. Problem with it is it needs some changes to configure.in (and associated files) for it to be worthwhile. Checks would need to be added to determine is threads are supported on the build platform and the needed compile and link flags. I'm trying to get this together for libpq too. > Just two questions: > - Is it neccessary to patch c.h for ecpg? No, but each ecpg source file would need something like: #ifdef HAVE_PTHREAD_H # include # include #endif It's in c.h just now for ease of testing. > - Not sure with the version numbering but I can fix that later. As discussed on pgsql-hackers after the 7.3 release any "binary incompatible" library change needs a major version number increase. Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] ECPG, threading and pooling
Shridhar Daithankar writes: > On 27 Jan 2003 at 14:06, Lee Kindness wrote: > > Michael Meskes writes: > > > Thanks a lot. I have no experience in multithreaded software development > > > so I cannot completely check your patch but it surely looks good. Shall > > > I commit it, or will you? I think we should get it into CVS for all to > > > test. > > Problem with it is it needs some changes to configure.in (and > > associated files) for it to be worthwhile. Checks would need to be > > added to determine is threads are supported on the build platform and > > the needed compile and link flags. I'm trying to get this together for > > libpq too. > Just out of curiosity, what happens when there are more than one competing > threading libraries? Like native threads and linuxthreads on freeBSD? It's all down to the checks in configure... Looking at many packages which have threads check in configure (e.g. openldap, mysql, mozilla, glib, ...) some let the user specify which theading library to use (e.g --with-threads=linuxthreads (or something)) while others just seem to pick up the first available... To be honest I am quite daunted by the checks needed (every other package seems to be doing the check differently!) and any/all help would be welcome! L. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Thread safe ecpg
Okay guys, i've been a bit tight of time recently to move forward on this, but I plan to do a small amount of work on the patches to clean them up so they can be merged into the sources. I'll also add a README to document what will need to be fudged in the makefile to get it building. Hopefully this should be workable until proper configure support is added? I'd love to say i'd get this done some evening this week, but it's unlikely. And i'm away in the States on business all next week... I'll see what I can do... Thanks, :Lee Michael Meskes writes: > On Mon, Feb 24, 2003 at 09:23:38PM +0530, Shridhar Daithankar<[EMAIL PROTECTED]> > wrote: > > I was just wondering.The patches for making ecpg thread safe that were > > floating around few days back, are they going to make in any near future > > releases? > > I sure hope so. AFAIK there are some small issues with it. That's why > they aren't committed yet. > > Michael ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] ecpg in REL7_3_2
Larry Rosenman writes: > <[EMAIL PROTECTED]> wrote: > > Bison is 1.28. Yacc (version 91.7.30) gives a similar error. > you need Bison 1.50 or later... Which you can get binary and src RPMs of from: http://services.csl.co.uk/postgresql/ L: ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Bison 1.875 now required.
For those using Linux, the RPMs at: http://services.csl.co.uk/postgresql/ are probably handy. L. Bruce Momjian writes: > Sorry, I meant bison 1.875 is now required, not 1.85. > > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup.| Newtown Square, Pennsylvania 19073 > ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
[HACKERS] ECPG thread-safety
Philip, You can find the "patch so far" at: http://services.csl.co.uk/postgresql/ along with a libpq thread-safe patch. The thread referenced by Bruce's email details some of the issues still to be looked at. The main work is integration with the build system (testing for and linking in POSIX threads) and testing. I'm more than happy for you to look at at, I haven't had the time I hoped to complete this. Thanks, Lee. Philip Yarra writes: > As I found out recently, ECPG is not thread-safe. This is a problem for us, as > we really need to be able to use ECPG* in a multi-threaded C application. > Michael mentioned that Lee is working on fixing these problems, but was not > sure when it would be complete. > > So, my questions are: > Lee/anyone, do you have an estimated completion date? > If not, can we (myself plus a "real" C programmer) help you? > > Regards, Philip Yarra. > > * Yes, we could use libpq, but ECPG minimises our porting from Informix. > Rewriting everything to use libpq would be too expensive. > ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] ECPG thread-safety
I must have spent at least a day looking at configure threads tests the past few months. Look at the tests in Python, OpenLDAP, MySQL, Apache 2.0, Perl, ... each and everyone of them is different! Ouch. Way above my experience in configure. One of the packages (sorry cannot find my notes on which one!) I looked at down the test on a per-OS level - supplying the tread compile and link flags used on that OS and CC to see if they worked... Probably the simplest way to go? L. Bruce Momjian writes: > > Would someone take those patches and hunt around for proper 'configure' > tests? I can do the configure coding, but I don't know what tests to > make. > > http://services.csl.co.uk/postgresql/ > > > --- > > Larry Rosenman wrote: > > > > > > --On Friday, March 28, 2003 10:54:23 -0500 Bruce Momjian > > <[EMAIL PROTECTED]> wrote: > > > > > > > > I have the libpq patch in my mailbox that just needs configure thread > > > testing code. I will get that in for 7.4. > > I can help with the UnixWare threads stuff. Also, Micheal Meskes now has > > an account > > on my UnixWare box. > > > > The threads are native on UW, but you need to invoke the cc -Kpthread > > option. > > > > LER > > > > > > > > - > > > -- > > > > > > Lee Kindness wrote: > > >> Philip, > > >> > > >> You can find the "patch so far" at: > > >> > > >> http://services.csl.co.uk/postgresql/ > > >> > > >> along with a libpq thread-safe patch. > > >> > > >> The thread referenced by Bruce's email details some of the issues > > >> still to be looked at. The main work is integration with the build > > >> system (testing for and linking in POSIX threads) and testing. > > >> > > >> I'm more than happy for you to look at at, I haven't had the time I > > >> hoped to complete this. > > >> > > >> Thanks, Lee. > > >> > > >> Philip Yarra writes: > > >> > As I found out recently, ECPG is not thread-safe. This is a problem > > >> > for us, as we really need to be able to use ECPG* in a > > >> > multi-threaded C application. Michael mentioned that Lee is working > > >> > on fixing these problems, but was not sure when it would be > > >> > complete. > > >> > > > >> > So, my questions are: > > >> > Lee/anyone, do you have an estimated completion date? > > >> > If not, can we (myself plus a "real" C programmer) help you? > > >> > > > >> > Regards, Philip Yarra. > > >> > > > >> > * Yes, we could use libpq, but ECPG minimises our porting from > > >> > Informix. Rewriting everything to use libpq would be too expensive. > > >> > > > >> > > > > > > -- > > > Bruce Momjian| http://candle.pha.pa.us > > > [EMAIL PROTECTED] | (610) 359-1001 > > > + If your life is a hard drive, | 13 Roberts Road > > > + Christ can be your backup.| Newtown Square, Pennsylvania > > > 19073 > > > > > > > > > ---(end of broadcast)--- > > > TIP 6: Have you searched our list archives? > > > > > > http://archives.postgresql.org > > > > > > > > > > > -- > > Larry Rosenman http://www.lerctr.org/~ler > > Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] > > US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 > > > > > > > > > > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] PostgreSQL libraries - PThread Support, but
Your call, but the "broken" call is in earlier glibc versions for
sure (if you're on a Linux box take a look in /usr/include - the
prototype is still there, may even get used depending on compiler
options!). I seem to remember compiling this on recent Solaris, HPUX,
Linux and AIX versions without hitting the "broken" version, but...
L.
Bruce Momjian writes:
>
> Lee, I have a question about this code:
>
> char *pqStrerror(int errnum, char *strerrbuf, size_t buflen)
> {
> #if defined HAVE_STRERROR_R
>/* reentrant strerror_r is available */
>strerror_r(errnum, strerrbuf, buflen);
>return strerrbuf;
> #elif defined HAVE_NONPOSIX_STRERROR_R
>/* broken (well early POSIX draft) strerror_r() which returns 'char *' */
>return strerror_r(errnum, strerrbuf, buflen);
> #else
>/* no strerror_r() available, just use strerror */
>return strerror(errnum);
> #endif
> }
>
> Why do we have to care about HAVE_NONPOSIX_STRERROR_R? Can't we just
> use the HAVE_STRERROR_R code in all cases?
>
> ---
>
> Lee Kindness wrote:
> Content-Description: message body text
>
> > Patch attached, along with new libpq-reentrant.c and libpq-reentrant.h
> > files for src/interfaces/libpq.
> >
> > Also at http://services.csl.co.uk/postgresql/
> >
> > Thanks, Lee.
> >
> > Lee Kindness writes:
> > > Ok guys, I propose that the new libpq diff and 2 source files which
> > > i'll soon send to pgsql-patches is applied to the source. This diff is
> > > a cleaned up version of the previous version with the wrapper
> > > functions moved out into their own file and more comments added. Also
> > > the use of crypt_r() has been removed (not worth the effort), the
> > > cpp defines have been renamed to be consistent with each other and
> > > Tom's concerns with loose #defines has been partly addressed.
> > >
> > > This diff does not include any configure changes. I plan to tackle
> > > this separately ASAP, and hopefully produce something more acceptable.
> > >
> > > I will add checks for appropriate compiler thread flags (for compiling
> > > libpq, and alow the removal of #defines in libpq-reentrant.h), and
> > > link flags & libs (for a planned threaded libpq test program and
> > > renentrant ecpg library). If a thread environment is found then check
> > > for the reentrant functions will be done.
> > >
> > > Looking at various open source projects configure.in files there seems
> > > to be little commonality in the thread test macros (telp gratefully
> > > accepted!), I currently think that something like the approach used by
> > > glib is most suitable (switch on OS).
> > >
> > > All sound acceptable?
> > >
> > > Thanks, Lee.
> > >
> > > Peter Eisentraut writes:
> > > > Lee Kindness writes:
> > > >
> > > > > Patches attached to make libpq thread-safe, now uses strerror_r(),
> > > > > gethostbyname_r(), getpwuid_r() and crypt_r() where available. Where
> > > > > strtok() was previously used strchr() is now used.
> > > >
> > > > AC_TRY_RUN tests are prohibited. Also, try to factor out some of these
> > > > huge tests into separate macros and put them into config/c-library.m4.
> > > > And it would be nice if there was some documentation about what was
> > > > checked for. If you just want to check whether gethostbyname_r() has 5 or
> > > > 6 arguments you can do that in half the space.
> >
>
> [ Attachment, skipping... ]
>
> [ Attachment, skipping... ]
>
> [ Attachment, skipping... ]
>
> --
> Bruce Momjian| http://candle.pha.pa.us
> [EMAIL PROTECTED] | (610) 359-1001
> + If your life is a hard drive, | 13 Roberts Road
> + Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 9: most folks find a random_page_cost between 1 or 2 is ideal
Re: [PATCHES] [HACKERS] PostgreSQL libraries - PThread Support, but
Bruce Momjian writes: > My point is why do we care whether it returns char * or nothing --- we > should just return strerrbuf in all cases. Ok, I see. Guess that is reasonable. L. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] ECPG and bytea
Guys, surely some one's done this before? I've tried using PQescapeBytea too, but still get (slightly) different output. If I try and insert "\x02\x01\x02\x03\x04hello\x05\x64\x99\x45" I get (int values of chars printed): INSERT: 2 1 2 3 4 104 101 108 108 111 5 100 -103 69 SELECT: 2 1 2 3 4 104 101 108 108 111 5 100 92 50 51 49 69 hoping someone has a quick answer... if not I'll post a full source example and look into it more... Thanks, L. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] Bulkloading using COPY - ignore duplicates?
Hello, I'm in the process of porting a large application from Ingres to PostgreSQL. We make heavy use of bulkloading using the 'COPY' statement in ESQL/C. Consider the SQL statements below (in a psql session on an arbitrary database): CREATE TABLE copytest(f1 INTEGER, f2 INTEGER); CREATE UNIQUE INDEX copytest_idx ON copytest USING BTREE(f1, f2); COPY copytest FROM '/tmp/copytest'; Given the file /tmp/copytest: 1 1 2 2 3 3 4 4 4 4 5 5 6 6 will result in the following output: ERROR: copy: line 5, Cannot insert a duplicate key into unique index copytest_idx However my application code is assuming that duplicate rows will simply be ignored (this is the case in Ingres, and I believe Oracle's bulkloader too). I propose modifying _bt_check_unique() in /backend/access/nbtree/nbtinsert.c to emit a NOTICE (rather than ERROR) elog() and return NULL (or appropriate) to the calling function if a duplicate key is detected and a 'COPY FROM' is in progress (add new parameter to flag this). Would this seem a reasonable thing to do? Does anyone rely on COPY FROM causing an ERROR on duplicate input? Would: WITH ON_DUPLICATE = CONTINUE|TERMINATE (or similar) need to be added to the COPY command (I hope not)? Thanks, -- Lee Kindness, Senior Software Engineer Concept Systems Limited. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Bulkloading using COPY - ignore duplicates?
Justin Clift writes: > Lee Kindness wrote: > > WITH ON_DUPLICATE = CONTINUE|TERMINATE (or similar) > I would suggest : > WITH ON_DUPLICATE = IGNORE|TERMINATE > purely for easier understanding, given there is no present standard nor > other databases' syntax to conform to. Personally I don't see the need, and think that 'COPY FROM' could well just go with the new semantics... Onto an implementation issue - _bt_check_unique() returns a TransactionId, my plans were to return NullTransactionId on a duplicate key but naturally this is used in the success scenario. Looking in backend/transam/transam.c I see: TransactionId NullTransactionId = (TransactionId) 0; TransactionId AmiTransactionId = (TransactionId) 512; TransactionId FirstTransactionId = (TransactionId) 514; >From this I'd gather <514 can be used as magic-values/constants, So would I be safe doing: TransactionId TransactionId = (TransactionId) 1; and return TransactionId from _bt_check_unique() back to _bt_do_insert()? Naturally is something meaningful. I presume all I need to know is if 'xwait' in _bt_check_unique() is ever '1'... Thanks, -- Lee Kindness, Senior Software Engineer Concept Systems Limited. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Bulkloading using COPY - ignore duplicates?
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > Would this seem a reasonable thing to do? Does anyone rely on COPY > > FROM causing an ERROR on duplicate input? > Yes. This change will not be acceptable unless it's made an optional > (and not default, IMHO, though perhaps that's negotiable) feature of > COPY. I see where you're coming from, but seriously what's the use/point of COPY aborting and doing a rollback if one duplicate key is found? I think it's quite reasonable to presume the input to COPY has had as little processing done on it as possible. I could loop through the input file before sending it to COPY but that's just wasting cycles and effort - Postgres has btree lookup built in, I don't want to roll my own before giving Postgres my input file! > The implementation might be rather messy too. I don't much care > for the notion of a routine as low-level as bt_check_unique knowing > that the context is or is not COPY. We might have to do some > restructuring. Well in reality it wouldn't be "you're getting run from copy" but rather "notice on duplicate, rather than error & exit". There is a telling comment in nbtinsert.c just before _bt_check_unique() is called: /* * If we're not allowing duplicates, make sure the key isn't already * in the index. XXX this belongs somewhere else, likely */ So perhaps dupes should be searched for before _bt_doinsert is called, or somewhere more appropriate? > > Would: > > WITH ON_DUPLICATE = CONTINUE|TERMINATE (or similar) > > need to be added to the COPY command (I hope not)? > It occurs to me that skip-the-insert might be a useful option for > INSERTs that detect a unique-key conflict, not only for COPY. (Cf. > the regular discussions we see on whether to do INSERT first or > UPDATE first when the key might already exist.) Maybe a SET variable > that applies to all forms of insertion would be appropriate. That makes quite a bit of sense. -- Lee Kindness, Senior Software Engineer Concept Systems Limited. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Bulkloading using COPY - ignore duplicates?
Okay, If I'm going to modify 'COPY INTO' to include 'ignore duplicates' functionality it looks like I'll have to add to the COPY syntax. The most obvious way is to add: WITH IGNORE DUPLICATES to the syntax. I'm going to need my hand held a bit for this! The grammar for COPY will need updating in gram.y and specifically the 'WITH' keyword will have 'IGNORE DUPLICATES' as well as 'NULL AS'. Any pointers? Thanks, Lee. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Bulkloading using COPY - ignore duplicates?
Lee Kindness writes: > If I'm going to modify 'COPY INTO' to include 'ignore duplicates' > functionality it looks like I'll have to add to the COPY syntax. The > most obvious way is to add: > WITH IGNORE DUPLICATES Or does it make more sense to add a 'COPY_IGNORE_DUPLICATES' SET parameter? Lee. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Bulkloading using COPY - ignore duplicates?
Peter Eisentraut writes:
> However, it seems to me that COPY ignoring duplicates can easily be
> done by preprocessing the input file.
Or by post-processing, like (error checking cut):
void import_shots(char *impfile, int lineshoot_id)
{
char tab_name[128];
char tab_temp[128];
frig_file(impfile); /* add the postgres header */
sprintf(tab_name, "shot_%d", lineshoot_id);
sprintf(tab_temp, "shot_%d_tmp", lineshoot_id);
sprintf(cmd, "CREATE TEMPORARY TABLE %s AS SELECT * FROM shot",
tab_temp);
EXEC SQL EXECUTE IMMEDIATE :cmd;
EXEC SQL COMMIT WORK; /* will not work without comit here! */
sprintf(cmd, "COPY BINARY %s FROM '%s'", tab_temp, impfile);
append_page_alloc(cmd, tab_name, impfile, 1);
EXEC SQL EXECUTE IMMEDIATE :cmd;
sprintf(cmd, "INSERT INTO %s SELECT DISTINCT ON(shot_time) * FROM %s",
tab_name, tab_temp);
EXEC SQL EXECUTE IMMEDIATE :cmd;
sprintf(cmd, "DROP TABLE %s", tab_temp);
EXEC SQL EXECUTE IMMEDIATE :cmd;
EXEC SQL COMMIT WORK ;
remove(impfile);
}
However this is adding significant time to the import
operation. Likewise I could loop round the input file first and hunt
for duplicates, again with a performance hit.
My main point is that Postgres can easily and quickly check for
duplicates during the COPY (as it does currently) and it adds zero
execution time to simply ignore these duplicate rows. Obviously this
is a useful feature otherwise Oracle, Ingres and other commercial
relational databases wouldn't feature similiar functionality.
Yes, in an ideal world the input to COPY should be clean and
consistent with defined indexes. However this is only really the case
when COPY is used for database/table backup and restore. It misses the
point that a major use of COPY is in speed optimisation on bulk
inserts...
Lee.
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] compiling libpq++ on Solaris with Sun SPRO6U2 (fixed
It's really quite simple, the Sun C compiler (acc) does not understand the -Wl flag, rather it passes the -R and -h options onto the linker verbatim. Given the only two (realistic) compiler choices under Solaris are gcc and acc it makes sense to support then both 'out of the box'. I keep a similar patch to Denis's (without the wierd hi-ascii characters in his) lying around for when i build a Solaris version. Regards, Lee Kindness. Peter Eisentraut writes: > Denis A Ustimenko writes: > > [change -Wl,-R to -R and -Wl,-h to -h] > > I'm having a difficult time understanding this. Both -R and -h are linker > options, not compiler options. So while the compiler driver might be nice > enough to recognize them as the former and pass them through, this change > just pushes these chances, and it doesn't add any theoretical change of > functionality. > > So, if you want this to be fixed, you're going to have to start with > explaining your problem, and then we can start looking for solutions. > > -- > Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter > > > ---(end of broadcast)--- > TIP 4: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] How do I get a list of DB's or Tables with a postgres SQL statement?
See the pg_database table: http://www.postgresql.org/idocs/index.php?catalog-pg-database.html Regards, Lee. Ron de Jong writes: > All, > > How do I get a list of DB's or Tables with a postgres SQL statement? > It needs to be an SQL statement otherwise perl/DBI/prepare won't parse it. > I know there is some funtions in psql: > > \l(show databases) > \d(show tables) > > which work fine from psql, but these > statements won't be parsed by perl, DBI > > I also know there is an object: > > @names = $dbh->tables; > > but what about databases? > > Please help. > > Ron de Jong > the Netherlands > > > > > > > ---(end of broadcast)--- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Compiling on Solaris with Sun compiler
Peter Eisentraut writes: > Lee Kindness writes: > > For your information I've attached the man page for the Sun C > > compiler, which explicitly lists the -h and -R flags. > I didn't read much farther than > acc (SPARC only) is not intended to be used directly on > Solaris 2.x. ;-) Touche, but the man page for the front-end (plain old cc) doesn't list options and only refers to the acc man page ;) Onto another Solaris compilation issue... After a simple './configure' on a stock Solaris 2.6 box the compilation of interfaces/ecpg/lib/execute.c fails due to the macro definition of 'gettext' to ''. This macro is invoked on the prototype of gettext() in libintl.h (included via locale.h). A './configure --enable-nls' is needed. To properly fix the problem either: 1. Don't include or use locale functions in execute.c unless --enable-locale has been specified. 2. In execute.c the include for locale.h whould be moved above that of postgres_fe.h 3. Replace '#define gettext' in c.h with something more unique (PG_gettext perhaps?) Regards, Lee Kindness. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Compiling on Solaris with Sun compiler
Peter Eisentraut writes: > Lee Kindness writes: > > Touche, but the man page for the front-end (plain old cc) doesn't list > > options and only refers to the acc man page ;) > Well, I'm stumped. All the Solaris compilers I've ever seen did support > and document the -Wl option. Well I never submitted my patch for building using the Sun compilers since I thought that the newer versions did support the -Wl option - I'm using an old (version 4) Sun compiler. However it seems that Denis it using revision 2 of the latest version 6 compiler! > > After a simple './configure' on a stock Solaris 2.6 box the > > compilation of interfaces/ecpg/lib/execute.c fails due to the macro > > definition of 'gettext' to ''. This macro is invoked on the prototype > > of gettext() in libintl.h (included via locale.h). > Fail how and why? Well in c.h there is the following define: #ifdef ENABLE_NLS #include #else #define gettext(x) (x) #endif #define gettext_noop(x) (x) so gettext() simply is the supplied parameter if --enable-nls is not supplied. However ecpg/execute.c has the following includes: #include "postgres_fe.h" #include #include Via postgres_fe.h gettext() gets defined as above. However locale.h also pulls in the systems libintl.h which has the following prototype: extern char *gettext(); which the preprocessor changes to: extern char *(); due to the gettext define in c.h. Naturally this makes the build fail. Configuring with --enable-nls gets round this but I don't require that functionality. Regards, Lee Kindness. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Bulkloading using COPY - ignore duplicates?
Guys,
I've made some inroads towards adding 'ignore duplicates'
functionality to PostgreSQL's COPY command. I've updated the parser
grammar for COPY FROM to now accept:
COPY [ BINARY ] table [ WITH OIDS ]
FROM { 'filename' | stdin }
[ [USING] DELIMITERS 'delimiter' ]
[ WITH [NULL AS 'null string']
[IGNORE DUPLICATES] ]
and added code to propagate this setting down to the CopyFrom function
in backend/commands/copy.c.
I also played around with _bt_check_unique, _bt_do_insert and btinsert
to return NULL on duplicate rather than elog(ERROR). Likewise
ExecInsertIndexTuples and index_insert were passed the
ignore_duplicate flag and index_insert changed to elog(ERROR) if the
return from the insert function was NULL and ignore_duplicate flag was
false.
These changes worked and gave the desired result for the COPY FROM
command, however as many mentioned these changes are far too low
level... After assessing the situation more fully, I believe the
following change in CopyFrom would be more suitable:
/* BEFORE ROW INSERT Triggers */
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT]
> 0)
{
HeapTuple newtuple;
newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
if (newtuple == NULL) /* "do nothing" */
skip_tuple = true;
else if (newtuple != tuple) /* modified by Trigger(s) */
{
heap_freetuple(tuple);
tuple = newtuple;
}
}
/* new code */
if( ignore_duplicates == true )
{
if( duplicate index value )
skip_tuple = true;
}
if (!skip_tuple)
{
Now I imagine 'duplicate index value' would be functionally similar to
_bt_check_unique but obviously higher level. Is there any existing
code with the functionality I desire? Can anyone point me in the right
way...
Thanks,
Lee Kindness.
Lee Kindness writes:
> I'm in the process of porting a large application from Ingres to
> PostgreSQL. We make heavy use of bulkloading using the 'COPY'
> statement in ESQL/C. Consider the SQL statements below (in a psql
> session on an arbitrary database):
>
> CREATE TABLE copytest(f1 INTEGER, f2 INTEGER);
> CREATE UNIQUE INDEX copytest_idx ON copytest USING BTREE(f1, f2);
> COPY copytest FROM '/tmp/copytest';
>
> Given the file /tmp/copytest:
>
> 1 1
> 2 2
> 3 3
> 4 4
> 4 4
> 5 5
> 6 6
>
> will result in the following output:
>
> ERROR: copy: line 5, Cannot insert a duplicate key into unique index copytest_idx
>
> However my application code is assuming that duplicate rows will
> simply be ignored (this is the case in Ingres, and I believe Oracle's
> bulkloader too). I propose modifying _bt_check_unique() in
> /backend/access/nbtree/nbtinsert.c to emit a NOTICE (rather than
> ERROR) elog() and return NULL (or appropriate) to the calling function
> if a duplicate key is detected and a 'COPY FROM' is in progress (add
> new parameter to flag this).
>
> Would this seem a reasonable thing to do? Does anyone rely on COPY
> FROM causing an ERROR on duplicate input? Would:
>
> WITH ON_DUPLICATE = CONTINUE|TERMINATE (or similar)
>
> need to be added to the COPY command (I hope not)?
>
> Thanks,
>
> --
> Lee Kindness, Senior Software Engineer
> Concept Systems Limited.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] ecpg - GRANT bug
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > The existing code in ecpg/preproc/preproc.y to handle the WITH option > > simply throws an error and aborts the processing... The patch below > > prevents the segfault and also passes on the WITH option to the > > backend, probably a better fix. > I agree. It shouldn't be ecpg's business to throw errors on behalf of > the backend, especially not "not yet implemented" kinds of errors. > That just causes ecpg to be more tightly coupled to a particular backend > version than it needs to be. In which case a number of other cases should be weeded out of parser.y and passed onto the backend: CREATE TABLE:GLOBAL TEMPORARY option. CREATE FUNCTION: IN/OUT/INOUT options (note there's a bug in parser.y there anyway, it would pass on 'oinut' for INOUT). COMMIT: AND [NO] CHAIN options? Where do these come from, it's not ANSI (i'd probably leave this one). Perhaps an ET_NOTICE should still be output however... Let me known if you want a patch for these cases too. Regards, Lee Kindness. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] ecpg - GRANT bug
Bruce Momjian writes:
> Lee Kindness writes:
> > In which case a number of other cases should be weeded out of
> > parser.y and passed onto the backend:
> > [ snip ]
> > Let me known if you want a patch for these cases too.
> Sure, send them on over.
Patch below, it changes:
1. A number of mmerror(ET_ERROR) to mmerror(ET_NOTICE), passing on
the (currently) unsupported options to the backend with warning.
2. Standardises warning messages in such cases.
3. Corrects typo in passing of 'CREATE FUNCTION/INOUT' parameter.
Patch:
? interfaces/ecpg/preproc/ecpg
Index: interfaces/ecpg/preproc/preproc.y
===
RCS file: /projects/cvsroot/pgsql/src/interfaces/ecpg/preproc/preproc.y,v
retrieving revision 1.161
diff -c -r1.161 preproc.y
*** interfaces/ecpg/preproc/preproc.y 2001/10/15 20:15:09 1.161
--- interfaces/ecpg/preproc/preproc.y 2001/10/16 09:15:53
***
*** 1074,1084
| LOCAL TEMPORARY { $$ = make_str("local temporary"); }
| LOCAL TEMP{ $$ = make_str("local temp"); }
| GLOBAL TEMPORARY {
! mmerror(ET_ERROR, "GLOBAL TEMPORARY TABLE is
not currently supported");
$$ = make_str("global temporary");
}
| GLOBAL TEMP {
! mmerror(ET_ERROR, "GLOBAL TEMPORARY TABLE is
not currently supported");
$$ = make_str("global temp");
}
| /*EMPTY*/ { $$ = EMPTY; }
--- 1074,1084
| LOCAL TEMPORARY { $$ = make_str("local temporary"); }
| LOCAL TEMP{ $$ = make_str("local temp"); }
| GLOBAL TEMPORARY {
! mmerror(ET_NOTICE, "Currently unsupported
CREATE TABLE/GLOBAL TEMPORARY will be passed to backend");
$$ = make_str("global temporary");
}
| GLOBAL TEMP {
! mmerror(ET_NOTICE, "Currently unsupported
CREATE TABLE/GLOBAL TEMP will be passed to backend");
$$ = make_str("global temp");
}
| /*EMPTY*/ { $$ = EMPTY; }
***
*** 1103,1110
{
if (strlen($4) > 0)
{
! sprintf(errortext, "CREATE
TABLE/COLLATE %s not yet implemented; clause ignored", $4);
! mmerror(ET_NOTICE, errortext);
}
$$ = cat_str(4, $1, $2, $3, $4);
}
--- 1103,1110
{
if (strlen($4) > 0)
{
! sprintf(errortext, "Currently
unsupported CREATE TABLE/COLLATE %s will be passed to backend", $4);
! mmerror(ET_NOTICE, errortext);
}
$$ = cat_str(4, $1, $2, $3, $4);
}
***
*** 1219,1225
}
| MATCH PARTIAL
{
! mmerror(ET_NOTICE, "FOREIGN KEY/MATCH PARTIAL not yet
implemented");
$$ = make_str("match partial");
}
| /*EMPTY*/
--- 1219,1225
}
| MATCH PARTIAL
{
! mmerror(ET_NOTICE, "Currently unsupported FOREIGN KEY/MATCH
PARTIAL will be passed to backend");
$$ = make_str("match partial");
}
| /*EMPTY*/
***
*** 1614,1620
| BACKWARD { $$ = make_str("backward"); }
| RELATIVE { $$ = make_str("relative"); }
| ABSOLUTE{
! mmerror(ET_NOTICE, "FETCH/ABSOLUTE not
supported, backend will use RELATIVE");
$$ = make_str("absolute");
}
;
--- 1614,1620
|
[HACKERS] delayed mail?
Tatsuo Ishii writes: > I saw over 7 hours delay between postgresql.org and sever1.pgsql.org. > Does anynone know what's happening here? I've seen massive delays too: Received: from mail.csl.co.uk by euphrates.csl.co.uk (8.9.3/ConceptI 2.4) id EAA08864; Tue, 16 Oct 2001 04:24:41 +0100 (BST) Received: from server1.pgsql.org by mail.csl.co.uk (8.11.1/ConceptO 2.3) id f9G3Oh219345; Tue, 16 Oct 2001 04:24:44 +0100 Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f9G3Ljq66512 for <[EMAIL PROTECTED]>; Mon, 15 Oct 2001 22:23:36 -0500 (CDT) (envelope-from [EMAIL PROTECTED]) Received: from mail1.hub.org (webmail.hub.org [216.126.85.1]) by postgresql.org (8.11.3/8.11.4) with ESMTP id f9FNaKP72316 for <[EMAIL PROTECTED]>; Mon, 15 Oct 2001 19:36:20 -0400 (EDT) (envelope-from [EMAIL PROTECTED]) Received: from candle.pha.pa.us (candle.navpoint.com [162.33.245.46]) by mail1.hub.org (8.11.3/8.11.4) with ESMTP id f9FGg8X57598 for <[EMAIL PROTECTED]>; Mon, 15 Oct 2001 12:42:08 -0400 (EDT) (envelope-from [EMAIL PROTECTED]) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id f9FGg6g02963 for [EMAIL PROTECTED]; Mon, 15 Oct 2001 12:42:06 -0400 (EDT) Message-Id: <[EMAIL PROTECTED]> and this one's days: Received: from mail.csl.co.uk by euphrates.csl.co.uk (8.9.3/ConceptI 2.4) id KAA11316; Tue, 16 Oct 2001 10:34:03 +0100 (BST) Received: from postgresql.org by mail.csl.co.uk (8.11.1/ConceptO 2.3) id f9G9Y3220747; Tue, 16 Oct 2001 10:34:03 +0100 Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) by postgresql.org (8.11.3/8.11.4) with SMTP id f9G0GHP83130; Mon, 15 Oct 2001 20:16:27 -0400 (EDT) (envelope-from [EMAIL PROTECTED]) Received: from candle.pha.pa.us (candle.navpoint.com [162.33.245.46]) by postgresql.org (8.11.3/8.11.4) with ESMTP id f9D4our63609; Sat, 13 Oct 2001 00:50:56 -0400 (EDT) (envelope-from [EMAIL PROTECTED]) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id f9D4oqo09577; Sat, 13 Oct 2001 00:50:52 -0400 (EDT) Message-Id: <[EMAIL PROTECTED]> Hardly any (yours omong the excepted, headers below) messages are coming through in a timely fashion: Received: from mail.csl.co.uk by euphrates.csl.co.uk (8.9.3/ConceptI 2.4) id PAA14131; Tue, 16 Oct 2001 15:17:46 +0100 (BST) Received: from server1.pgsql.org by mail.csl.co.uk (8.11.1/ConceptO 2.3) id f9GEHl222496; Tue, 16 Oct 2001 15:17:47 +0100 Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f9GEEEs48462 for <[EMAIL PROTECTED]>; Tue, 16 Oct 2001 09:16:31 -0500 (CDT) (envelope-from [EMAIL PROTECTED]) Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2]) by postgresql.org (8.11.3/8.11.4) with ESMTP id f9GCDVP15349 for <[EMAIL PROTECTED]>; Tue, 16 Oct 2001 08:13:32 -0400 (EDT) (envelope-from [EMAIL PROTECTED]) Received: from sranhm.sra.co.jp (sranhm [133.137.13.152]) by sraigw.sra.co.jp (8.9.3/3.7W-sraigw) with ESMTP id VAA16107 for <[EMAIL PROTECTED]>; Tue, 16 Oct 2001 21:13:28 +0900 (JST) Received: from localhost (IDENT:[EMAIL PROTECTED] [133.137.84.24]) by sranhm.sra.co.jp (8.9.3+3.2W/3.7W-srambox) with ESMTP id VAA10335 for <[EMAIL PROTECTED]>; Tue, 16 Oct 2001 21:13:26 +0900 Message-Id: <[EMAIL PROTECTED]> Lee. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Column names - time
Tom Lane writes: > There is an up-to-date list of keywords in the documentation: > >http://www.ca.postgresql.org/users-lounge/docs/7.1/postgres/sql-keywords-appendix.html Thanks for the info. Would I be right in saying that the status of time (unreserved for PostgreSQL) for 7.2 needs to be changed to reserved since it can no-longer be used as an unquoted column identifier? Perhaps the other time related identifiers too. Regards, Lee Kindness. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] ecpg - GRANT bug
Bill Studenmund writes: > I think this patch is wrong. Wouldn't it be better to make the line number > in yylineno be correct? Also, there are users of the line number in pcg.l > which you didn't change. > Looking at it, I don't see why the line number is off. It is initialized > to 1 at the begining and whenever a new file is included. In the generated > code, it is incrimented whenever a '\n' is found. Strange... The main reason I split the patch from the previous one for ecpg was for this reason - I didn't think it was the correct patch myself. However after a serious hunt for the root of the problem I've found that it is actually working correctly in the 7.2 sources and I was picking up an ecpg from a 7.1.3ish build (which only contained an ecpg binary). Appologies for the false hunt! For the record it was fixed in pgc.l 1.79 (meskes 13-Jun-01). Regards, Lee. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Btree index ignored on SELECT until VACUUM ANALYZE
When adding an index to a (quite large, ~2 million rows) table PostgreSQL continues to do sequential lookups until VACUUM ANALYZE is run. Naturally performance is poor. The CREATE INDEX statement takes considerable time. Seen with 7.1.3 on Intel Linux (RedHat 7.0 & 7.1 and Solaris 2.6. In the example below the data file (8 MB) can be found at: http://services.csl.co.uk/postgresql/obs.gz Consider the session below: lkind@elsick:~% createdb obs_test CREATE DATABASE lkind@elsick:~% psql obs_test obs_test=# CREATE TABLE obs (setup_id INTEGER, time REAL, value REAL, bad_data_flag SMALLINT); CREATE obs_test=# COPY obs FROM '/user/lkind/obs'; COPY obs_test=# SELECT COUNT(*) FROM obs; count - 1966593 (1 row) obs_test=# CREATE UNIQUE INDEX obs_idx ON obs USING BTREE(setup_id, time); CREATE obs_test=# EXPLAIN SELECT * FROM obs WHERE setup_id = 300 AND time = 118; NOTICE: QUERY PLAN: Seq Scan on obs (cost=0.00..42025.90 rows=197 width=14) EXPLAIN obs_test=# VACUUM ANALYZE obs ; VACUUM obs_test=# EXPLAIN SELECT * FROM obs WHERE setup_id = 300 AND time = 118; NOTICE: QUERY PLAN: Index Scan using obs_idx on obs (cost=0.00..9401.60 rows=1 width=14) EXPLAIN obs_test=# \q ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] ecpg - GRANT bug
Tom Lane writes:
> Uh, isn't the correct fix
> ! $$ = cat_str(8, make_str("grant"), $2, make_str("on"), $4, $5,
> make_str("to"), $7, $8);
> ISTM your patch loses the opt_with_grant clause. (Of course the
> backend doesn't currently accept that clause anyway, but that's no
> reason for ecpg to drop it.)
My patch doesn't loose the option, it's never been passed on anyway:
opt_with_grant: WITH GRANT OPTION
{
mmerror(ET_ERROR, "WITH GRANT OPTION is not
supported. Only relation owners can set privileges");
}
| /*EMPTY*/
;
The existing code in ecpg/preproc/preproc.y to handle the WITH option
simply throws an error and aborts the processing... The patch below
prevents the segfault and also passes on the WITH option to the
backend, probably a better fix.
Regards, Lee.
Index: interfaces/ecpg/preproc/preproc.y
===
RCS file: /projects/cvsroot/pgsql/src/interfaces/ecpg/preproc/preproc.y,v
retrieving revision 1.159
diff -c -r1.159 preproc.y
*** interfaces/ecpg/preproc/preproc.y 2001/10/14 12:07:57 1.159
--- interfaces/ecpg/preproc/preproc.y 2001/10/15 09:06:29
***
*** 1693,1699
GrantStmt: GRANT privileges ON opt_table relation_name_list TO grantee_list
opt_with_grant
{
! $$ = cat_str(7, make_str("grant"), $2,
make_str("on"), $4, $5, make_str("to"), $7);
}
;
--- 1693,1699
GrantStmt: GRANT privileges ON opt_table relation_name_list TO grantee_list
opt_with_grant
{
! $$ = cat_str(8, make_str("grant"), $2,
make_str("on"), $4, $5, make_str("to"), $7, $8);
}
;
***
*** 1769,1779
| grantee_list ',' grantee { $$ = cat_str(3, $1, make_str(","),
$3); }
;
! opt_with_grant: WITH GRANT OPTION
! {
! mmerror(ET_ERROR, "WITH GRANT OPTION is not
supported. Only relation owners can set privileges");
!}
! | /*EMPTY*/
;
--- 1769,1776
| grantee_list ',' grantee { $$ = cat_str(3, $1, make_str(","),
$3); }
;
! opt_with_grant: WITH GRANT OPTION { $$ = make_str("with grant option"); }
! | /*EMPTY*/ { $$ = EMPTY; }
;
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Column names - time
In moving from 7.1.3 to 7.2devel (for bug fixes) we've encountered a problem with a, previously valid, column name: time. In 7.1.3 the following worked: CREATE TABLE test(time INTEGER); while in 7.2devel it results in a parse error: ERROR: parser: parse error at or near "time" Looking at the source I see this is a result of 'time' being added to ColLabel in backend/parser/gram.y earlier this month. This effects interface code and database migration using pg_dump. Obviously a new column name will have to be used, however is there a definitive list of keywords to avoid so such an occurance wouldn't happen in a production system? Currently these include: abort all analyse analyze and any asc between binary bit both case cast char character check cluster coalesce collate column constraint copy cross current_date current_time current_timestamp current_user dec decimal default deferrable desc distinct do else end except exists explain extract false float for foreign freeze from full global group having ilike initially in inner intersect into inout is isnull join leading left like limit listen load local lock move natural nchar new not notnull nullif null numeric off offset old on only or order out outer overlaps position precision primary public references reset right select session_user setof show some substring table then time timestamp to trailing transaction trim true union unique unknown user using vacuum varchar verbose when where Best Regards, Lee Kindness.. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] ECPG: FETCH ALL|n FROM cursor - Memory allocation?
Assuming the following fetch statement in embedded SQL/C:
EXEC SQL FETCH ALL IN selectFromTable_cur INTO
:array1,
:array2;
is memory automatically allocated (by experimentation I guess so)?
Should the input pointers be NULL initialised? How should the memory
be freed?
Assuming the following fetch statement:
while( 1 )
{
EXEC SQL FETCH 1000 IN selectFromTable_cur INTO
:array1,
:array2;
if( (sqlca.sqlcode < 0) || (sqlca.sqlcode != 0) )
break;
}
is memory automatically allocated (by experimentation I guess so)?
Should the input pointers be NULL initialised before each fetch, or
only before the first one? How should the memory be freed?
Any pointers to useful documentation?
Thanks, Lee Kindness.
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
[HACKERS] PATCH: Memory leaks on start-up
Guys, attached is a patch to fix two memory leaks on start-up. The
first is when freeaddrinfo has been used to free memory allocated by
getaddrinfo2 (so freeaddrinfo2 should be used). The second is 2 leaks
when creating the lock (PostgreSQL and socket) files.
The diff is against last night's CVS HEAD.
L.
diff -cr pgsql.old/src/backend/libpq/pqcomm.c pgsql/src/backend/libpq/pqcomm.c
*** pgsql.old/src/backend/libpq/pqcomm.c2003-06-12 08:36:51.0 +0100
--- pgsql/src/backend/libpq/pqcomm.c2003-07-21 22:58:39.0 +0100
***
*** 363,369
added++;
}
! freeaddrinfo(addrs);
if (!added)
{
--- 363,369
added++;
}
! freeaddrinfo2(family, addrs);
if (!added)
{
diff -cr pgsql.old/src/backend/utils/init/miscinit.c
pgsql/src/backend/utils/init/miscinit.c
*** pgsql.old/src/backend/utils/init/miscinit.c 2003-06-27 20:08:37.0 +0100
--- pgsql/src/backend/utils/init/miscinit.c 2003-07-22 00:08:39.0 +0100
***
*** 842,848
/*
* Arrange for automatic removal of lockfile at proc_exit.
*/
! on_proc_exit(UnlinkLockFile, PointerGetDatum(strdup(filename)));
return true;/* Success! */
}
--- 842,848
/*
* Arrange for automatic removal of lockfile at proc_exit.
*/
! on_proc_exit(UnlinkLockFile, PointerGetDatum(filename));
return true;/* Success! */
}
***
*** 850,875
bool
CreateDataDirLockFile(const char *datadir, bool amPostmaster)
{
! charlockfile[MAXPGPATH];
!
! snprintf(lockfile, sizeof(lockfile), "%s/postmaster.pid", datadir);
! if (!CreateLockFile(lockfile, amPostmaster, true, datadir))
! return false;
! /* Save name of lockfile for RecordSharedMemoryInLockFile */
! strcpy(directoryLockFile, lockfile);
return true;
}
bool
CreateSocketLockFile(const char *socketfile, bool amPostmaster)
{
! charlockfile[MAXPGPATH];
!
! snprintf(lockfile, sizeof(lockfile), "%s.lock", socketfile);
! if (!CreateLockFile(lockfile, amPostmaster, false, socketfile))
! return false;
! /* Save name of lockfile for TouchSocketLockFile */
! strcpy(socketLockFile, lockfile);
return true;
}
--- 850,873
bool
CreateDataDirLockFile(const char *datadir, bool amPostmaster)
{
! snprintf(directoryLockFile, sizeof(directoryLockFile), "%s/postmaster.pid",
datadir);
! if (!CreateLockFile(directoryLockFile, amPostmaster, true, datadir))
! {
! *directoryLockFile = '\0';
! return false;
! }
return true;
}
bool
CreateSocketLockFile(const char *socketfile, bool amPostmaster)
{
! snprintf(socketLockFile, sizeof(socketLockFile), "%s.lock", socketfile);
! if (!CreateLockFile(socketLockFile, amPostmaster, false, socketfile))
! {
! *socketLockFile = '\0';
! return false;
! }
return true;
}
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] threads.c
It was in libpq in my original patch, not port. L. Peter Eisentraut writes: > I have a few problems with the file threads.c: > > 1. It doesn't contain any threads, so the name is slightly inappropriate. > > 2. It's not a porting module (since pqStrerror, etc. are not system > functions), so it doesn't belong into src/port. > > 3. It's only used by libpq, so why not put it into the libpq directory? > > Comments? ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [PATCHES] PATCH: Memory leaks on start-up
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > I'd have to disagree with regards to the memory leaks not being worth > > a mention - any such leak can cause problems when the PostgreSQL > > installation is either unattended, long-living andor has very high > > connection levels. > I don't see how. We are talking about two strings, no more, no less, > that live for exactly the duration of the postmaster run. Explain to me > how any of your above conditions will affect this code in the slightest? For the lock file leak yeah - it's just 2 strings once in a blue moon. However in the more general case of freeaddrinfo and the eniron wizardry (which is also leaking, and is done every new instance, yeah?) it does build up. Anyway, it's no reason to be lazy/messy, is it? L. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [PATCHES] PATCH: Memory leaks on start-up
Tom, happier with the attached patch?
I'd have to disagree with regards to the memory leaks not being worth
a mention - any such leak can cause problems when the PostgreSQL
installation is either unattended, long-living andor has very high
connection levels. Half a kilobyte on start-up isn't negligible in
this light.
Regards, Lee.
Tom Lane writes:
> Lee Kindness <[EMAIL PROTECTED]> writes:
> > Guys, attached is a patch to fix two memory leaks on start-up.
>
> I do not like the changes to miscinit.c. In the first place, it is not
> a "memory leak" to do a one-time allocation of state for a proc_exit
> function. A bigger complaint is that your proposed change introduces
> fragile coupling between CreateLockFile and its callers, in order to
> save no resources worth mentioning. More, it introduces an assumption
> that the globals directoryLockFile and socketLockFile don't change while
> the postmaster is running. UnlinkLockFile should unlink the file that
> it was originally told to unlink, regardless of what happens to those
> globals.
>
> If you are intent on spending code to free stuff just before the
> postmaster exits, a better fix would be for UnlinkLockFile to free its
> string argument after using it.
Index: src/backend/libpq/pqcomm.c
===
RCS file: /projects/cvsroot/pgsql-server/src/backend/libpq/pqcomm.c,v
retrieving revision 1.157
diff -u -r1.157 pqcomm.c
--- src/backend/libpq/pqcomm.c 12 Jun 2003 07:36:51 - 1.157
+++ src/backend/libpq/pqcomm.c 22 Jul 2003 14:16:46 -
@@ -363,7 +363,7 @@
added++;
}
- freeaddrinfo(addrs);
+ freeaddrinfo2(family, addrs);
if (!added)
{
Index: src/backend/utils/init/miscinit.c
===
RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/init/miscinit.c,v
retrieving revision 1.104
diff -u -r1.104 miscinit.c
--- src/backend/utils/init/miscinit.c 27 Jun 2003 19:08:37 - 1.104
+++ src/backend/utils/init/miscinit.c 22 Jul 2003 14:16:46 -
@@ -673,8 +673,15 @@
static void
UnlinkLockFile(int status, Datum filename)
{
- unlink((char *) DatumGetPointer(filename));
- /* Should we complain if the unlink fails? */
+ char *fname = (char *)DatumGetPointer(filename);
+ if( fname != NULL )
+{
+ if( unlink(fname) != 0 )
+ {
+ /* Should we complain if the unlink fails? */
+ }
+ free(fname);
+}
}
/*
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] libpq_r
Guys, take a look at what was done in libpq to make it thread-safe... No locks! No overheaded - just using "proper" reentrant functions... If we have libpq_r then we're making a complete hash of it all - being reentrant is good, even if you're not using threads! Now, ecpg is another issue... L. Bruce Momjian writes: > Shridhar Daithankar wrote: > > I repeat what I have said earlier. If there are two libraries A using libecpg_r > > and B, using libecpg, then program linking against both of them is going to > > have tough time living with symbol conflicts. > > > > I suppose problem will be reproducible even under freeBSD if you try to create > > a postgresql function in C which uses threads. Link the library against libc_r > > and link postgresql against libc. It would run into problems. > > > > I am just stating my experiences.I might have missed solution to this problem. > > > > But overall I like GNU libc approach of everything thread safe by default. If > > thread performance is an issue, then it should be improved. Not worked around > > with two libraries. > > I thought glibc was the one to introduce libc_r in the first place --- > are they making libc thread-safe now? > > What OS's are still using libc_r for threaded-ness? I never liked that > approach myself, and I resist adding it to our setup unless it is > required. > > One problem now is that we don't have a way to create a separate libpq_r > for operating systems that use libc_r. We just create libpq and it is > thread-safe. > > As for the configure flag, we still need it because we don't know the > flags required by all our supported OS's. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] libpq_r
Bruce Momjian writes: > Lee Kindness wrote: > > Guys, take a look at what was done in libpq to make it > > thread-safe... No locks! No overheaded - just using "proper" reentrant > > functions... > > If we have libpq_r then we're making a complete hash of it all - being > > reentrant is good, even if you're not using threads! > > Now, ecpg is another issue... > I think the issue is that using a threaded library to link into libpq > could have locking stuff. > > My guess is that if the OS has separate threaded libs, we have to mimic > that stuff. But there are NO thread primitives/calls in libpq - it's just a normal shared library, it has has no thread voodoo! A threaded app/library using libpq doesn't have to do anything special... libpq doesn't need link the threads library (and it explicitly should not) - only define _REENTRANT. The reentrant calls libpq may be using are in the standard system libraries (not the system thread library), the same libraries that the app/libs will be linked against. Now, libecpg on the otherhand does need mutex locks compiled in. Thus if this is enabled then anyone who links against it also needs to link in threads (unless the system is "intelligent" like Solaris and includes empty stubs of the thread functions in libc). However in this day and age the overhead of the locks are negligible and it is more desirable to have one library... L. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] libpq_r
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > Bruce Momjian writes: > >>> My guess is that if the OS has separate threaded libs, we have to mimic > >>> that stuff. > > But there are NO thread primitives/calls in libpq > That's not the point. The point is stuff that isn't necessarily visible > in the source code --- such as what method it uses to get at "errno", > whether it's linked to thread-safe versions of malloc and other libc > routines, etc. > If the OS supplies both libc and libc_r, it is unlikely to be a good > idea to link a threaded libpq with libc, or a non-threaded libpq with > libc_r. No, that is the point exactly... I know fine well the visibility of errno and the different ways it may be implemented. You'll find it hard to come across an OS which doesn't "do the right thing" WRT to using reentrant functions and "wrappers" if _REENTRANT is defined during the compile... That is ALL that is needed, lets not complicate the issue... And what do you mean by "threaded libpq" and "non-threaded libpq" - as I say above there are NO thread primitives being used in libpq. It's just clean reentrant code... This change/patch should be done irregardless of the move to thread safety! L. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] libpq_r
Tom Lane writes: > If the OS supplies both libc and libc_r, it is unlikely to be a good > idea to link a threaded libpq with libc, or a non-threaded libpq with > libc_r. What modern OS has libc_r? The majors (Linux, Soalris, HPUX, AIX) certainly do not. L. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] libpq_r
Tom Lane writes: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Now I see what you are saying, that _REENTRANT just makes it reentrant, > > and doesn't have a downside in terms of performance. > That's at best an unsupported assertion. Why would the platform bother > with supplying two copies of libc if they didn't think there was a > performance hit? Better remove transactions then, yeah? Performace hit! Profile it and see how minor (or likely non-existent) it is... L. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] libpq_r
Bruce Momjian writes: > However, notice the flags needed under Linux: > > THREAD_CFLAGS="-D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS" > > or are you saying libpq needs only the first one because libpq, itself, > doesn't use threads --- interesting distinction, and perhaps a way we > can get by with one libpq even on platforms that require *_r > libraries. Urgh, I think I was a bit over zealous witht he first patch and just bundled the later two in... After looking around the includes they don't do anything under Linux. I think they're used on AIX. however the best way to do this is to look at another package which has a wide distribution and wide OS base - OpenLDAP being a good example (or even MySQL!)- and see how they do things... No need to reinvent the wheel! L. ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] libpq_r
Bruce Momjian writes: > Peter Eisentraut wrote: > > Bruce Momjian writes: > > > THREAD_CFLAGS="-D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS" > > Those flags are bogus. You don't need any flags under Linux. > Is that true for all versions of Linux? Don't we need the first one so > we compile cleanly. Yeah, you want to define _REENTRANT... > Also, the remaining to linux settings: > THREAD_LIBS="-lpthread" > NEED_REENTRANT_FUNC_NAMES=yes > I assume -lpthread does have an effect on the library, or does it? You DO NOT want -lpthread for libpq, but you do want it for ecpg. L. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] libpq_r
Peter Eisentraut writes: > Larry Rosenman writes: > > I beg to differ. Explicitly, on UnixWare, the header, > > reproduced below, under fair use, show an EXPLICIT difference in what > > happens with _REENTRANT: > Hmm, I was too optimistic. I guess we'll just have to handcraft a > different solution for each platform. But clearly on some platforms we'll > need a libpq_r, so for the reasons I outlined in my initial post, it'd be > good to provide one on all platforms. Sorry, but can you elaborate? I can't see any need with the above reasoning. I managed to find your original post (amazingly hard when you're in digest mode and the post has an empty subject!) and I think the responses in this thread have demonstrated that there is no need for a libpq_r.so, but there is need for a libecpg_r.so I guess we're going to have get down to the code level to demonstrate this! If you compile a example library with _REENTRANT and use _r functions and the function version of errno then it can be used no problem by a library compiled without _REENTRANT and expecting errno to be an int... Unfortunately i'm leaving on holiday tonight until Monday so will not be able to get the example done... Regards, Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] ECPG: 7.4 and a "to" Variable
Guys, looking at ecpg from CVS HEAD's 7.4. The following code
fragement:
EXEC SQL INCLUDE sqlca;
EXEC SQL WHENEVER SQLERROR call sqlprint;
void lofsdb_GetMinMaxRxStations(int *from, int *to)
{
EXEC SQL BEGIN DECLARE SECTION;
int l_from = 0;
int l_to = 0;
EXEC SQL END DECLARE SECTION;
*from = 0;
*to = 0;
EXEC SQL BEGIN;
EXEC SQL SELECT MIN(from_station), MAX(to_station)
INTO :l_from, :l_to
FROM attr_tables
WHERE basetab LIKE 'attr_rx_%';
if( sqlca.sqlcode == 0 )
{
*from = l_from;
*to = l_to;
}
EXEC SQL COMMIT;
}
when processed using:
/var/lib/pgsql/74b/bin/ecpg -t -I/var/lib/pgsql/74b/include -o x.c x.pc
results in the following error:
x.pc:4: ERROR: syntax error at or near "to"
However this works ok on 7.3.x and when the "to" variable is renamed
(e.g. to "to_rx"). Obviously TO is an SQL keyword, but it's not being
used within an EXEC SQL definition, so shouldn't break things.
Regards, Lee.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
[HACKERS] 7.4 COPY BINARY Format Change
Guys,
I've been testing 7.4 against an application today. Recompiled
everything against the new libraries. However the application includes
code which builds up a bulkload file in the documented 7.1 format. The
documentation for COPY goes on at length about the format being
forward compatible...
However of course it's changed in 7.4 for the following minor reasons:
1. Header is now PGCOPY\n\377\r\n\0 rather than PGBCOPY\n\377\r\n\0
2. Integers within the header (but not the data itself) are now in
network order, rather than native order
3. The "integer layout field" has disappeared.
4. typlen is now an int32 rather than an int16 plus an additional
int32 if a varlen type.
I've attached a patch which lets COPY read in the 7.1 format. However
i'm not convinced this is the right way to go - I think the format
which is output by 7.4 should be identical to the 7.1 format. The
"integer layout field" could be used to determine if byteswapping is
required on reading. The other changes seem to be unnecessary? If the
typlen change is kept it should be flagged in the flags field rather
than requiring a completely new format - this would allow old readers
to gracefully fail and old & new files to be read in by 7.4...
It's extremely counter-productive to break backward compatibility for
such whimsical changes! This will hurt those updating to 7.4 once it
is released...
So, I expect the patch below to be rejected - I'll happily rework the
patch to revert 7.4 to a version of the 7.1 format which results in
the same feature gain but without forfeiting backward
compatibility. Let me know.
Thanks. Lee.
Index: src/backend/commands/copy.c
===
RCS file: /projects/cvsroot/pgsql-server/src/backend/commands/copy.c,v
retrieving revision 1.205
diff -c -b -r1.205 copy.c
*** src/backend/commands/copy.c 1 Aug 2003 00:15:19 - 1.205
--- src/backend/commands/copy.c 1 Aug 2003 14:53:35 -
***
*** 91,102
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print);
static char *CopyReadAttribute(const char *delim, CopyReadResult *result);
! static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
!Oid typelem,
bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
! static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/*
* Static communication variables ... pretty grotty, but COPY has
--- 91,107
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print);
static char *CopyReadAttribute(const char *delim, CopyReadResult *result);
! static Datum CopyReadBinaryAttribute(int version, int column_no,
!FmgrInfo *flinfo, Oid typelem,
!bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
! static const char BinarySignature74[11] = "PGCOPY\n\377\r\n\0";
! static const char BinarySignature71[12] = "PGBCOPY\n\377\r\n\0";
! #define BINARY_FMT_74x 740
! #define BINARY_FMT_71x 710
! #define BINARY_FMT_CUR BINARY_FMT_74x
/*
* Static communication variables ... pretty grotty, but COPY has
***
*** 140,148
static intCopyPeekChar(void);
static void CopyDonePeek(int c, bool pickup);
static void CopySendInt32(int32 val);
! static int32 CopyGetInt32(void);
static void CopySendInt16(int16 val);
! static int16 CopyGetInt16(void);
/*
* Send copy start/stop messages for frontend copies. These have changed
--- 145,153
static intCopyPeekChar(void);
static void CopyDonePeek(int c, bool pickup);
static void CopySendInt32(int32 val);
! static int32 CopyGetInt32(int version);
static void CopySendInt16(int16 val);
! static int16 CopyGetInt16(int version);
/*
* Send copy start/stop messages for frontend copies. These have changed
***
*** 571,581
* CopyGetInt32 reads an int32 that appears in network byte order
*/
static int32
! CopyGetInt32(void)
{
uint32 buf;
CopyGetData(&buf, sizeof(buf));
return (int32) ntohl(buf);
}
--- 576,589
* CopyGetInt32 reads an int32 that appears in network byte order
*/
static int32
! CopyGetInt32(int version)
{
uint32 buf;
CopyGetData(&buf, sizeof(buf));
+ if( version == BINARY_FMT_71x )
+ return buf;
+ else
return (int32) ntohl(buf);
}
***
*** 595,605
* CopyGetInt16 reads an int16 that appears in network byte order
*/
static int16
! CopyGetInt16(void)
{
uint16
Re: [HACKERS] 7.4 COPY BINARY Format Change
Tom, Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > I've attached a patch which lets COPY read in the 7.1 format. However > > i'm not convinced this is the right way to go - I think the format > > which is output by 7.4 should be identical to the 7.1 format. > > You are greatly underestimating the changes that occurred in COPY BINARY. > If the format difference had been as minor as you think, I would not > have gratuitously broken compatibility. > > The real change that occurred here is that the individual data fields > go through per-datatype send/receive routines, which in addition to > implementing a mostly machine-independent binary format also provide > defenses against bad input data. > > To continue to read the old COPY BINARY format, we'd have to bypass > those routines and allow direct read of the internal data formats. > This was a security risk before and would be a much bigger one now, > seeing that we allow COPY BINARY FROM STDIN to unprivileged users. It > is trivial to crash the backend by feeding it bad internal-format > data. Well in that case the docs need attention. They describe the "envelope" surrounding the tuples, but no mention is made of the format they are in. It is reasonable to assume that this format was the native binary format, as in earlier releases. I've got applications which create binary "bulkload" files which are loaded into the database using COPY FROM. Currently they write the data out using simple fwrite calls. What do I need to do to make this code work with 7.4? Is there any docs describing the "binary" format for each of the datatypes or do I need to reverse-engineer a dump file or look in the source? Are the routines in libpq/pqformat.c intended to be used by client applications to read/write the binary COPY files? If so they also need documented in the libpq docs and that documentation linked to from the COPY docs. > (I don't believe that the patch works anyway, given that you aren't doing > anything to disable use of the per-datatype receive routine. It might > work as-is for text fields, and for integers on bigendian machines, but > not for much else.) Yeah, I didn't spend a lot of effort in that respect - after all I said myself I didn't see the patch being accepted... > We are not going back to the pre-7.4 format. Sorry. Well as pointed out in my earlier message nothing has changed which requires the format to change - there is no real reason it's now "PGCOPY" and the integer layout field has disappeared. The change for the byte swapping should have been indicated by an entry in the flags field. I am still willing to make a patch which does this (to aid those writing COPY format files) and to fully support the reading of the old format tuples. However i'm not going to waste both our time if this patch is not going to be positively considered... I think it's worthwhile reiterating that this change will be a real pain for PostgreSQL users when migrating to 7.4. To be honest i'd probably stick with 7.3 until the subsequent major release. Have a think what benefit this incompatibility gives users of COPY BINARY... I can't think of much use of byte swapping when 99% of the use of COPY BINARY FROM is to improve performance over using INSERT. Both the reader and writer will be using the same binary integer/float/etc formats! So, will I look at implementing these changes? Or not? L. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.4 COPY BINARY Format Change
Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > Well in that case the docs need attention. They describe the > > "envelope" surrounding the tuples, but no mention is made of the > > format they are in. It is reasonable to assume that this format was > > the native binary format, as in earlier releases. > Yeah, there should be some mention of that in the COPY ref page I guess > --- it's mentioned in the frontend protocol chapter, but not under COPY. > In my defense I'd point out that the contents of individual fields have > never been documented under COPY. True, the docs have always skipped the specifics for the tuples. But now that the format has evolved beyond a simple dump of the bytes the tuple format does need discussing. > > What do I need to do to make this > > code work with 7.4? Is there any docs describing the "binary" format > > for each of the datatypes or do I need to reverse-engineer a dump file > > or look in the source? > ATM, I'd recommend looking in the sources to see what the datatype > send/receive routines do. > > I have been thinking about documenting the binary formats during beta, > but am unsure where to put the info. We never documented the internal > formats before either, so there's no obvious place. Perhaps the documentation of the binary format should be taken out of the COPY docs and moved into the client interfaces documentation? the COPY docs would of course reference the new location. Just now the tuples could be "documented" simply by referring the reader to the relevant functions in the relevant source files. After all the source is the best documentation for this sort of thing. > > Are the routines in libpq/pqformat.c intended > > to be used by client applications to read/write the binary COPY files? > They are not designed to be used outside the backend environment, > although possibly some enterprising person could adapt them. I am not > sure there's any value in it though. Copying the backend code helps > only if what you want to get out of the transmission is the same as the > backend's internal format, which for anything more complex than > int/float/text seems a bit dubious. I think there is a lot of use for a binary COPY file API within libpq - routines to open a file, write/read a header and write/read common datatypes. This would remove the need for most people using the binary version of COPY to even know the file format. This would also isolate people who use this API from any future changes. Would libpq or contrib be the best place for this? Would you agree this is a good idea for 7.4? I've already got something along these lines: extern FILE *lofsdb_Bulk_Open(char **filename); extern void lofsdb_Bulk_Close(FILE *f, char *filename); extern void lofsdb_Bulk_Write_NCols(FILE *f, short ncols); extern void lofsdb_Bulk_Write(FILE *f, void *data, size_t sz, size_t count, short ind); extern void lofsdb_Bulk_WriteText(FILE *f, char *data, short ind); extern void lofsdb_Bulk_WriteBytea(FILE *f, char *data, size_t len, short ind); extern void lofsdb_Bulk_WriteTime(FILE *f, double t, short ind); extern void lofsdb_Bulk_WriteTimeNow(FILE *f); which could form the basis of a contrib module to handle writing out 7.1 through to 7.4 format files. Naturally lofsdb_Bulk_Write needs to go and be replaced by specific functions. > > Well as pointed out in my earlier message nothing has changed which > > requires the format to change - there is no real reason it's now > > "PGCOPY" and the integer layout field has disappeared. > Given that the interpretation of the field contents has changed > drastically, I thought it better to make an obvious incompatible > change. We could perhaps have kept the skeleton the same, but to > what end? An app trying to read or write the file as if it were > pre-7.4 data would fail miserably anyway. Yeah, but someone (actually you!) went to the effort of making the 7.1 format extensible and documenting it as such... It could have handled the changes. > > I am still willing to make a patch which does this (to aid those > > writing COPY format files) and to fully support the reading of the old > > format tuples. However i'm not going to waste both our time if this > > patch is not going to be positively considered... > My vote will be to reject it because of the security problem. In which case I think my time would be better spent looking at the API described above. > > I can't think of much use of byte swapping when 99% of the > > use of COPY BINARY FROM is to improve performance over using > > INSERT. Both the reader and writer will be using the same binary > > intege
Re: [HACKERS] 7.4 COPY BINARY Format Change
Tom, Tom Lane writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > However, is COPY BINARY meant/designed to be used as transfer or > > backup mechanism? > > I think you're overlooking a key consideration: COPY BINARY is not > an isolated feature anymore. By design it uses the same data > representations as are used for binary query parameters and results > in the rest of the 7.4 FE/BE protocol. Yeah, what i've overlooked is that an implementation detail now forms part of an external interface into PostgreSQL - this is a major change. > I could see some value in providing byte-swapping routines in libpq > to convert between local and network representations for integers and > floats. The underlying facilities (ntohl etc) are readily available, > of course, but it's a small matter that is easy to get wrong. > > I'm not sure it's worth packaging up COPY BINARY logic per se. I think > you'd end up with an API not materially simpler than dealing with the > format directly. And I'm unconvinced it'd actually be used widely, > whereas I do expect binary transfer of individual values to be common. Would I be right is guessing a binary CURSOR would return in values in the same format as a binary COPY, hence your expectation of more individual transfers/conversions? Actually with the new FE/BE protocol there is little call for the binary cursor now, yeah? What I proposed in my email yesterday is really just completing the new functions (PQnfields, PQputCopyData, PQgetCopyData and friends) described at: http://developer.postgresql.org/docs/postgres/libpq-copy.html so they don't stop at just giving you a blob of binary data and saying it has n fields - functions would be available to iterate over the fields and get the data out in a format which is immediately useful. Without this do you not think PQgetCopyData is of limited use except for being used by psql (which I guess isn't using it yet). Same for the writing functions. This is slightly different from my earlier example (on the connection rather than file-based) but functionally similar. BTW, do you have any examples of using PQgetCopyData - none in the source and can't find anything with Google. Regards, Lee. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Thread-safe configuration option appears to
Larry Rosenman writes: > What the fr*** harm is it in passing -D_REENTRANT into the libpq build on > UnixWare > irregardless of the --with-threads* flag? Indeed for every other sane system out there. Folk are messing around with the thread stuff using here-say as knowledge. We want to compile with _REENTRANT, always! Anything needed in ADDITON to _REENTRANT (i.e thread link flags for libecpg) can be discovered by configure and applied to the build. Even for the rest of PostgreSQL, it will work A-OK if we are already sticking to the rules (i.e. including when using errno). L. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] --enable-thread-safety broken + patch regressions
Bruce, the changes you made yesterday to configure for --enable-thread-safety have broken the build, at least for Linux on Redhat 9. Also, I took the opportunity to look at port/threads.c. It is missing important functionality compaired to the patch I originally submitted. For getpwuid_r, gethostbyname_r and strerror_r there are three possible scenarios: 1. The OS doesn't have it (but the non _r function can still be thread safe (i.e. HPUX 11)). 2. The OS has it, but the implmentation doesn't match the POSIX spec. 3. The OS has it, and the implmentation matches the POSIX spec. Case 3 is not being considered. In my original patch this was handled by the pqGetpwuid etc functions simply being defined to getpwuid_r (except for pqStrerror). I remember discussing with you that the implementation of pqStrerror didn't really need the distinction between the two _r versions. However I think the others do, and the native/correct _r calls should be #defined in if they match the POSIX spec. It's also worth considering that when the _r function is available AND the normal function is also thread-safe then the _r version should still be used since it has a clean API which removes unneeded locking within the old function. I've still got the latest (and earlier with some configure work) patches I submitted up at: http://services.csl.co.uk/postgresql/ Thanks, Lee. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] threads/UnixWare
I've not been keeping up with the thread re who has what version of getpwuid_r... But just to clarify things the "right" version is: int getpwuid_r(uid_t uid, struct passwd *pwd, char *buffer, size_t bufsize, struct passwd **result); documented at: http://www.opengroup.org/onlinepubs/007904975/functions/getpwuid.html My email to Bruce which he tacked onto his reply is a good summary of the current problems with the thread stuff. L. Larry Rosenman writes: > > > --On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian > <[EMAIL PROTECTED]> wrote: > > > > > Actually, your getpwuid_r is the old, pre-POSIX format. The attached > > email has the configure tests. I was hoping we wouldn't need them, but > > it seems we may. > Err, SCO claims SUSv2, the Single Unix Specification Version 2. V3 > **JUST** came > out. > > I'll look at Lee's stuff. > > > > > > -- > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] > US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 > ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Release changes
Bruce, I know it's a bit picky but both below should be along the lines of "Allow thread-safe library" since they are not "threaded" per se. L. Bruce Momjian writes: > Libpq > Allow threaded with --enable-thread-safety (Lee Kindness, Bruce) > Miscellaneous Interfaces > Allow threaded ecpg with --enable-thread-safety (Lee Kindness, Bruce) ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] 7.4 COPY BINARY Format Change
I've just sent off patches to pgsql-patches to: 1. Slight clarification to the COPY BINARY format docs 2. A contrib/binarycopy module which wraps-up the detail of creating a file which can be used as input to COPY BINARY. User can create either 7.1 or 7.4 format files using the same API, without needing to know the file format, without needing to know the individual binary format of each field and without needing to explicitly byte-swap. #2 will be used extensively within Concept Systems code which interfaces to PostgreSQL. It really simplifies the creation of the binary files. Thanks, Lee. Lee Kindness writes: > Tom Lane writes: > > Lee Kindness <[EMAIL PROTECTED]> writes: > > > Well in that case the docs need attention. They describe the > > > "envelope" surrounding the tuples, but no mention is made of the > > > format they are in. It is reasonable to assume that this format was > > > the native binary format, as in earlier releases. > > Yeah, there should be some mention of that in the COPY ref page I guess > > --- it's mentioned in the frontend protocol chapter, but not under COPY. > > In my defense I'd point out that the contents of individual fields have > > never been documented under COPY. > > True, the docs have always skipped the specifics for the > tuples. But now that the format has evolved beyond a simple dump of > the bytes the tuple format does need discussing. > > > > What do I need to do to make this > > > code work with 7.4? Is there any docs describing the "binary" format > > > for each of the datatypes or do I need to reverse-engineer a dump file > > > or look in the source? > > ATM, I'd recommend looking in the sources to see what the datatype > > send/receive routines do. > > > > I have been thinking about documenting the binary formats during beta, > > but am unsure where to put the info. We never documented the internal > > formats before either, so there's no obvious place. > > Perhaps the documentation of the binary format should be taken out of > the COPY docs and moved into the client interfaces documentation? the > COPY docs would of course reference the new location. Just now the > tuples could be "documented" simply by referring the reader to the > relevant functions in the relevant source files. After all the source > is the best documentation for this sort of thing. > > > > Are the routines in libpq/pqformat.c intended > > > to be used by client applications to read/write the binary COPY files? > > They are not designed to be used outside the backend environment, > > although possibly some enterprising person could adapt them. I am not > > sure there's any value in it though. Copying the backend code helps > > only if what you want to get out of the transmission is the same as the > > backend's internal format, which for anything more complex than > > int/float/text seems a bit dubious. > > I think there is a lot of use for a binary COPY file API within libpq > - routines to open a file, write/read a header and write/read common > datatypes. This would remove the need for most people using the binary > version of COPY to even know the file format. This would also isolate > people who use this API from any future changes. > > Would libpq or contrib be the best place for this? Would you agree > this is a good idea for 7.4? I've already got something along these > lines: > > extern FILE *lofsdb_Bulk_Open(char **filename); > extern void lofsdb_Bulk_Close(FILE *f, char *filename); > extern void lofsdb_Bulk_Write_NCols(FILE *f, short ncols); > extern void lofsdb_Bulk_Write(FILE *f, void *data, size_t sz, size_t count, short > ind); > extern void lofsdb_Bulk_WriteText(FILE *f, char *data, short ind); > extern void lofsdb_Bulk_WriteBytea(FILE *f, char *data, size_t len, short ind); > extern void lofsdb_Bulk_WriteTime(FILE *f, double t, short ind); > extern void lofsdb_Bulk_WriteTimeNow(FILE *f); > > which could form the basis of a contrib module to handle writing out > 7.1 through to 7.4 format files. Naturally lofsdb_Bulk_Write needs to > go and be replaced by specific functions. > > > > Well as pointed out in my earlier message nothing has changed which > > > requires the format to change - there is no real reason it's now > > > "PGCOPY" and the integer layout field has disappeared. > > Given that the interpretation of the field contents has changed > > drastically, I thought it better to make an obvious incompat
Re: [HACKERS] threads/UnixWare
Larry, haven't managed to look at that patch... But stuffed for time just now - just about to head off for the weekend. I'm hoping to spend a bit of time on this on Tuesday! So, i'll see how things have progressed then. L. Larry Rosenman writes: > --On Friday, August 08, 2003 11:53:34 +0100 Lee Kindness > > I've not been keeping up with the thread re who has what version of > > getpwuid_r... But just to clarify things the "right" version is: > > > > int getpwuid_r(uid_t uid, struct passwd *pwd, char *buffer, > > size_t bufsize, struct passwd **result); > > > > documented at: > > > > http://www.opengroup.org/onlinepubs/007904975/functions/getpwuid.html > > > > My email to Bruce which he tacked onto his reply is a good summary of > > the current problems with the thread stuff. > Ok. Lee: Did you see the patch I posted? Can you help on the configure > test(s) > needed to set HAVE_POSIX_GETPWUID_R? > > I **THINK** I did the patch right to handle it if it's set. I forced it > for UnixWare > and it seems to work. > > Larry Rosenman writes: > > > --On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian > > > <[EMAIL PROTECTED]> wrote: > > > > Actually, your getpwuid_r is the old, pre-POSIX format. The attached > > > > email has the configure tests. I was hoping we wouldn't need them, > > but > > it seems we may. > > > Err, SCO claims SUSv2, the Single Unix Specification Version 2. V3 > > > **JUST** came > > > out. > > > > > > I'll look at Lee's stuff. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] --enable-thread-safety broken + patch regressions
Bruce Momjian writes: > Lee Kindness wrote: > > Bruce, the changes you made yesterday to configure for > > --enable-thread-safety have broken the build, at least for Linux on > > Redhat 9. > OK, how did I break things? Can you show me the failure. After a: ./configure --prefix=/var/lib/pgsql/74b --enable-thread-safety a compile of port/threads.c fails with: gcc -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -I../../src/include -c -o threads.o threads.c threads.c: In function `pqGetpwuid': threads.c:49: too few arguments to function `getpwuid_r' threads.c:49: warning: assignment makes pointer from integer without a cast threads.c: In function `pqGethostbyname': threads.c:74: warning: passing arg 5 of `gethostbyname_r' from incompatible pointer type threads.c:74: too few arguments to function `gethostbyname_r' threads.c:74: warning: assignment makes pointer from integer without a cast And this is what brought me to the issue below... The POSIX version are getting picked up but handled like broken versions... What info would help here? config.log? > > Also, I took the opportunity to look at port/threads.c. It is missing > > important functionality compaired to the patch I originally > > submitted. For getpwuid_r, gethostbyname_r and strerror_r there are > > three possible scenarios: > > > > 1. The OS doesn't have it (but the non _r function can still be thread > > safe (i.e. HPUX 11)). > > > > 2. The OS has it, but the implmentation doesn't match the POSIX spec. > > > > 3. The OS has it, and the implmentation matches the POSIX spec. > > > > Case 3 is not being considered. In my original patch this was handled > > by the pqGetpwuid etc functions simply being defined to getpwuid_r > > (except for pqStrerror). > > I believe what we did was that there was no way to test for #3 (at the > time), so we just went with the normal function and the POSIX one, and > were going to see what happened to see if anyone needed the non-POSIX > one. Do we have any platforms that need it? Well the code in thread.c will only work if the _r function is the broken non-POSIX version. > > I remember discussing with you that the implementation of pqStrerror > > didn't really need the distinction between the two _r > > versions. However I think the others do, and the native/correct _r > > calls should be #defined in if they match the POSIX spec. > > > > It's also worth considering that when the _r function is available AND > > the normal function is also thread-safe then the _r version should > > still be used since it has a clean API which removes unneeded locking > > within the old function. > > We have that already. Have you looked in the template files. There you > control whether you should use _r functions. > > Also, I doubt that the locking really has any performance hit to > it. As do I, but people are using this as an argument for the dumb libpq_r library idea! > > I've still got the latest (and earlier with some configure work) > > patches I submitted up at: > I just looked at this --- I have not seem them before. Everything on that page has been posted/linked to hackers and patches. > Seems theading requires four things, potentially: > > compile flags > link flags > link libraries > special functions > > While your configure checks can detect the existance of the last one, > they don't tell us what to do if they don't exist --- are the normal > ones thread-safe. > > So, the big question is whether we gain by having detection of non-posix > functions or whether it is better to just have template control it. We want to define & implement wrapper functions with the same API as the POSIX versions of the _r functions we need. If we have the POSIX versions then the replacement simply needs to be a #define to it. Otherwise a stub function is implemented to wrap around either the broken/old _r function or the legacy function (which may be thread safe). It's getting to the stage I think this isn't going to be done correctly in time for 7.4... L. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Unixware Patch (Was: Re: Beta2 Tag'd and Bundled ...)
Bruce Momjian writes: > Marc G. Fournier wrote: > > On Sat, 30 Aug 2003, Bruce Momjian wrote: > > > > > Yes, and that is the complex part because _some_ non-*_r functions are > > > thread-safe, and some are not. I have to determine if we have other > > > such platforms before I figure out how to fix it in the cleanest way. > > > > Long shot ... is there some way of writing a configure test for this? > > Right now, it sounds like we're going to be hitting alot of trial-n-error > > if there isn't ... > > How would we test if a function is thread-safe? I can't think of a > reliable way, and hence my warning that this adjusting could take a > while. You don't... and you simply shouldn't care. If there is a_r version available then we should use it - even if the plain version is "safe". Just think of this as is it were a normal "port" issue. If an OS doesn't have zxczxc_r() then we need to write a zxczxc_r() wrapper function which calls zxczxc() and has the same signature as zxczxc_r(). L. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] thread safety
Probably because I worked on thread safety and produced a patch. If someone done the same for PITR and produced a patch i'm sure it would have generated much more interest. I couldn't have done PITR, so no loss of resource there. Was Bruce planning to do the PITR work? If so I guess a lot of his time's been spent on integrating patches and the like - leaving less time for other developments. L. Mendola Gaetano writes: > I seen on this list a lot of energy ( also little flames involving SCO > & Co. ) spent on thread safety; > was really necessary spent so much energy in this direction? > I was at Fosdem in Bruxelles ( I spoke there about the use > of postgres in my project ) and I seen al people there > was really exicited about the anticipation of Bruce Momjian > about the PITR in 7.4 but how we know there was no time for it > > May be I'm wrong but I'd like know why thread safety was so > necessary. > > > Regards > Gaetano Mendola > ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Unixware Patch (Was: Re: Beta2 Tag'd and Bundled ...)
Guys, too much thought is being spent on this... 1. For the _r functions we "need" we should ALWAYS use them if the system we are building on has them - they WILL be thread-safe. 2. If the system is missing a _r function then we implement a wrapper to call the normal non-_r version. However we do NOT make this wrapper call thread-safe - we assume the non-_r version already is. Together both steps ensure the code calling the _r function is readable (just one function signature) and that libpq will be thread-safe if the system C library is thread-safe. So this will catch all modern UNIX OSs. We really don't want to go deeper than this - of we do so we're wasting time on odd-ball systems that aren't thread-safe anyway - so for them libpq's thread-safety is of no consequence. L. ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Unixware Patch (Was: Re: Beta2 Tag'd and Bundled ...)
"Tom Lane" <[EMAIL PROTECTED]> writes: > Lee Kindness <[EMAIL PROTECTED]> writes: > > Guys, too much thought is being spent on this... > > 1. For the _r functions we "need" we should ALWAYS use them if the > > system we are building on has them - they WILL be thread-safe. > > > 2. If the system is missing a _r function then we implement a wrapper > > to call the normal non-_r version. However we do NOT make this wrapper > > call thread-safe - we assume the non-_r version already is. > > That assumption is exactly what Peter is unhappy about. With the above > approach we will happily build a "thread safe" library on systems that > are in fact not thread safe at all. Peter wants --enable-thread-safety > to fail on non-safe systems. Yeah, I see that as Peter's main objection too. But it really isn't worth the trouble, on such systems a user's app will be so horribly broken WRT threads anyway. It isn't our job to point out the shortcomings of their system. Really such systems don't exist - if libc doesn't have _r calls and the traditional ones are not thread-safe then do you think it'll have a threading library which can run >1 thread concurrently? On such a system the users troubles will start long before PostgreSQL if they play with threads. L. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Unixware Patch (Was: Re: Beta2 Tag'd and Bundled ...)
"Larry Rosenman" <[EMAIL PROTECTED]> writes: > then how do we *PROVE* thread-safety on a particular platform? You're not going to be able to prove it anyway! L. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Unixware Patch (Was: Re: Beta2 Tag'd and Bundled ...)
Tom Lane writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > On the other hand, things like, getpwnam, strtok, etc have non-thread-safe > > APIs. They can never be made thread-safe. The *_r versions of these functions > > are standardized and required. If they don't exist then the platform simply > > does not support threads. > > This statement is simply false. A platform can build thread-safe > versions of those "unsafe" APIs if it makes the return values point > to thread-local storage. Some BSDs do it that way. Accordingly, any > simplistic "we must have _r to be thread-safe" approach is > incorrect. No, it's not. Using the _r functions on such systems is BETTER because the API is clean and the function can be implmented in a reentrant and thread-safe fashion wuithout the need for thread local storage or mutex locking. L. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
