[HACKERS] additional GCC warnings
Recent versions of GCC support some additional warning flags that I think would be useful to enable for building PostgreSQL: -Wmissing-declarations (Warn if a global function is defined without a previous declaration.) -Wdeclaration-after-statement (Recent versions of GCC allow declarations and statements to be intermixed in C; enabling this flag would enforce the current convention of avoiding this style.) -Wold-style-definition (Warn if KR-style function definitions are used.) For the most part it won't take a lot of work to enable these flags. Some code we imported into the tree from elsewhere might need to be updated for -Wold-style-definition, but that should be easy. Since some of these flags were added to GCC relatively recently, we would probably need to test if the local gcc supports the flags via configure. Comments? -Neil ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.4 changes
Tom Lane wrote: If anyone has any pending 7.4 fixes, getting them in in the next few days would be a Good Plan. Do we want to backport tighter security for plperl? In particular, insisting on Safe.pm = 2.09 and removing the :base_io set of ops? cheers andrew ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] 7.4 changes
Andrew Dunstan wrote: Tom Lane wrote: If anyone has any pending 7.4 fixes, getting them in in the next few days would be a Good Plan. Do we want to backport tighter security for plperl? In particular, insisting on Safe.pm = 2.09 and removing the :base_io set of ops? And it would also be nice if we could add contrib/cube/expected/cube_1.out to the 7.4 branch, I think, so that more platforms could pass the contrib installcheck tests. cheers andrew ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] spinlocks: generalizing non-locking test
Currently, the assembly for TAS() on x86 does a non-locking test before using an atomic operation to attempt to acquire the spinlock: __asm__ __volatile__( cmpb$0,%1 \n jne 1f \n lock\n xchgb %0,%1 \n 1: \n : +q(_res), +m(*lock) : : memory, cc); The reason this is a good idea is that if we fail to immediately acquire the spinlock, s_lock() will spin SPINS_PER_DELAY times in userspace calling TAS() each time before going to sleep. If we do an atomic operation for each spin, this generates a lot more bus traffic than is necessary. Doing a non-locking test (followed by an atomic operation to acquire the spinlock if appropriate) is therefore better on SMP systems. Currently x86 is the only platform on which we do this -- ISTM that all the other platforms that implement spinlocks via atomic operations could benefit from this technique. We could fix this by tweaking each platform's assembler to add a non-blocking test, but there might be an easier way. Rather than modifying platform-specific assembler, I believe this C sequence is equivalent to the non-locking test: volatile slock_t *lock = ...; if (*lock == 0) TAS(lock); Because the lock variable is volatile, the compiler should reload it from memory for each loop iteration. (If this is actually not a sufficient non-locking test, please let me know...) We could add a new s_lock.h macro, TAS_INNER_LOOP(), whose default implementation would be: #define TAS_INNER_LOOP(lock) \ if ((*lock) == 0) \ TAS(lock); And then remove the x86-specific non-locking test from TAS. Comments? -Neil ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Nearing final release?
Greg Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: I think the cleanest solution is probably to add a state flag indicating whether ParseComplete should generate a PGresult or not. Adding a flag is going to be enough for synchronous queries. But it seems like it has no chance of working for asynchronous queries. It would have to keep track of what all the pending requests are and the expected results. Say what? Neither libpq nor the backend support multiple queries in-flight at the same time. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] 7.4 changes
Andrew Dunstan [EMAIL PROTECTED] writes: Do we want to backport tighter security for plperl? In particular, insisting on Safe.pm = 2.09 and removing the :base_io set of ops? I'd vote not: 7.4.5 = 7.4.6 is not an update that people would expect to break their plperl code ... regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] spinlocks: generalizing non-locking test
Neil Conway [EMAIL PROTECTED] writes: Currently x86 is the only platform on which we do this -- ISTM that all the other platforms that implement spinlocks via atomic operations could benefit from this technique. Do you have any actual evidence for that opinion? ISTM this is dependent on a large set of assumptions about the CPU's bus behavior, boiling down to the conclusion that an extra conditional branch is cheaper than a locked bus cycle. It would be a serious error to transfer that conclusion to non-Intel chips without investigation. (For that matter, I'm not that thrilled about it even for the Intel chips, since the extra test is certainly 100% wasted cycles on any single-CPU machine, or indeed anywhere that the lock is not contended for.) I believe that an extra test would be a dead loss on PPC, for instance, because the initial load is already nonblocking there. if (*lock == 0) TAS(lock); This will certainly not work on HPPA, and it really shouldn't assume that zero is the non-locked state, and what happened to testing the TAS result? On the whole it seems far easier to stick an extra load into the asm code on those platforms where we think it's a win. Especially since we have already done that ;-) regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Open Items
I have annotated the open items: --- PostgreSQL 8.0 Open Items = Current version at http://candle.pha.pa.us/cgi-bin/pgopenitems. Changes --- * Win32 o fix query cancel in psql This requires a new thread to handle the control-C signal o fix shared memory on Win2k terminal server We might be able to just mark this as not supported. o Handle lost signals on backend startup (eg. shutdown, config file changes, etc); signals are SIG_DFL on startup The problem here is that the postmaster might send signals to a child before the signal handlers are installed. We don't have this problem on unix because we fork and inherit the signal handlers. * Tablespace o fix ambiguity for objects using default tablespaces CREATE TABLE test(a int4) tablespace loc; CREATE UNIQUE INDEX test_a_idx3 ON test(a) tablespace pg_default; Index test_a_idx3 has a tablespace value of zero. pg_get_indexdef() thinks the index is in loc even though it is in pg_default. And if you move test the index is thought to move too. The confusion is that there are not separate defaults for database/schema/table. o fix case where template db already uses target tablespace -- from database test CREATE tablespace blah location '/bjm/tmp'; CREATE TABLE foo (i int) tablespace blah; CREATE DATABASE foo tablespace blah template test; ERROR: could not initialize database directory DETAIL: Directory /u/pgsql/data/pg_tblspc/17232/17235 already exists. o remove non-portable TABLESPACE clause from CREATE TABLE and use a new default_tablespace SET variable This same method is used for WITH/WITHOUT OIDS. * allow libpq to check parameterized data types * adjust bgwriter defaults, allow disabling * synchonize supported encodings and docs Completed Since Previoius Beta -- * cleanup FRONTEND use in /port, malloc, elog * update encoding list to include win1250 * make pgxs install by default * Tablespace o fix error message when creating objects in schema that has a dropped tablespace as its default * Win32 o disable readline-required psql options o fix SSL compiles o add binary version stamps o fix signal-safe socket handler for SSL o start pg_autovacuum easily o remove log timezone string from log_line_prefix '%t' o fix MinGW libpq to export only required symbols o fix MSVC build to export SSL symbols -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] strange result from contrib/seg regression on windows
Andrew Dunstan [EMAIL PROTECTED] writes: Does anyone have a clue why? Evidently your sscanf isn't setting errno for overflow --- look at seg_atof() in segparse.y. This code shows several other signs of severe brain death, actually, like leaking 256 bytes on each call for an error message buffer that it doesn't even need :-( I'd recommend ripping the whole thing out and replacing with a call to float4in. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Open Items
Bruce Momjian [EMAIL PROTECTED] writes: o fix ambiguity for objects using default tablespaces CREATE TABLE test(a int4) tablespace loc; CREATE UNIQUE INDEX test_a_idx3 ON test(a) tablespace pg_default; Index test_a_idx3 has a tablespace value of zero. pg_get_indexdef() thinks the index is in loc even though it is in pg_default. And if you move test the index is thought to move too. The confusion is that there are not separate defaults for database/schema/table. I don't think it's nearly as complicated as all that; it's just a thinko in pg_get_indexdef. http://developer.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/adt/ruleutils.c.diff?r1=1.182r2=1.183 regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] strange result from contrib/seg regression on windows
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Does anyone have a clue why? Evidently your sscanf isn't setting errno for overflow --- look at seg_atof() in segparse.y. This code shows several other signs of severe brain death, actually, like leaking 256 bytes on each call for an error message buffer that it doesn't even need :-( I'd recommend ripping the whole thing out and replacing with a call to float4in. Yeah. If anyone can do this quickly I'd be grateful. Otherwise it will probably be a few days before I get it done. Alternatively, could we not just use strtof instead of sscanf? cheers andrew ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] plans for bitmap indexes?
Tom Lane wrote: I believe that the term bitmap index is also used with a different meaning wherein it actually does describe a particular kind of on-disk index structure, with one bit per table row. IMHO building in-memory bitmaps (the first idea) is a very good idea to pursue for Postgres. I'm not at all sold on on-disk bitmap indexes, though ... those I suspect *are* sufficiently replaced by partial indexes. I believe that the benefit of on-disk bitmap indexes is supposed to be reduced storage size (compared to btree). In the cases where I have put them to use, they certainly occupy considerably less disk than a comparable btree index - provided there are not too many district values in the indexed column. regards Mark ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] tsearch2 regression test path problem
The makefile for tsearch2 has this rule: tsearch2.sql: tsearch.sql.in sed -e 's,MODULE_PATHNAME,$$libdir/$(MODULE_big),g' \ -e 's,DATA_PATH,$(datadir)/contrib,g' $ $@ The problem is that on Windows, $(datadir) is typically a virtual Msys path (or least typically it is for me) which is not understood by a running backend, so when it goes to load the file with make installcheck it can't find it. I'm not sure if there is some nicely portable way around this. Any suggestions? cheers andrew ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] DETERMINISTIC as synonym for IMMUTABLE
Hello, While reviewing PostgreSQL 8's SQL:2003 conformance (see the documentation mailing list), I've come across a - possibly - easy-to-implement conformance improvement for CREATE FUNCTION: SQL:2003 uses the word DETERMINISTIC where PostgreSQL uses IMMUTABLE. And SQL:2003 uses NOT DETERMINISTIC where PostgreSQL uses STABLE or VOLATILE. I suggest that DETERMINISTIC be added as a synonym for IMMUTABLE and that NOT DETERMINISTIC be added as an alias for VOLATILE in PostgreSQL's grammar for CREATE FUNCTION. I might try creating a patch, but I'd rather spend my time finishing the conformance review. -- Greetings from Troels Arvin, Copenhagen, Denmark ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] DETERMINISTIC as synonym for IMMUTABLE
Troels Arvin [EMAIL PROTECTED] writes: I suggest that DETERMINISTIC be added as a synonym for IMMUTABLE and that NOT DETERMINISTIC be added as an alias for VOLATILE in PostgreSQL's grammar for CREATE FUNCTION. These do NOT mean the same thing. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] tsearch2 regression test path problem
Andrew Dunstan [EMAIL PROTECTED] writes: The problem is that on Windows, $(datadir) is typically a virtual Msys path (or least typically it is for me) which is not understood by a running backend, Huh? Explain that in English please ... because I would think that what you are suggesting the behavior is would break a lot more things than just tsearch2. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] DETERMINISTIC as synonym for IMMUTABLE
On Sun, 17 Oct 2004 18:08:00 -0400, Tom Lane wrote: I suggest that DETERMINISTIC be added as a synonym for IMMUTABLE and that NOT DETERMINISTIC be added as an alias for VOLATILE in PostgreSQL's grammar for CREATE FUNCTION. These do NOT mean the same thing. I'm having a hard time seeing the difference between DETERMINISTIC and IMMUTABLE. My suggestion for NOT DETERMINISTIC==VOLATILE is because VOLATILE seems to be the least strict of the three PostgreSQL volatility categories. Do you disagree on both, or just the last one? -- Greetings from Troels Arvin, Copenhagen, Denmark ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] tsearch2 regression test path problem
Andrew Dunstan [EMAIL PROTECTED] writes: I'm not sure if there is some nicely portable way around this. Any suggestions? Now that I think about it, we already had an open issue with tsearch2, which is that the way it's doing this, a relocatable installation is impossible. I think the most reasonable fix is to change the readstoplist() function so that it will accept relative paths and interpret them as relative to the share directory (which is $datadir as seen by the Makefiles; we can get the correct runtime location from get_share_path). Then we can just put contrib/english.stop into the pg_ts_dict entry and not have to interpolate any build-time path at all. Will see about doing this. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] tsearch2 regression test path problem
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: The problem is that on Windows, $(datadir) is typically a virtual Msys path (or least typically it is for me) which is not understood by a running backend, Huh? Explain that in English please ... because I would think that what you are suggesting the behavior is would break a lot more things than just tsearch2. It does not, at least for me. In fact I spent many hours a few months ago making sure that we could run make check from Msys, despite the dual-set-of-paths - see the changes to pg_regress.sh that were made around then. In fact, of the whole gamut of checks my buildfarm script runs (make check, make installcheck, and contrib's make installcheck), this is the *only* failure due to a path problem. cheers andrew ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] tsearch2 regression test path problem
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: I'm not sure if there is some nicely portable way around this. Any suggestions? Now that I think about it, we already had an open issue with tsearch2, which is that the way it's doing this, a relocatable installation is impossible. I think the most reasonable fix is to change the readstoplist() function so that it will accept relative paths and interpret them as relative to the share directory (which is $datadir as seen by the Makefiles; we can get the correct runtime location from get_share_path). Then we can just put contrib/english.stop into the pg_ts_dict entry and not have to interpolate any build-time path at all. Will see about doing this. Excellent. cheers andrew ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] DETERMINISTIC as synonym for IMMUTABLE
Troels Arvin [EMAIL PROTECTED] writes: On Sun, 17 Oct 2004 18:08:00 -0400, Tom Lane wrote: These do NOT mean the same thing. I'm having a hard time seeing the difference between DETERMINISTIC and IMMUTABLE. Well, the spec is somewhat self-contradictory on the point, but I think their intention is to model it after their notion of a deterministic query: A query expression or query specification is possibly non- deterministic if an SQL-implementation might, at two different times where the state of the SQL-data is the same, produce results that differ by more than the order of the rows due to General Rules that specify implementation-dependent behavior. [SQL99 4.17] Notice that it is okay for a deterministic query to produce different results when the content of the database changes; therefore this is not IMMUTABLE in our terms. It is however stronger than our STABLE condition (for example, now() is STABLE but is not deterministic per the above definition). It appears to me that they are thinking of functions like SELECT value FROM table WHERE pkey = $1 which is deterministic per their definition and also according to (what I think is) the common meaning of deterministic. We could label this function as STABLE, but not IMMUTABLE; however we have no category that captures the notion that it can't change as long as the database content doesn't change. What it actually says about deterministic functions in 4.23 is: An SQL-invoked routine is either deterministic or possibly non- deterministic. An SQL-invoked function that is deterministic always returns the same return value for a given list of SQL argument values. An SQL-invoked procedure that is deterministic always returns the same values in its output and inout SQL parameters for a given list of SQL argument values. An SQL-invoked routine is possibly non-deterministic if, during invocation of that SQL- invoked routine, an SQL-implementation might, at two different times when the state of the SQL-data is the same, produce unequal results due to General Rules that specify implementation-dependent behavior. This is clearly bogus as written since it claims that there are only two possibilities when there are more than two. Any ordinary function that selects from the database will satisfy neither their deterministic nor their possibly non-deterministic definitions. I think that they meant to define SQL functions as nondeterministic if they act like or contain nondeterministic queries; for instance 13.5 says 3) An SQL procedure statement S is possibly non-deterministic if and only if at least one of the following is satisfied: a) S is a select statement: single row that is possibly non- deterministic. b) S contains a routine invocation whose subject routine is an SQL-invoked routine that possibly modifies SQL-data. c) S generally contains a query specification or a query expression that is possibly non-deterministic. d) S generally contains a datetime value function, CURRENT_ USER, CURRENT_ROLE, SESSION_USER, or SYSTEM_USER. Anybody know whether the SQL2003 text clarifies the intent at all? In any case, whether or not you think DETERMINISTIC means IMMUTABLE, I don't think it's very helpful to identify NOT DETERMINISTIC with VOLATILE. As a counterexample, now() is NOT DETERMINISTIC, but it isn't VOLATILE. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Open Items
Bruce Momjian [EMAIL PROTECTED] writes: o remove non-portable TABLESPACE clause from CREATE TABLE and use a new default_tablespace SET variable I'm coming around to the conclusion that this is simply a bad idea. The problem with having such a SET variable is that it plays hob with the existing definition about where schemas and tables get a default tablespace from. Which source wins (the database or schema default tablespace, or the SET variable)? And why? The only really clean way to have a SET variable for this is to forget about schema- or table-based defaults. Do we want to do that? (Hey, it'd solve the problem with schema tablespaces being droppable, because there wouldn't *be* any such thing as a schema's tablespace anymore. But on the whole this seems like a step backward in usability.) What we might want to do is invent a --notablespace option for pg_dump, comparable to --noowner, to let someone make a dump that contains no TABLESPACE clauses. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Open Items
On Sun, 17 Oct 2004, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: o remove non-portable TABLESPACE clause from CREATE TABLE and use a new default_tablespace SET variable I'm coming around to the conclusion that this is simply a bad idea. I feel the same way for more or less the reasons you outline. What we might want to do is invent a --notablespace option for pg_dump, comparable to --noowner, to let someone make a dump that contains no TABLESPACE clauses. That's a useful feature but I'm not sure it solves the problem people originally put (to me at least). User has data in a tablespace on a seperate device. The device crashes fatally and the user needs to restore the database. All the user's dumps contain tablespace clauses because the user did not anticipate the device dying. This, I think, is why people wanted to either ignore tablespace clauses, have an override or something else. I still think, however, that a workable solution is to bring up a new system, create the tablespaces on some online partition, and pg_restore the dump. pg_dump does not dump CREATE TABLESPACE so we wont encounter problems there. Have I missed something there? (Highly likely as I am still pre-coffee). Gavin ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] spinlocks: generalizing non-locking test
On Mon, 2004-10-18 at 04:13, Tom Lane wrote: Do you have any actual evidence for that opinion? ISTM this is dependent on a large set of assumptions about the CPU's bus behavior, boiling down to the conclusion that an extra conditional branch is cheaper than a locked bus cycle. I think the conditional branch is effectively free: we're spinning in a busy-wait loop, so we're effectively throwing away cycles until the lock is free anyway. The important things are: (a) we interfere as little as possible with concurrent system activity while spinning (b) we notice promptly that the lock is free. ISTM that doing a non-locking test before the locking test achieves those two conditions. (For that matter, I'm not that thrilled about it even for the Intel chips, since the extra test is certainly 100% wasted cycles on any single-CPU machine, or indeed anywhere that the lock is not contended for.) Yes, but the proper fix for this is not to spin at all on UP machines; I have a grungy patch to do this that I will clean up and send in for 8.1. if (*lock == 0) TAS(lock); This will certainly not work on HPPA, and it really shouldn't assume that zero is the non-locked state, and what happened to testing the TAS result? Erm, perhaps I should have emphasized that that was pseudo-code :) A more realistic implementation would be: #define TAS_INNER_LOOP(lock) (S_LOCK_FREE(lock) ? TAS(lock) : 1) -Neil ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] additional GCC warnings
On Mon, 2004-10-18 at 03:50, Tom Lane wrote: -Wmissing-declarations (Warn if a global function is defined without a previous declaration.) Hm? We have always used that one. We've always used -Wmissing-prototypes. The documentation states that -Wmissing-prototypes instructs GCC to: Warn if a global function is defined without a previous prototype declaration. This warning is issued even if the definition itself provides a prototype. The aim is to detect global functions that fail to be declared in header files. Whereas -Wmissing-declarations does: Warn if a global function is defined without a previous declaration. Do so even if the definition itself provides a prototype. Use this option to detect global functions that are not declared in header files. Which doesn't make the difference in behavior between the two options clear to me. Can anyone clarify this? BTW, are these changes appropriate for 8.0? -Neil ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] spinlocks: generalizing non-locking test
Neil Conway [EMAIL PROTECTED] writes: On Mon, 2004-10-18 at 04:13, Tom Lane wrote: Do you have any actual evidence for that opinion? ISTM this is dependent on a large set of assumptions about the CPU's bus behavior, boiling down to the conclusion that an extra conditional branch is cheaper than a locked bus cycle. I think the conditional branch is effectively free: we're spinning in a busy-wait loop, so we're effectively throwing away cycles until the lock is free anyway. Only once we've begun to spin. The first time through, it's not at all clear whether the extra test is worthwhile --- it's certainly a win if the lock is always already held, and certainly a loss if the lock is always free, and otherwise you have to do some benchmarking to decide if you want it or not. We have the ASM-level test on those platforms where people seem to think that it is worthwhile, but not everywhere. As near as I can tell, your proposal is to never have the extra test on the initial TAS (because you'll remove it from the ASM) and to have it always on repeated tests (inside the spin loop). This is not what is done elsewhere AFAICS, and I'm not prepared to take on faith that it's a win. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] additional GCC warnings
Neil Conway [EMAIL PROTECTED] writes: -Wmissing-declarations (Warn if a global function is defined without a previous declaration.) Hm? We have always used that one. We've always used -Wmissing-prototypes. We've always used both. See Makefile.global.in: ifeq ($(GCC), yes) CFLAGS += -Wall -Wmissing-prototypes -Wmissing-declarations endif ... Which doesn't make the difference in behavior between the two options clear to me. Can anyone clarify this? Hmm, it looks like -Wmissing-prototypes may be a superset of -Wmissing-declarations --- it seems to say that the latter will be content with a KR style declaration (extern int foo();) but the former will not. If that's a correct reading then we could drop -Wmissing-declarations. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] additional GCC warnings
On Mon, 2004-10-18 at 12:03, Tom Lane wrote: We've always used -Wmissing-prototypes. We've always used both. My apologies -- I don't know where I got the opposite impression. Hmm, it looks like -Wmissing-prototypes may be a superset of -Wmissing-declarations --- it seems to say that the latter will be content with a KR style declaration (extern int foo();) but the former will not. If that's a correct reading then we could drop -Wmissing-declarations. Ok, that makes sense. -Neil ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Open Items
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: o fix ambiguity for objects using default tablespaces CREATE TABLE test(a int4) tablespace loc; CREATE UNIQUE INDEX test_a_idx3 ON test(a) tablespace pg_default; Index test_a_idx3 has a tablespace value of zero. pg_get_indexdef() thinks the index is in loc even though it is in pg_default. And if you move test the index is thought to move too. The confusion is that there are not separate defaults for database/schema/table. I don't think it's nearly as complicated as all that; it's just a thinko in pg_get_indexdef. http://developer.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/adt/ruleutils.c.diff?r1=1.182r2=1.183 OK, I see that now. Item marked as done. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Interesting bug in tablespaces
This has been fixed in current CVS. --- Christopher Kings-Lynne wrote: There is a confusing bug in tablespaces. Here is examples: OK, let's create a table with the 3 possible tablespaces of indexes: test=# create table test(a int4) tablespace loc; CREATE TABLE test=# create unique index test_a_idx on test(a); CREATE INDEX test=# create unique index test_a_idx2 on test(a) tablespace loc; CREATE INDEX test=# create unique index test_a_idx3 on test(a) tablespace pg_default; CREATE INDEX test=# \d test Table public.test Column | Type | Modifiers +-+--- a | integer | Indexes: test_a_idx UNIQUE, btree (a) test_a_idx2 UNIQUE, btree (a) test_a_idx3 UNIQUE, btree (a) Tablespace: loc test=# select relname, reltablespace from pg_class where relname like 'test%'; relname | reltablespace -+--- test| 17229 test_a_idx | 17229 test_a_idx2 | 17229 test_a_idx3 | 0 (4 rows) Note that psql (and pg_dump) will (because of pg_get_indexdef()) think that test_a_idx3 is in tablespace 'loc', even though it's in tablespace 'pg_default'. Now, let's make it worse: test=# alter table test set tablespace loc2; ALTER TABLE test=# \d test Table public.test Column | Type | Modifiers +-+--- a | integer | Indexes: test_a_idx UNIQUE, btree (a) TABLESPACE loc test_a_idx2 UNIQUE, btree (a) TABLESPACE loc test_a_idx3 UNIQUE, btree (a) Tablespace: loc2 test=# select relname, reltablespace from pg_class where relname like 'test%'; relname | reltablespace -+--- test| 17279 test_a_idx | 17229 test_a_idx2 | 17229 test_a_idx3 | 0 (4 rows) Now, it thinks test_a_idx3 is in loc2. pg_dump will dump it like that as well, so when it's restored, test_a_idx3 will be recreated in loc2. Chris ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Open Items
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: o remove non-portable TABLESPACE clause from CREATE TABLE and use a new default_tablespace SET variable I'm coming around to the conclusion that this is simply a bad idea. The problem with having such a SET variable is that it plays hob with the existing definition about where schemas and tables get a default tablespace from. Which source wins (the database or schema default tablespace, or the SET variable)? And why? The only really clean way to have a SET variable for this is to forget about schema- or table-based defaults. Do we want to do that? (Hey, it'd solve the problem with schema tablespaces being droppable, because there wouldn't *be* any such thing as a schema's tablespace anymore. But on the whole this seems like a step backward in usability.) Agreed, a step backwards, but see below. What we might want to do is invent a --notablespace option for pg_dump, comparable to --noowner, to let someone make a dump that contains no TABLESPACE clauses. Yea, that would work, but we went through so much work to allow SQL standard DDL statements, and it seems a shame to break it just for tablespaces. And, having it be a separate SET would also allow the tablespace creation to fail and still get the objects created. (If the explicit_tablespace doesn't exist during CREATE, we throw a warning. This would contrast with a create _failure_ when the tablespace doesn't exist and you say 'TABLESPACE t1' in CREATE.) So there were actually two uses for this, one for standards compliance, and the other was for flexibility in restoring to a system where the tablespaces can't be created. The SET could give us different behavior (warning vs. error) which would be useful for pg_dump. Could we call it explicit_tablespace and when it is , it is the default, but when it isn't it is just like using 'TABLESPACE t1' in the CREATE, but throws a warning instead of an error if the tablespace doesn't exist? My assumption is that it would not be like the default_with_oids variable usage by pg_dump because it would be reset to '' (default) by pg_dump after each time it is used. I assume explicit_tablespace would always override the schema or database tablespace because it is explicit. In fact this would partially fix the TODO we have: * Allow database recovery where tablespaces can't be created When a pg_dump is restored, all tablespaces will attempt to be created in their original locations. If this fails, the user must be able to adjust the restore process. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] V3 protocol gets out of sync on messages that cause allocation failures
(Tom: this is not as severe a problem as I first thought)
If a client sends a V3 message that is sufficiently large to cause a
memory allocation failure on the backend when allocating space to read
the message, the backend gets out of sync with the protocol stream.
For example, sending this:
FE= Parse(stmt=null,query=SELECT $1,oids={17})
FE= Bind(stmt=null,portal=null,$1=stream of 10 bytes)
provokes this:
ERROR: out of memory
DETAIL: Failed on request of size 1073741823.
FATAL: invalid frontend message type 0
What appears to be happening is that the backend goes into error
recovery as soon as the allocation fails (just after reading the message
length), and never does the read() of the body of the Bind message. So
it falls out of sync, and tries to interpret the guts of the Bind as a
new message. Bad server, no biscuit.
I was concerned that this was exploitable in applications that pass
hostile binary parameters as protocol-level parameters, but it doesn't
seem possible as the bytes at the start of a Bind are not under the
control of the attacker and don't form a valid message.
The CopyData message could probably be exploited, but it seems unlikely
that (security-conscious) applications will pass hostile data directly
in a CopyData message.
I haven't looked at a fix to this in detail (I'm not really familiar
with the backend's error-recovery path), but it seems like one easy
option is to treate all errors that occur while a message is in the
process of being read as FATAL?
-O
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Open Items
Gavin Sherry wrote: On Sun, 17 Oct 2004, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: o remove non-portable TABLESPACE clause from CREATE TABLE and use a new default_tablespace SET variable I'm coming around to the conclusion that this is simply a bad idea. I feel the same way for more or less the reasons you outline. What we might want to do is invent a --notablespace option for pg_dump, comparable to --noowner, to let someone make a dump that contains no TABLESPACE clauses. That's a useful feature but I'm not sure it solves the problem people originally put (to me at least). User has data in a tablespace on a seperate device. The device crashes fatally and the user needs to restore the database. All the user's dumps contain tablespace clauses because the user did not anticipate the device dying. This, I think, is why people wanted to either ignore tablespace clauses, have an override or something else. See my posting about using 'explicit_tablespace' and having it issue a warning if it doesn't exist during create. I still think, however, that a workable solution is to bring up a new system, create the tablespaces on some online partition, and pg_restore the dump. pg_dump does not dump CREATE TABLESPACE so we wont encounter problems there. pg_dumpall does dump CREATE TABLESPACE. It depends if he is using pg_dump on individual databases or pg_dumpall. If he is using pg_dump he is also pg_dumpall --globals-only and the global restore is where he can made the adjustments. Of course, the 'explicit_tablespace' works with pg_dumpall too because everything destined for a non-existant tablespace goes into the default for that object. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Getting rid of AtEOXact Buffers (was Re: [Testperf-general] Re: [PERFORM] First set of OSDL Shared Memscalability results, some wierdness ...)
Seeing as I've missed the last N messages... I'll just reply to this one, rather than each of them in turn... Tom Lane [EMAIL PROTECTED] wrote on 16.10.2004, 18:54:17: I wrote: Josh Berkus writes: First off, two test runs with OProfile are available at: http://khack.osdl.org/stp/298124/ http://khack.osdl.org/stp/298121/ Hmm. The stuff above 1% in the first of these is Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 10 samples %app name symbol name ... 9203692.1332 postgres AtEOXact_Buffers ... In the second test AtEOXact_Buffers is much lower (down around 0.57 percent) but the other suspects are similar. Since the only difference in parameters is shared_buffers (36000 vs 9000), it does look like we are approaching the point where AtEOXact_Buffers is a problem, but so far it's only a 2% drag. Yes... as soon as you first mentioned AtEOXact_Buffers, I realised I'd seen it near the top of the oprofile results on previous tests. Although you don't say this, I presume you're acting on the thought that a 2% drag would soon become a much larger contention point with more users and/or smaller transactions - since these things are highly non-linear. It occurs to me that given the 8.0 resource manager mechanism, we could in fact dispense with AtEOXact_Buffers, or perhaps better turn it into a no-op unless #ifdef USE_ASSERT_CHECKING. We'd just get rid of the special case for transaction termination in resowner.c and let the resource owner be responsible for releasing locked buffers always. The OSDL results suggest that this won't matter much at the level of 1 or so shared buffers, but for 10 or more buffers the linear scan in AtEOXact_Buffers is going to become a problem. If the resource owner is always responsible for releasing locked buffers, who releases the locks if the backend crashes? Do we need some additional code in bgwriter (or?) to clean up buffer locks? We could also get rid of the linear search in UnlockBuffers(). The only thing it's for anymore is to release a BM_PIN_COUNT_WAITER flag, and since a backend could not be doing more than one of those at a time, we don't really need an array of flags for that, only a single variable. This does not show in the OSDL results, which I presume means that their test case is not exercising transaction aborts; but I think we need to zap both routines to make the world safe for large shared_buffers values. (See also http://archives.postgresql.org/pgsql-performance/2004-10/msg00218.php) Yes, that's important. Any objection to doing this for 8.0? As you say, if these issues are definitely kicking in at 10 shared_buffers - there's a good few people out there with 800Mb shared_buffers already. Could I also suggest that we adopt your earlier suggestion of raising the bgwriter parameters as a permanent measure - i.e. changing the defaults in postgresql.conf. That way, StrategyDirtyBufferList won't immediately show itself as a problem when using the default parameter set. It would be a shame to remove one obstacle only to leave another one following so close behind. [...and that also argues against an earlier thought to introduce more fine grained values for the bgwriter's parameters, ISTM] Also, what will Vacuum delay do to the O(N) effect of FlushRelationBuffers when called by VACUUM? Will the locks be held for longer? I think we should do some tests while running a VACUUM in the background also, which isn't part of the DBT-2 set-up, but perhaps we might argue *it should be for the PostgreSQL version*? Dare we hope for a scalability increase in 8.0 after all Best Regards, Simon Riggs ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Re: [HACKERS] Getting rid of AtEOXact Buffers (was Re: [Testperf-general] Re: [PERFORM] First set of OSDL Shared Memscalability results, some wierdness ...)
[EMAIL PROTECTED] writes: If the resource owner is always responsible for releasing locked buffers, who releases the locks if the backend crashes? The ensuing system reset takes care of that. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
