Re: [HACKERS] 7.2.3 vacuum bug
Is this a TODO?
---
Tom Lane wrote:
Rod Taylor [EMAIL PROTECTED] writes:
ERROR: RelationClearRelation: relation 11584078 deleted while still in
use
I've been unable to come up with a test case that will cause the
problem, seems to be timing related. The queries that are currently
running when these errors occur do a lot or work with temp tables that
are frequently truncated.
Hm. vacuum.c tries to avoid this class of problem:
/*
* Race condition -- if the pg_class tuple has gone away since the
* last time we saw it, we don't need to vacuum it.
*/
if (!SearchSysCacheExists(RELOID,
ObjectIdGetDatum(relid),
0, 0, 0))
{
CommitTransactionCommand(true);
return true;/* okay 'cause no data there */
}
...
onerel = relation_open(relid, lmode);
but on reflection it's clear that this doesn't really prevent a race
condition. If the table is already exclusive-locked by a DROP TABLE
that hasn't committed yet (eg, the implicit DROP that happens when temp
tables are cleared out at backend exit), then the syscache lookup will
go fine, but the relation_open() routine blocks waiting for lock and
eventually fails.
What would probably work better is to first lock the relation OID,
then see if we can open the relation or not.
Thinking further, it's really kinda bogus that LockRelation() works on
an already-opened Relation; if possible we should acquire the lock
before attempting to create a relcache entry. (We only need to know the
OID and the relisshared status before we can make a locktag, so it'd be
possible to acquire the lock using only the contents of the pg_class row.)
Not sure how much code restructuring might be involved to make this
happen, but it'd be worth thinking about for 7.4.
regards, tom lane
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
--
Bruce Momjian| http://candle.pha.pa.us
[EMAIL PROTECTED] | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3 vacuum bug
Added to TODO. --- Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Is this a TODO? Yes. Maybe * Acquire lock on a relation before building a relcache entry for it I'm not quite sure yet how this should interact with the case where you already have a relcache entry, but certainly the existing behavior of build the whole entry and then acquire lock is not good. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3 vacuum bug
On Thu, 31 Oct 2002, Tom Lane wrote: Neil Conway [EMAIL PROTECTED] writes: Ok, fair enough -- I agree that we should treat the two cases differently. But one thing I think we should do in any case is improve the wording of the error message. Got a suggestion? Change: RelationClearRelation: relation 25172 deleted while still in use to: RelationClearRelation: a relation (id: 25172) was deleted while still in use ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3 vacuum bug
Found another: ERROR: cannot find attribute 2 of relation pg_temp_12100_0 On Thu, 2002-10-31 at 11:33, scott.marlowe wrote: On Thu, 31 Oct 2002, Tom Lane wrote: Neil Conway [EMAIL PROTECTED] writes: Ok, fair enough -- I agree that we should treat the two cases differently. But one thing I think we should do in any case is improve the wording of the error message. Got a suggestion? Change: RelationClearRelation: relation 25172 deleted while still in use to: RelationClearRelation: a relation (id: 25172) was deleted while still in use -- Rod Taylor ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3 vacuum bug
Rod Taylor [EMAIL PROTECTED] writes: Found another: ERROR: cannot find attribute 2 of relation pg_temp_12100_0 Can you reproduce that? It could be that this just represents someone's temp table deletion committing while VACUUM is partway through trying to build a relcache entry to open the relation. If so, it is only another manifestation of the should-lock-before-relation-open problem. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3 vacuum bug
On Thu, 2002-10-31 at 13:03, Tom Lane wrote: Rod Taylor [EMAIL PROTECTED] writes: Found another: ERROR: cannot find attribute 2 of relation pg_temp_12100_0 Can you reproduce that? It could be that this just represents someone's temp table deletion committing while VACUUM is partway through trying to build a relcache entry to open the relation. If so, it is only another manifestation of the should-lock-before-relation-open problem. Yes, but not easily (very timing dependent), takes a lot of worker processes to throw it. So it's likely a part of the locking issue. -- Rod Taylor ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] 7.2.3 vacuum bug
ERROR: RelationClearRelation: relation 11584078 deleted while still in use I've been unable to come up with a test case that will cause the problem, seems to be timing related. The queries that are currently running when these errors occur do a lot or work with temp tables that are frequently truncated. -- Rod Taylor ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3 vacuum bug
Rod Taylor [EMAIL PROTECTED] writes: ERROR: RelationClearRelation: relation 11584078 deleted while still in use I was going to report a similar error that arises in a different situation: client 1: CREATE TABLE a (b int); BEGIN; DROP TABLE a; -- wait client 2: SELECT * FROM a; client 1: COMMIT; Now, client 2 will receive RelationClearRelation: relation 25172 deleted while still in use, rather than Relation a does not exist, as you might expect. Not sure if it's the same bug, or just a different problem... Cheers, Neil -- Neil Conway [EMAIL PROTECTED] || PGP Key ID: DB3C29FC ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3 vacuum bug
On Wed, 2002-10-30 at 15:38, Neil Conway wrote: Rod Taylor [EMAIL PROTECTED] writes: ERROR: RelationClearRelation: relation 11584078 deleted while still in use I was going to report a similar error that arises in a different situation: Probably a different look at the same problem. -- Rod Taylor ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3 vacuum bug
Neil Conway [EMAIL PROTECTED] writes: client 1: CREATE TABLE a (b int); BEGIN; DROP TABLE a; -- wait client 2: SELECT * FROM a; client 1: COMMIT; Now, client 2 will receive RelationClearRelation: relation 25172 deleted while still in use, rather than Relation a does not exist, as you might expect. But relation a *does* exist at the start of client 2's operation. While I'm not here to defend the exact phrasing of this error message, it does seem to me that it's appropriate to give a different error message than what appears when the table wasn't found at all. An example of why the two cases shouldn't be folded together: suppose that client 2's schema search path is myschema, public, and that client 1 creates/drops myschema.a while there is also a public.a. client 2 will locate myschema.a as the meaning of a, and one way or another it is going to error out when myschema.a gets dropped from underneath it --- it will not (and shouldn't IMHO) go back and repeat the schema search to find public.a. But a user who gets a Relation a does not exist error message in such a scenario would be justifiably confused. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.2.3 vacuum bug
Tom Lane [EMAIL PROTECTED] writes: But relation a *does* exist at the start of client 2's operation. While I'm not here to defend the exact phrasing of this error message, it does seem to me that it's appropriate to give a different error message than what appears when the table wasn't found at all. Ok, fair enough -- I agree that we should treat the two cases differently. But one thing I think we should do in any case is improve the wording of the error message. Cheers, Neil -- Neil Conway [EMAIL PROTECTED] || PGP Key ID: DB3C29FC ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3 vacuum bug
Rod Taylor [EMAIL PROTECTED] writes:
ERROR: RelationClearRelation: relation 11584078 deleted while still in
use
I've been unable to come up with a test case that will cause the
problem, seems to be timing related. The queries that are currently
running when these errors occur do a lot or work with temp tables that
are frequently truncated.
Hm. vacuum.c tries to avoid this class of problem:
/*
* Race condition -- if the pg_class tuple has gone away since the
* last time we saw it, we don't need to vacuum it.
*/
if (!SearchSysCacheExists(RELOID,
ObjectIdGetDatum(relid),
0, 0, 0))
{
CommitTransactionCommand(true);
return true;/* okay 'cause no data there */
}
...
onerel = relation_open(relid, lmode);
but on reflection it's clear that this doesn't really prevent a race
condition. If the table is already exclusive-locked by a DROP TABLE
that hasn't committed yet (eg, the implicit DROP that happens when temp
tables are cleared out at backend exit), then the syscache lookup will
go fine, but the relation_open() routine blocks waiting for lock and
eventually fails.
What would probably work better is to first lock the relation OID,
then see if we can open the relation or not.
Thinking further, it's really kinda bogus that LockRelation() works on
an already-opened Relation; if possible we should acquire the lock
before attempting to create a relcache entry. (We only need to know the
OID and the relisshared status before we can make a locktag, so it'd be
possible to acquire the lock using only the contents of the pg_class row.)
Not sure how much code restructuring might be involved to make this
happen, but it'd be worth thinking about for 7.4.
regards, tom lane
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] 7.2.3 vacuum bug
Neil Conway [EMAIL PROTECTED] writes: Ok, fair enough -- I agree that we should treat the two cases differently. But one thing I think we should do in any case is improve the wording of the error message. Got a suggestion? regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.2.3 patching done
On Mon, Oct 07, 2002 at 09:22:38PM +0200, Peter Eisentraut wrote: Tom Lane writes: But the source distribution hasn't *got* any binary files. There are some under doc/src/graphics, and then there are doc/postgres.tar.gz and doc/man.tar.gz. And what about publishing xdelta patches? -- Alvaro Herrera (alvherre[a]dcc.uchile.cl) Escucha y olvidarás; ve y recordarás; haz y entenderás (Confucio) ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.2.3 patching done
Hello! BTW, is it possible to have just patch against previous version (to reduce traffic and CPU)? I.e. something like 7.2.2-7.2.3.diff.gz? -- WBR, Yury Bokhoncovich, Senior System Administrator, NOC of F1 Group. Phone: +7 (3832) 106228, ext.140, E-mail: [EMAIL PROTECTED] Unix is like a wigwam -- no Gates, no Windows, and an Apache inside. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3 patching done
Yury Bokhoncovich wrote: Hello! BTW, is it possible to have just patch against previous version (to reduce traffic and CPU)? I.e. something like 7.2.2-7.2.3.diff.gz? In some releases, it is possible, in others we add/remove files and it isn't possible. I think because it isn't always possible we normally don't do it. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3 patching done
Hello! On Tue, 1 Oct 2002, Bruce Momjian wrote: In some releases, it is possible, in others we add/remove files and it isn't possible. I think because it isn't always possible we normally don't do it. I think it's enough to do diffs for minor release (i.e. 7.2.2-7.2.3, 7.3.0-7.3.1 and so on). BTW, I had no problems with patching Linux kernel this way (e.g. having vanilla 2.2.16 then sequentially patch for 2.2.17, .18, .19, .20, .21, .22) though there were added directories. -- WBR, Yury Bokhoncovich, Senior System Administrator, NOC of F1 Group. Phone: +7 (3832) 106228, ext.140, E-mail: [EMAIL PROTECTED] Unix is like a wigwam -- no Gates, no Windows, and an Apache inside. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3 patching done
Bruce Momjian writes: BTW, is it possible to have just patch against previous version (to reduce traffic and CPU)? I.e. something like 7.2.2-7.2.3.diff.gz? In some releases, it is possible, in others we add/remove files and it isn't possible. I think because it isn't always possible we normally don't do it. Adding or removing files isn't the problem (see -N option). Binary files are the problem. Using xdelta would be safe, though. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3 patching done
Peter Eisentraut wrote: Bruce Momjian writes: BTW, is it possible to have just patch against previous version (to reduce traffic and CPU)? I.e. something like 7.2.2-7.2.3.diff.gz? In some releases, it is possible, in others we add/remove files and it isn't possible. I think because it isn't always possible we normally don't do it. Adding or removing files isn't the problem (see -N option). Binary files are the problem. Do we change any binary files in minor releases, or even major ones? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3 patching done
Bruce Momjian [EMAIL PROTECTED] writes: Peter Eisentraut wrote: Adding or removing files isn't the problem (see -N option). Binary files are the problem. Do we change any binary files in minor releases, or even major ones? But the source distribution hasn't *got* any binary files. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3 patching done
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Peter Eisentraut wrote: Adding or removing files isn't the problem (see -N option). Binary files are the problem. Do we change any binary files in minor releases, or even major ones? But the source distribution hasn't *got* any binary files. Yea, that was sort of my point. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3?
Should an advisory be issued for production sites to not perform a vacuum full with a notice that a bug fix will be coming shortly? Greg On Sat, 2002-09-28 at 13:45, Justin Clift wrote: Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. This will allow production sites to run the 7.2 series and also do VACUUM FULL won't it? If so, then the idea is already pretty good. :-) Which other fixes would be included? Regards and best wishes, Justin Clift Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: [HACKERS] 7.2.3?
Greg Copeland [EMAIL PROTECTED] writes: Should an advisory be issued for production sites to not perform a vacuum full with a notice that a bug fix will be coming shortly? People seem to be misunderstanding the bug. Whether your vacuum is FULL or not (or VERBOSE or not, or ANALYZE or not) is not relevant. The dangerous thing is to execute a VACUUM that's not a single-table VACUUM *as a non-superuser*. The options don't matter. If you see any notices about skipping tables out of VACUUM, then you are at risk. I'm not averse to issuing an announcement, but let's be sure we have the details straight. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] 7.2.3 patching done
I'm done making back-patches for 7.2.3. Over to you, Bruce ... attached is the REL7_2_STABLE branch history since 7.2.2. regards, tom lane 2002-09-30 16:57 tgl * src/backend/utils/adt/: date.c, datetime.c (REL7_2_STABLE): Back-patch fixes to work around broken mktime() in recent glibc releases. 2002-09-30 16:47 tgl * src/backend/: commands/async.c, tcop/postgres.c (REL7_2_STABLE): Back-patch fix for bad SIGUSR2 interrupt handling during backend shutdown. 2002-09-30 16:24 tgl * src/: backend/storage/lmgr/s_lock.c, include/storage/s_lock.h (REL7_2_STABLE): Back-patch fix for correct TAS operation on multi-CPU PPC machines. 2002-09-30 16:18 tgl * src/backend/: bootstrap/bootstrap.c, storage/buffer/buf_init.c, storage/lmgr/lwlock.c, storage/lmgr/proc.c (REL7_2_STABLE): Back-patch fix for 'can't wait without a PROC structure' failures: remove separate ShutdownBufferPoolAccess exit callback, and do the work in ProcKill instead, before we delete MyProc. 2002-09-30 15:55 tgl * src/: backend/access/transam/clog.c, backend/access/transam/xlog.c, backend/bootstrap/bootstrap.c, backend/tcop/utility.c, include/access/xlog.h (REL7_2_STABLE): Back-patch fix to ensure a checkpoint occurs before truncating CLOG, even if no recent WAL activity has occurred. 2002-09-30 15:45 tgl * src/backend/commands/vacuum.c (REL7_2_STABLE): Back-patch fix to not change pg_database.datvacuumxid or truncate CLOG when an unprivileged user runs VACUUM. 2002-09-20 17:37 tgl * src/backend/utils/adt/ruleutils.c (REL7_2_STABLE): Back-patch fix for failure to dump views containing FULL JOIN USING. The bug is not present in CVS tip due to cleanup of JOIN handling, but 7.2.* is broken. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] 7.2.3 patching done
OK, 7.2.3 is all branded and ready to go. HISTORY/release.sgml shows: --- Release Notes Release 7.2.3 Release date: 2002-10-01 This has a variety of fixes from 7.2.2, including fixes to prevent possible data loss. -- Migration to version 7.2.3 A dump/restore is *not* required for those running 7.2.X. -- Changes Prevent possible compressed transaction log loss (Tom) Prevent non-superuser from increasing most recent vacuum info (Tom) Handle pre-1970 date values in newer versions of glibc (Tom) Fix possible hang during server shutdown Prevent spinlock hangs on SMP PPC machines (Tomoyuki Niijima) Fix pg_dump to properly dump FULL JOIN USING (Tom) --- Tom Lane wrote: I'm done making back-patches for 7.2.3. Over to you, Bruce ... attached is the REL7_2_STABLE branch history since 7.2.2. regards, tom lane 2002-09-30 16:57 tgl * src/backend/utils/adt/: date.c, datetime.c (REL7_2_STABLE): Back-patch fixes to work around broken mktime() in recent glibc releases. 2002-09-30 16:47 tgl * src/backend/: commands/async.c, tcop/postgres.c (REL7_2_STABLE): Back-patch fix for bad SIGUSR2 interrupt handling during backend shutdown. 2002-09-30 16:24 tgl * src/: backend/storage/lmgr/s_lock.c, include/storage/s_lock.h (REL7_2_STABLE): Back-patch fix for correct TAS operation on multi-CPU PPC machines. 2002-09-30 16:18 tgl * src/backend/: bootstrap/bootstrap.c, storage/buffer/buf_init.c, storage/lmgr/lwlock.c, storage/lmgr/proc.c (REL7_2_STABLE): Back-patch fix for 'can't wait without a PROC structure' failures: remove separate ShutdownBufferPoolAccess exit callback, and do the work in ProcKill instead, before we delete MyProc. 2002-09-30 15:55 tgl * src/: backend/access/transam/clog.c, backend/access/transam/xlog.c, backend/bootstrap/bootstrap.c, backend/tcop/utility.c, include/access/xlog.h (REL7_2_STABLE): Back-patch fix to ensure a checkpoint occurs before truncating CLOG, even if no recent WAL activity has occurred. 2002-09-30 15:45 tgl * src/backend/commands/vacuum.c (REL7_2_STABLE): Back-patch fix to not change pg_database.datvacuumxid or truncate CLOG when an unprivileged user runs VACUUM. 2002-09-20 17:37 tgl * src/backend/utils/adt/ruleutils.c (REL7_2_STABLE): Back-patch fix for failure to dump views containing FULL JOIN USING. The bug is not present in CVS tip due to cleanup of JOIN handling, but 7.2.* is broken. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3?
On Sun, 2002-09-29 at 07:19, Lamar Owen wrote: On Saturday 28 September 2002 09:23 pm, Bruce Momjian wrote: Justin Clift wrote: Alvaro Herrera wrote: I agree with Lamar that upgrading is a very difficult process right As a simple for the user approach, would it be too-difficult-to-bother-with to add to the postmaster an ability to start up with the data files from the previous version, for it to recognise an old data format automatically, then for it to do the conversion process of the old data format to the new one before going any further? Sounds like a pain to create initially, but nifty in the end. Yes, we could, but if we are going to do that, we may as well just automate the dump/reload. Automating the dump/reload is fraught with pitfalls. Been there; done that; got the t-shirt. The dump from the old version many times requires hand-editing for cases where the complexity is above a certain threshold. The 7.2-7.3 threshold is just a little lower than normal. Our whole approach to the system catalog is wrong for what Justin (and many others would like to see). With MySQL, for instance, one can migrate on a table-by-table basis from one table type to another. As older table types are continuously supported, one can upgrade each table in turn as you need the featureset supported by that tabletype. The initial Postgres design had a notion of StorageManager's, which should make this very easy indeed, if it had been kept working . IIRC the black box nature of storage manager interface was broken at latest when adding WAL (if it had really been there in the first place). -- Hannu ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3?
On Sun, 2002-09-29 at 09:47, Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: What would that converter need: [snip] I think that should be enough for converting table files. I'd like to experiment with something like this when I have some free time. Maybe next year... It's difficult to say anything convincing on this topic without a specific conversion requirement in mind. Localized conversions like 7.3's tuple header change could be done on a page-by-page basis as you suggest. (In fact, one reason I insisted on putting in a page header version number was to leave the door open for such a converter, if someone wants to do one.) But one likely future format change for user data is combining parent and child tables into a single physical table, per recent inheritance thread. (I'm not yet convinced that that's feasible or desirable, I'm just using it as an example of a possible conversion requirement.) You can't very well do that page-by-page; it'd require a completely different approach. I started to think about possible upgrade strategy for this scenario and came up with a whole new way for the whole storage : We could extend our current way of 1G split files for inheritance, so that each inherited table is in its own (set of) physical files which represent a (set of) 1G segment(s) for the logical file definition of all parent. This would even work for both single and multiple inheritance ! In this case the indexes (which enforce the uniquenaess and are required for RI) would see the thing as a single file and can use plain TIDs. The process of mapping from TID.PAGENR to actual file will happen below the level visible to executor. It would also naturally cluster similar tuples. Aa an extra bonus migration can be done only by changing system catalogs and recreating indexes. It will limit the size of inherited structure to at most 16K different tables (max unsigned int/pagesize), but I don't think this will be a real limit anytime soon. - Hannu ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] 7.2.3?
Hannu Krosing [EMAIL PROTECTED] writes: The initial Postgres design had a notion of StorageManager's, which should make this very easy indeed, if it had been kept working . But the storage manager interface was never built to hide issues like tuple representation --- storage managers just deal in raw pages. I doubt it would have helped in the least for anything we've been concerned about. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.2.3?
On Sun, 2002-09-29 at 19:28, Tom Lane wrote: Hannu Krosing [EMAIL PROTECTED] writes: The initial Postgres design had a notion of StorageManager's, which should make this very easy indeed, if it had been kept working . But the storage manager interface was never built to hide issues like tuple representation --- storage managers just deal in raw pages. I had an impression that SM was meant to be a little higher-level. IIRC the original Berkeley Postgres had at one point a storage manager for write-once storage on CDWr jukeboxes. the README in src/backend/storage/smgr still contains mentions about Sony jukebox drivers. http://www.ndim.edrc.cmu.edu/postgres95/www/pglite1.html also claims this: Version 3 appeared in 1991 and added support for multiple storage managers, an improved query executor and a rewritten rewrite rule system. For the most part, releases since then have focused on portability and reliability. I doubt it would have helped in the least for anything we've been concerned about. Yes, it seems that we do not have a SM in the semse I hoped. Still, if we could use a clean SM interface over old page format, then the tuple conversion could be done there. That of course would need the storage manager to be aware of old/new tuple structures ;( - Hannu ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] 7.2.3?
I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3?
Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. This will allow production sites to run the 7.2 series and also do VACUUM FULL won't it? If so, then the idea is already pretty good. :-) Which other fixes would be included? Regards and best wishes, Justin Clift Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3?
On Saturday 28 September 2002 02:36 pm, Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. IMHO, I believe a 7.2.3 is worthwhile. It isn't _that_ much effort, is it? I am most certainly of the school of thought that backporting serious issues into the last stable release is a Good Thing. I don't think a released 7.3 should prevent us from a 7.2.4 down the road, either -- or even a 7.1.4 if a serious security issue were to be found there. Probably not a 7.0.4, though. And definitely not a 6.5.4. Some people can have great difficulty migrating -- if we're not going to make it easy for people to migrate, we should support older versions with fixes. IMHO, of course. If it hasn't already, a fix for the Red Hat 7.3/glibc mktime(3) issue (workaround really) would be nice, as I understand the 7.3 branch has one. RPM's will take me all of an hour if I'm at work when it's released. That is if my wife doesn't go into labor first (she's at 37 weeks and having Braxton-Hicks already). #4. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3?
Justin Clift dijo: Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. This will allow production sites to run the 7.2 series and also do VACUUM FULL won't it? If so, then the idea is already pretty good. :-) Which other fixes would be included? At least the VACUUM code should prevent VACUUM from running inside a function. At least one user has been bitten by it. Memory leaks and such in the PL modules should be backported also. -- Alvaro Herrera (alvherre[a]atentus.com) El sentido de las cosas no viene de las cosas, sino de las inteligencias que las aplican a sus problemas diarios en busca del progreso. (Ernesto Hernández-Novich) ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] 7.2.3?
Alvaro Herrera [EMAIL PROTECTED] writes: Memory leaks and such in the PL modules should be backported also. This is getting out of hand :-( 7.2 is in maintenance status at this point. I'm willing to do backports for bugs that cause data loss, like this VACUUM/CLOG issue. Performance problems are not on the radar screen at all (especially not when the putative fixes for them haven't received much of any testing, and are barely worthy to be called beta status). We do not have either the developer manpower or the testing resources to do more than the most minimal maintenance on back versions. Major back-port efforts just aren't going to happen. If they did, they would significantly impact our ability to work on 7.3 and up; does that seem like a good tradeoff to you? regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 7.2.3?
Tom Lane dijo: Alvaro Herrera [EMAIL PROTECTED] writes: Memory leaks and such in the PL modules should be backported also. This is getting out of hand :-( Yes, I agree with you. Major back-port efforts just aren't going to happen. If they did, they would significantly impact our ability to work on 7.3 and up; does that seem like a good tradeoff to you? I understand the issue. I also understand that is very nice for PostgreSQL to advance very quickly, and requiring backports (and subsequent slowdown) is not nice at all. However, for users it's very important to have the fixes present in newer versions... _without_ the burden of having to upgrade! I agree with Lamar that upgrading is a very difficult process right now. Requiring huge amounts of disk space and database downtime to do dump/restore is in some cases too high a price to pay. So maybe the upgrading process should be observed instead of wasting time on people trying to stay behind because of the price of that process. Maybe there is some way of making the life easier for the upgrader. Let's see, when you upgrade there are basically two things that change: a) system catalogs Going from one version to another requires a number of changes: new tuples, deleted tuples, new attributes, deleted attributes. On-line transforming syscatalogs for the three first types seems easy. The last one may be difficult, but it also may not be, I'm not sure. It will require a standalone backend for shared relations and such, but hey, it's much cheaper than the process that's required now. b) on-disk representation of user data This is not easy. Upgrading means changing each filenode from one version to another; it requires a tool that understands both (and more than two) versions. It also requires a backend that is able to detect that a page is not the version it should, and either abort or convert it on the fly (this last possibility seems very nice). Note that only tables should be converted: other objects (indexes) should just be rebuilt. There are other things that change. For example, dependencies are new in 7.3; building them without the explicit schema construction seems difficult, but it's certainly possible. The implicit/explicit cast system is also new, but it doesn't depend on user data (except for user defined datatypes, and that should be done manually by the user), so should just be created from scratch. Is this at least remotely possible to do? -- Alvaro Herrera (alvherre[a]atentus.com) La fuerza no está en los medios físicos sino que reside en una voluntad indomable (Gandhi) ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3?
On Sat, 28 Sep 2002, Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. The vacuum thing is big enough that there should be since as always people aren't going to move immediately forward with a major version change. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Upgrade process (was Re: [HACKERS] 7.2.3?)
Alvaro Herrera [EMAIL PROTECTED] writes: Maybe there is some way of making the life easier for the upgrader. Let's see, when you upgrade there are basically two things that change: a) system catalogs b) on-disk representation of user data [much snipped] Yup. I see nothing wrong with the pg_upgrade process that we've previously used for updating the system catalogs, however. Trying to do it internally in some way will be harder and more dangerous (ie, much less reliable) than relying on schema-only dump and restore followed by moving the physical data. Updates that change the on-disk representation of user data are much harder, as you say. But I think they can be made pretty infrequent. We've only had two such updates that I know of in Postgres' history: adding WAL in 7.1 forced some additions to page headers, and now in 7.3 we've changed tuple headers for space-saving reasons, and fixed some problems with alignment in array data. pg_upgrade could have worked for the 7.2 cycle, but it wasn't done, mostly for lack of effort. Going forward I think we should try to maintain compatibility of on-disk user data and ensure that pg_upgrade works. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3?
On Saturday 28 September 2002 04:14 pm, Tom Lane wrote: 7.2 is in maintenance status at this point. I'm willing to do backports for bugs that cause data loss, like this VACUUM/CLOG issue. Performance problems are not on the radar screen at all (especially not when the putative fixes for them haven't received much of any testing, and are barely worthy to be called beta status). A fix that is beta-quality for a non-serious issue (serious issues being of the level of the VACUUM/CLOG issue) is, in my mind at least, not for inclusion into a _stable_ release. Simple fixes (the localtime versus mktime fix) might be doable, but might not depending upon the particular fix, how difficult the packport, etc. But 7.2 is considered _stable_ -- and I agree that this means maintenance mode only. Only the most trivial or the most serious problems should be tackled here. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: Upgrade process (was Re: [HACKERS] 7.2.3?)
On Saturday 28 September 2002 04:57 pm, Tom Lane wrote: 7.3 we've changed tuple headers for space-saving reasons, and fixed some problems with alignment in array data. Going forward I think we should try to maintain compatibility of on-disk user data and ensure that pg_upgrade works. This is of course a two-edged sword. 1.) Keeping pg_upgrade working, which depends upon pg_dump working; 2.) Maintaining security fixes for 7.2 for a good period of time to come, since migration from 7.2 to 7.2 isn't easy. If pg_upgrade is going to be the cookie, then let's all try to test the cookie. I'll certainly try to do my part. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Upgrade process (was Re: [HACKERS] 7.2.3?)
Lamar Owen [EMAIL PROTECTED] writes: This is of course a two-edged sword. 1.) Keeping pg_upgrade working, which depends upon pg_dump working; ... which we have to have anyway, of course ... 2.) Maintaining security fixes for 7.2 for a good period of time to come, since migration from 7.2 to 7.2 isn't easy. True, but I think we'll have to deal with that anyway. Even if the physical database upgrade were trivial, people are going to find application compatibility problems due to schemas and other 7.3 changes. So we're going to have to expend at least some work on fixing critical 7.2.* problems. (I just want to keep a tight rein on how much.) regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3?
Alvaro Herrera wrote: snip I agree with Lamar that upgrading is a very difficult process right now. Requiring huge amounts of disk space and database downtime to do dump/restore is in some cases too high a price to pay. So maybe the upgrading process should be observed instead of wasting time on people trying to stay behind because of the price of that process. As a simple for the user approach, would it be too-difficult-to-bother-with to add to the postmaster an ability to start up with the data files from the previous version, for it to recognise an old data format automatically, then for it to do the conversion process of the old data format to the new one before going any further? Sounds like a pain to create initially, but nifty in the end. :-) Regards and best wishes, Justin Clift snip -- Alvaro Herrera (alvherre[a]atentus.com) La fuerza no está en los medios físicos sino que reside en una voluntad indomable (Gandhi) -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3?
Justin Clift wrote: Alvaro Herrera wrote: snip I agree with Lamar that upgrading is a very difficult process right now. Requiring huge amounts of disk space and database downtime to do dump/restore is in some cases too high a price to pay. So maybe the upgrading process should be observed instead of wasting time on people trying to stay behind because of the price of that process. As a simple for the user approach, would it be too-difficult-to-bother-with to add to the postmaster an ability to start up with the data files from the previous version, for it to recognise an old data format automatically, then for it to do the conversion process of the old data format to the new one before going any further? Sounds like a pain to create initially, but nifty in the end. Yes, we could, but if we are going to do that, we may as well just automate the dump/reload. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 7.2.3?
On Saturday 28 September 2002 09:23 pm, Bruce Momjian wrote: Justin Clift wrote: Alvaro Herrera wrote: I agree with Lamar that upgrading is a very difficult process right As a simple for the user approach, would it be too-difficult-to-bother-with to add to the postmaster an ability to start up with the data files from the previous version, for it to recognise an old data format automatically, then for it to do the conversion process of the old data format to the new one before going any further? Sounds like a pain to create initially, but nifty in the end. Yes, we could, but if we are going to do that, we may as well just automate the dump/reload. Automating the dump/reload is fraught with pitfalls. Been there; done that; got the t-shirt. The dump from the old version many times requires hand-editing for cases where the complexity is above a certain threshold. The 7.2-7.3 threshold is just a little lower than normal. Our whole approach to the system catalog is wrong for what Justin (and many others would like to see). With MySQL, for instance, one can migrate on a table-by-table basis from one table type to another. As older table types are continuously supported, one can upgrade each table in turn as you need the featureset supported by that tabletype. Yes, I know that doesn't fit our existing model of 'all in one' system catalogs. And the solution doesn't present itself readily -- but one day someone will see the way to do this, and it will be good. It _will_ involve refactoring the system catalog schema so that user 'system catalog' metadata and system 'system catalog' data aren't codependent. A more modular data storage approach at a level above the existing broken storage manager modularity will result, and things will be different. However, the number of messages on this subject has increased; one day it will become an important feature worthy of core developer attention. That will be a happy day for me, as well as many others. I have not the time to do it myself; but I can be a gadfly, at least. In the meantime we have pg_upgrade for the future 7.3 - 7.4 upgrade. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.2.3?
Bruce Momjian dijo: Justin Clift wrote: Alvaro Herrera wrote: As a simple for the user approach, would it be too-difficult-to-bother-with to add to the postmaster an ability to start up with the data files from the previous version, for it to recognise an old data format automatically, then for it to do the conversion process of the old data format to the new one before going any further? Yes, we could, but if we are going to do that, we may as well just automate the dump/reload. I don't think that's an acceptable solution. It requires too much free disk space and too much time. On-line upgrading, meaning altering the databases on a table-by-table basis (or even page-by-page) solves both problems (binary conversion sure takes less than converting to text representation and parsing it to binary again). I think a converting postmaster would be a waste, because it's unneeded functionality 99.999% of the time. I'm leaning towards an external program doing the conversion, and the backend just aborting if it finds old or in-conversion data. The converter should be able to detect that it has aborted and resume conversion. What would that converter need: - the old system catalog (including user defined data) - the new system catalog (ditto, including the schema) - the storage manager subsystem I think that should be enough for converting table files. I'd like to experiment with something like this when I have some free time. Maybe next year... -- Alvaro Herrera (alvherre[a]atentus.com) I think my standards have lowered enough that now I think 'good design' is when the page doesn't irritate the living fuck out of me. (JWZ) ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.2.3?
Alvaro Herrera [EMAIL PROTECTED] writes: What would that converter need: [snip] I think that should be enough for converting table files. I'd like to experiment with something like this when I have some free time. Maybe next year... It's difficult to say anything convincing on this topic without a specific conversion requirement in mind. Localized conversions like 7.3's tuple header change could be done on a page-by-page basis as you suggest. (In fact, one reason I insisted on putting in a page header version number was to leave the door open for such a converter, if someone wants to do one.) But one likely future format change for user data is combining parent and child tables into a single physical table, per recent inheritance thread. (I'm not yet convinced that that's feasible or desirable, I'm just using it as an example of a possible conversion requirement.) You can't very well do that page-by-page; it'd require a completely different approach. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
