Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-04 Thread Zdenek Kotala
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 09:51:59PM -0400, Tom Lane wrote: Out-of-line datums aren't the only issue, either: consider inline compressed datums. A data representation change, even one that is known not to increase the ordinary uncompressed size of the datum, could ea

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-04 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 09:51:59PM -0400, Tom Lane wrote: > Out-of-line datums aren't the only issue, either: consider inline > compressed datums. A data representation change, even one that is known > not to increase the ordinary uncompressed size of the datum, could > easily render it slightly l

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-04 Thread Zdenek Kotala
Tom Lane wrote: I do not expect that old code will work with new index structure. I want to keep both implementation and old index will be processed by old code and new one will be processed by new implementation. Each will have different OID and pg_class.relam will point to correct implemen

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > I do have an objection though: TOAST data. Even were the bufmgr to be > provided with a mechanism to find a data type conversion function I > don't see what it would do about a toasted datum. Urgh, that *is* a nasty thought :-( Out-of-line datums aren't

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > Gregory Stark <[EMAIL PROTECTED]> writes: >> "Tom Lane" <[EMAIL PROTECTED]> writes: >>> 2. Page format conversion is WAL-logged as a complete page replacement > >> It seems we get that for free. By definition any modification to a page after >> conversion

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Richard Huxton <[EMAIL PROTECTED]> writes: > One thing no-one's mentioned is how we're going to deal with definitive > incompatibilities. I don't really think that in-place update changes that story at all. The advice has always been "read the release notes and test your applications before updat

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > "Tom Lane" <[EMAIL PROTECTED]> writes: >> 2. Page format conversion is WAL-logged as a complete page replacement > It seems we get that for free. By definition any modification to a page after > conversion will be the first record for that page since a c

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > 2. Page format conversion is WAL-logged as a complete page replacement It seems we get that for free. By definition any modification to a page after conversion will be the first record for that page since a checkpoint. It just means we have to force full_

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Martijn van Oosterhout <[EMAIL PROTECTED]> writes: > On Tue, Jul 03, 2007 at 11:36:03AM -0400, Tom Lane wrote: >> ... (Thought experiment: a page is read in during crash recovery >> or PITR slave operation, and discovered to have the old format.) > Hmm, actually, what's the problem with PITR resto

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Richard Huxton
Tom Lane wrote: Keep in mind that if your proposal involves any serious limitation on the developers' freedom to refactor internal backend APIs or change catalog representations around, it *will be rejected*. Do not have any illusions on that point. It'll be a tough enough sell freezing on-disk

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Tom Lane wrote: Again, you are setting yourself up for complete failure if you insist on having every possible nicety in the first version. An incremental approach is far more likely to succeed than a "big bang". Yes, I know. I don't want to solve everything in one patch. I just looking for

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > My thinking is that when a page in the old format is read in, it's > converted to the new format before doing anything else with it. Yeah, I'm with Heikki on this. What I see as a sane project definition is: * pg_migrator or equivalent to convert

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Zdenek Kotala <[EMAIL PROTECTED]> writes: > It is not downgrade. It is about keep old structure until user says > convert to the new data structure. As Martijn already pointed out, the odds of problems surfacing only after that conversion starts seem high enough to render the whole idea a bit poi

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 07:13:29PM +0200, Zdenek Kotala wrote: pg_migrator is separate tool which requires old postgres version and I would like to have solution in postgres binary without old version presence. Very often new postgres version is store in same locat

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Zdenek Kotala <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Yeah, I'm with Heikki on this. What I see as a sane project definition >> is: >> >> * pg_migrator or equivalent to convert the system catalogs >> * a hook in ReadBuffer to allow a data page conversion procedure to >> be applied, on th

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 07:13:29PM +0200, Zdenek Kotala wrote: > pg_migrator is separate tool which requires old postgres version and I > would like to have solution in postgres binary without old version > presence. Very often new postgres version is store in same location > (e.g. /usr/bin) and

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 11:36:03AM -0400, Tom Lane wrote: > I'm not sure it's feasible to expect that we can change representations > of user-defined types, either. I don't see how you would do that > without catalog access (to look up the UDT), and the page conversion > procedure is going to have

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: My thinking is that when a page in the old format is read in, it's converted to the new format before doing anything else with it. Yeah, I'm with Heikki on this. What I see as a sane project definition is: * pg_migrator or equiva

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: This is actually a bit of a problem. We would need to know when we read in a page what the tupledescriptor for that relation looks like to know which fields are varlena. I'm not sure how easy it wou

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: This is actually a bit of a problem. We would need to know when we read in a page what the tupledescriptor for that relation looks like to know which fields are varlena. I'm not sure how easy it wou

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: > >This is actually a bit of a problem. We would need to know when we > >read in a page what the tupledescriptor for that relation looks like > >to know which fields are varlena. I'm not sure how easy it would be > >to arrange for

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Heikki Linnakangas wrote: Gregory Stark wrote: Conceivably we could grab another infomask bit to indicate "uses new-style varlenas" and then have heaptuple.c understand how to convert them in place. But that leads to a ton of memory management or page locking problems. My thinking is tha

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Gregory Stark wrote: "Heikki Linnakangas" <[EMAIL PROTECTED]> writes: As before, upgrade can be done, it's just a matter of someone scratching the itch. pg_migrator can handle the catalog changes. Doing the page conversion from 8.2 -> 8.3 is possible, and it could be done on-the-fly inside Post

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Heikki Linnakangas wrote: Zdenek Kotala wrote: Project Goals - ... 3) Do not require an old version of PostgreSQL to be installed. Why not? Having two versions installed at the same time doesn't seem like a problem to me. You can remove the old version as soon as the upgrade is done. T

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
"Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > As before, upgrade can be done, it's just a matter of someone scratching the > itch. pg_migrator can handle the catalog changes. Doing the page conversion > from 8.2 -> 8.3 is possible, and it could be done on-the-fly inside PostgreSQL > the firs

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Zdenek Kotala wrote: Project Goals - ... 3) Do not require an old version of PostgreSQL to be installed. Why not? Having two versions installed at the same time doesn't seem like a problem to me. You can remove the old version as soon as the upgrade is done. 4) Allow a fallback to orig

[HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
I attach In-Place upgrade project concept. Any technical details about implementation of each part will be sent later (after concept acceptance). Please, let me know your comments. thanks Zdenek In-place Upgrade project --- Overview PostgreSQL com