Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: > Bruce Momjian wrote: > > Tom Lane wrote: > > > "Joshua D. Drake" writes: > > > > On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: > > > >> I think it will be confusing if we change the name, so I vote to not > > > >> change the name. > > > > > > > Actually, I would vote yes to change the name. > > > > > > I lean that way too. If there were no history involved, we'd certainly > > > prefer pg_upgrade to pg_migrator. > > > > Yeah, that was my feeling too. People like "pg_upgrade", or something > > else? I will add some text like "pg_upgrade (formerly pg_migrator)" in > > the docs. > > OK, seems people like pg_upgrade, but do we call it "pgupgrade" or > "pg_upgrade"? I don't see consistent naming in /contrib: > > pg_buffercache/ > pg_freespacemap/ > pg_standby/ > pg_stat_statements/ > pg_trgm/ > pgbench/ > pgcrypto/ > pgrowlocks/ > pgstattuple/ > > The original 7.2 name was "pg_upgrade": > > http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/contrib/pg_upgrade/Attic/ Added to /contrib as 'pg_upgrade'. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
2010/5/6 Tom Lane : > Bruce Momjian writes: >> OK, seems people like pg_upgrade, but do we call it "pgupgrade" or >> "pg_upgrade"? > pg_upgrade sounds good. I just bet that some users will want it to upgrade their postgresql from 9.0.0 to 9.0.1.. > The latter. The former is unreadable. > > regards, tom lane > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Cédric Villemain -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Thu, May 6, 2010 at 8:10 PM, Thom Brown wrote: > You will call it pg_upgrade. I have spoken. > Thom LOL. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian writes: > OK, seems people like pg_upgrade, but do we call it "pgupgrade" or > "pg_upgrade"? The latter. The former is unreadable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On 6 May 2010 20:55, Bruce Momjian wrote: > Bruce Momjian wrote: > > Tom Lane wrote: > > > "Joshua D. Drake" writes: > > > > On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: > > > >> I think it will be confusing if we change the name, so I vote to not > > > >> change the name. > > > > > > > Actually, I would vote yes to change the name. > > > > > > I lean that way too. If there were no history involved, we'd certainly > > > prefer pg_upgrade to pg_migrator. > > > > Yeah, that was my feeling too. People like "pg_upgrade", or something > > else? I will add some text like "pg_upgrade (formerly pg_migrator)" in > > the docs. > > OK, seems people like pg_upgrade, but do we call it "pgupgrade" or > "pg_upgrade"? I don't see consistent naming in /contrib: > >pg_buffercache/ >pg_freespacemap/ >pg_standby/ >pg_stat_statements/ > pg_trgm/ >pgbench/ >pgcrypto/ >pgrowlocks/ >pgstattuple/ > > The original 7.2 name was "pg_upgrade": > > > http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/contrib/pg_upgrade/Attic/ > > -- > > You will call it pg_upgrade. I have spoken. Thom
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: > Tom Lane wrote: > > "Joshua D. Drake" writes: > > > On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: > > >> I think it will be confusing if we change the name, so I vote to not > > >> change the name. > > > > > Actually, I would vote yes to change the name. > > > > I lean that way too. If there were no history involved, we'd certainly > > prefer pg_upgrade to pg_migrator. > > Yeah, that was my feeling too. People like "pg_upgrade", or something > else? I will add some text like "pg_upgrade (formerly pg_migrator)" in > the docs. OK, seems people like pg_upgrade, but do we call it "pgupgrade" or "pg_upgrade"? I don't see consistent naming in /contrib: pg_buffercache/ pg_freespacemap/ pg_standby/ pg_stat_statements/ pg_trgm/ pgbench/ pgcrypto/ pgrowlocks/ pgstattuple/ The original 7.2 name was "pg_upgrade": http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/contrib/pg_upgrade/Attic/ -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Alvaro Herrera writes: > > Excerpts from Bruce Momjian's message of jue may 06 11:19:27 -0400 2010: > >> It seems copying over pg_statistic would require preservation of > >> pg_class.oid. Right now we only preserve pg_class.relfilenode. > > > That could be fixed easily by creating a throwaway table which included the > > qualified table name instead of the OID, and reloading it into pg_statistic > > after the migration. > > Yeah. Casting to and from regclass would do the trick too. I will add this idea to the pg_migrator TODO file. I didn't bother with this in the 8.3 -> 8.4 migration because we changed the default number of statistics collected in 8.4. This would be a 9.1 item. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Alvaro Herrera writes: > Excerpts from Bruce Momjian's message of jue may 06 11:19:27 -0400 2010: >> It seems copying over pg_statistic would require preservation of >> pg_class.oid. Right now we only preserve pg_class.relfilenode. > That could be fixed easily by creating a throwaway table which included the > qualified table name instead of the OID, and reloading it into pg_statistic > after the migration. Yeah. Casting to and from regclass would do the trick too. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Excerpts from Bruce Momjian's message of jue may 06 11:19:27 -0400 2010: > It seems copying over pg_statistic would require preservation of > pg_class.oid. Right now we only preserve pg_class.relfilenode. That could be fixed easily by creating a throwaway table which included the qualified table name instead of the OID, and reloading it into pg_statistic after the migration. Perhaps this could be done as a regular task in the old database before bringing the server down. -- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Bruce Momjian writes: > > Jesper Krogh wrote: > >> I should have written: > >> Why isn't statistics copied over or why doesnt pg_migrator run analyze by > >> itself? > > > Yeah, the statistics are part of the system tables, and system tables > > are fully handled by pg_dumpall --schema-only (except for statistics). > > There might be changes in the system table statistics format that would > > break if pg_migrator tried to migrate the statistics. > > Right, it's not really practical for pg_migrator to just copy over the > statistics without any intelligence. We might at some time choose to > teach it which stats could be copied safely, but that hardly seems like > something to do in version 1.0 --- and anyway it could not be a 100% > solution. It seems copying over pg_statistic would require preservation of pg_class.oid. Right now we only preserve pg_class.relfilenode. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Bruce Momjian writes: > > Jesper Krogh wrote: > >> I should have written: > >> Why isn't statistics copied over or why doesnt pg_migrator run analyze by > >> itself? > > > Yeah, the statistics are part of the system tables, and system tables > > are fully handled by pg_dumpall --schema-only (except for statistics). > > There might be changes in the system table statistics format that would > > break if pg_migrator tried to migrate the statistics. > > Right, it's not really practical for pg_migrator to just copy over the > statistics without any intelligence. We might at some time choose to > teach it which stats could be copied safely, but that hardly seems like > something to do in version 1.0 --- and anyway it could not be a 100% > solution. > > > And if pg_migrator ran analyze itself, it would greatly increase its > > great migration times! > > Exactly. It's not a win for pg_migrator to not give you back control of > the database until everything has been ANALYZEd. That's a task that can > likely be done in background. I have to keep those sub-minute migration times for bragging rights. :-) -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian writes: > Jesper Krogh wrote: >> I should have written: >> Why isn't statistics copied over or why doesnt pg_migrator run analyze by >> itself? > Yeah, the statistics are part of the system tables, and system tables > are fully handled by pg_dumpall --schema-only (except for statistics). > There might be changes in the system table statistics format that would > break if pg_migrator tried to migrate the statistics. Right, it's not really practical for pg_migrator to just copy over the statistics without any intelligence. We might at some time choose to teach it which stats could be copied safely, but that hardly seems like something to do in version 1.0 --- and anyway it could not be a 100% solution. > And if pg_migrator ran analyze itself, it would greatly increase its > great migration times! Exactly. It's not a win for pg_migrator to not give you back control of the database until everything has been ANALYZEd. That's a task that can likely be done in background. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: > > The database (of a reasonable size) is useless until statistics is > > available. > > > > I guess it is because pg_dump/restore doesn't do it either. > > Yeah, the statistics are part of the system tables, and system tables > are fully handled by pg_dumpall --schema-only (except for statistics). > There might be changes in the system table statistics format that would > break if pg_migrator tried to migrate the statistics. Right now > pg_migrator is immune from any system table changes, and I would like to > keep it that way. > > And if pg_migrator ran analyze itself, it would greatly increase its > great migration times! Forgot the :-). -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Jesper Krogh wrote: > On 2010-05-06 06:41, Alvaro Herrera wrote: > > Excerpts from Jesper Krogh's message of jue may 06 00:32:09 -0400 2010: > > > > > >> Q: I read you pdf, why isn't statistics copied over? It seems to be the > >> last > >> part missing from doing an upgrade in a few minutes. > >> > > Seems fraught with peril, and a bit pointless. What's so bad about having > > to > > run ANALYZE afterwards? > > > > There is nothing directly "bad" about it.. but: > > It's just "an extra step, that might be overseen and is absolutely > required". > > I should have written: > Why isn't statistics copied over or why doesnt pg_migrator run analyze by > itself? > > The database (of a reasonable size) is useless until statistics is > available. > > I guess it is because pg_dump/restore doesn't do it either. Yeah, the statistics are part of the system tables, and system tables are fully handled by pg_dumpall --schema-only (except for statistics). There might be changes in the system table statistics format that would break if pg_migrator tried to migrate the statistics. Right now pg_migrator is immune from any system table changes, and I would like to keep it that way. And if pg_migrator ran analyze itself, it would greatly increase its great migration times! -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Jesper Krogh writes: > I did go the painful way (dump+restore) at that point in time. > It was an 8.1 - 8.3 migration. Since then data has grown and the dump > restore is even less favorable on the 8.3 -> 9.0 migration. > > So in general the pg_migrator way seems to be the only way to aviod > the slony way which is orders of magnitude more complicated. Right in the middle there's the Londiste way. It works like Slony but the setup is much easier. Regards, -- Dimitri Fontaine PostgreSQL DBA, Architecte -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On 2010-05-06 06:41, Alvaro Herrera wrote: Excerpts from Jesper Krogh's message of jue may 06 00:32:09 -0400 2010: Q: I read you pdf, why isn't statistics copied over? It seems to be the last part missing from doing an upgrade in a few minutes. Seems fraught with peril, and a bit pointless. What's so bad about having to run ANALYZE afterwards? There is nothing directly "bad" about it.. but: It's just "an extra step, that might be overseen and is absolutely required". I should have written: Why isn't statistics copied over or why doesnt pg_migrator run analyze by itself? The database (of a reasonable size) is useless until statistics is available. I guess it is because pg_dump/restore doesn't do it either. Jesper -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Excerpts from Jesper Krogh's message of jue may 06 00:32:09 -0400 2010: > Q: I read you pdf, why isn't statistics copied over? It seems to be the last > part missing from doing an upgrade in a few minutes. Seems fraught with peril, and a bit pointless. What's so bad about having to run ANALYZE afterwards? -- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On 2010-05-06 01:45, Bruce Momjian wrote: Jesper Krogh wrote: On 2010-05-03 23:09, Bruce Momjian wrote: Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) While I can't improve on Robert's reply, I can supply a PDF about how pg_migrator works: http://momjian.us/main/presentations/technical.html#pg_migrator There is a huge amount of users to whom pg_migrator is "at least" a big a feature as HS+SR is. Last dump/restore was a 24 hours process in one of our installations. I think it was due to in-efficiency in handling BYTEA types in the process (but not sure). But I'm one of the few guys who seem to have an infinite amount of time for reading on mailing lists, but without my knowledge from reading this list I would never have run pg_migrator on my production data if I had to pick it from pg_foundry. So, did you use "copy" or "link" mode, and how fast was the pg_migrator upgrade? I did go the painful way (dump+restore) at that point in time. It was an 8.1 - 8.3 migration. Since then data has grown and the dump restore is even less favorable on the 8.3 -> 9.0 migration. So in general the pg_migrator way seems to be the only way to aviod the slony way which is orders of magnitude more complicated. Q: I read you pdf, why isn't statistics copied over? It seems to be the last part missing from doing an upgrade in a few minutes. Jesper -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > "Joshua D. Drake" writes: > > On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: > >> I think it will be confusing if we change the name, so I vote to not > >> change the name. > > > Actually, I would vote yes to change the name. > > I lean that way too. If there were no history involved, we'd certainly > prefer pg_upgrade to pg_migrator. Yeah, that was my feeling too. People like "pg_upgrade", or something else? I will add some text like "pg_upgrade (formerly pg_migrator)" in the docs. I will also add something about the fact that there is no guarantee that pg_upgrade will work with all future major Postgres releases, per Tom's concern. FYI, I specifically labeled backend changes as "binary upgrade" because I wanted to make sure those changes were useful for other binary upgrade tools, in case someone wanted to create another one. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
"Joshua D. Drake" writes: > On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: >> I think it will be confusing if we change the name, so I vote to not >> change the name. > Actually, I would vote yes to change the name. I lean that way too. If there were no history involved, we'd certainly prefer pg_upgrade to pg_migrator. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Wed, 2010-05-05 at 20:24 -0400, Robert Haas wrote: > On Wed, May 5, 2010 at 7:44 PM, Bruce Momjian wrote: > > Alvaro Herrera wrote: > >> > >> So what was the conclusion here? Is pg_migrator going to be in contrib > >> for beta2 or 3, after cleaning it up? > > > > Thanks for asking. :-) I can add pg_migrator to contrib by the end of > > next week, so it will be in beta2. I will remove 8.4 as a migration > > target, which will allow the removal of some C code and documentation > > warnings. Unless I hear otherwise, I will start on it in the next few > > days. Total work will be < 8 hours, including testing. > > > > One outstanding question is whether we want to rename pg_migrator to > > something clearer, like pg_upgrade or pg_binary_upgrade. (pg_upgrade > > was the original name for this migration method in the 1998.) I am > > slightly concerned that the "migration" word is too associated with > > cross-database-product migration. (There are no mentions of > > "pg_migrator" in our CVS now, except for an 8.4 release note item > > mention when pg_dump --binary-upgrade was added.) > > I think it will be confusing if we change the name, so I vote to not > change the name. Actually, I would vote yes to change the name. Once its in contrib, we likely never will and this isn't really a migration tool. It is an upgrade tool. Joshua D. Drake > > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise Postgres Company > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Wed, May 5, 2010 at 7:44 PM, Bruce Momjian wrote: > Alvaro Herrera wrote: >> >> So what was the conclusion here? Is pg_migrator going to be in contrib >> for beta2 or 3, after cleaning it up? > > Thanks for asking. :-) I can add pg_migrator to contrib by the end of > next week, so it will be in beta2. I will remove 8.4 as a migration > target, which will allow the removal of some C code and documentation > warnings. Unless I hear otherwise, I will start on it in the next few > days. Total work will be < 8 hours, including testing. > > One outstanding question is whether we want to rename pg_migrator to > something clearer, like pg_upgrade or pg_binary_upgrade. (pg_upgrade > was the original name for this migration method in the 1998.) I am > slightly concerned that the "migration" word is too associated with > cross-database-product migration. (There are no mentions of > "pg_migrator" in our CVS now, except for an 8.4 release note item > mention when pg_dump --binary-upgrade was added.) I think it will be confusing if we change the name, so I vote to not change the name. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Jesper Krogh wrote: > On 2010-05-03 23:09, Bruce Momjian wrote: > > Robert Haas wrote: > > > >> On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine > >> wrote: > >> > >>> Now you tell me how awful this idea really is :) > >>> > >> I'm not sure I can count that high. :-) > >> > > While I can't improve on Robert's reply, I can supply a PDF about how > > pg_migrator works: > > > > http://momjian.us/main/presentations/technical.html#pg_migrator > > > > > There is a huge amount of users to whom pg_migrator is "at least" > a big a feature as HS+SR is. > > Last dump/restore was a 24 hours process in one of our installations. > I think it was due to in-efficiency in handling BYTEA types in the > process (but not sure). > > But I'm one of the few guys who seem to have an infinite amount of > time for reading on mailing lists, but without my knowledge from > reading this list I would never have run pg_migrator on my production > data if I had to pick it from pg_foundry. So, did you use "copy" or "link" mode, and how fast was the pg_migrator upgrade? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Alvaro Herrera wrote: > > So what was the conclusion here? Is pg_migrator going to be in contrib > for beta2 or 3, after cleaning it up? Thanks for asking. :-) I can add pg_migrator to contrib by the end of next week, so it will be in beta2. I will remove 8.4 as a migration target, which will allow the removal of some C code and documentation warnings. Unless I hear otherwise, I will start on it in the next few days. Total work will be < 8 hours, including testing. One outstanding question is whether we want to rename pg_migrator to something clearer, like pg_upgrade or pg_binary_upgrade. (pg_upgrade was the original name for this migration method in the 1998.) I am slightly concerned that the "migration" word is too associated with cross-database-product migration. (There are no mentions of "pg_migrator" in our CVS now, except for an 8.4 release note item mention when pg_dump --binary-upgrade was added.) -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On 2010-05-03 23:09, Bruce Momjian wrote: Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) While I can't improve on Robert's reply, I can supply a PDF about how pg_migrator works: http://momjian.us/main/presentations/technical.html#pg_migrator There is a huge amount of users to whom pg_migrator is "at least" a big a feature as HS+SR is. Last dump/restore was a 24 hours process in one of our installations. I think it was due to in-efficiency in handling BYTEA types in the process (but not sure). But I'm one of the few guys who seem to have an infinite amount of time for reading on mailing lists, but without my knowledge from reading this list I would never have run pg_migrator on my production data if I had to pick it from pg_foundry. Just my 0.25€ Jesper -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
So what was the conclusion here? Is pg_migrator going to be in contrib for beta2 or 3, after cleaning it up? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Greg Smith wrote: > Bruce Momjian wrote: > > As a summary, let me list the migrations pg_migrator supports: > > 8.3 -> 8.4 > > 8.4 -> 9.0 > > 8.3 -> 9.0 > > Surprisingly, it is 8.3 -> 8.4 that has the most restrictions because it > > doesn't have access to the features we added in Postgres 9.0. > > Tom is right that the code could be cleaned up if we removed 8.3 -> 8.4, > > but more importantly the documentation would be clearer. > > > > I think it's extremely valuable that either 8.3 or 8.4 can be upgraded > to 9.0. But let's face it: in the long run, the number of people who > are going to use pg_migrator for a 8.3->8.4 migration, but that's > haven't already done so, is a small user base. The feature set > improvement in 8.4 had a lot of great stuff, but few that were > compelling from a "now I can do something completely impossible before!" > standpoint. As was noted recently during the "Native DB replication for > PG" discussion over on pgsql-general last week, there are plenty of > people happily running a stable 8.3 who just ignore 8.4 altogether for > that reason. > > The replication features in 9.0 are compelling in that way though, and I > expect to see plenty of upgrades to that version from both 8.3 and 8.4 > installs. If that works fine right now, I would prefer to see that > documented as a special case two-versions at once situation that people > shouldn't necessarily expect in the future, but certainly valuable to > keep going if the maintenance burden isn't so bad. Until we have some kind of "delete the incompatibile format" step in major releases, my bet is that pg_migrator will support many previous major versions, or not work at all. It is hard to see why it would work for some major versions and not others given our current code. > 2) Deprecate the pg_migrator hosted on pg_foundry as only being > recommended for limited 8.3->8.4 upgrades. Essentially stop active > development on the version there, and focus on the one in contrib/ > instead. People who want an improved 8.3->8.4 tool can always contract > with someone to backport fixes needed for their particular use case. I > think we're past the point where the community at large (meaning: > mainly Bruce right now) should be expected to do that, now that 9.0 is > coming out, so long as 8.3 to 9.0 conversions are available too. I > can't imagine suggesting to anyone that they upgrade in-place from 8.3 > to 8.4 right now. Everybody I talk to who isn't already on 8.4 is > delaying upgrades in anticipation of 9.0 later this year or early next. Assuming pg_migrator moves to /contrib, I don't plan on doing much to improve the pgfoundry version unless people find specific bugs. I have not released a 9.0-compatible version there for that reason. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: > On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine > wrote: > > Now you tell me how awful this idea really is :) > > I'm not sure I can count that high. :-) While I can't improve on Robert's reply, I can supply a PDF about how pg_migrator works: http://momjian.us/main/presentations/technical.html#pg_migrator -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: As a summary, let me list the migrations pg_migrator supports: 8.3 -> 8.4 8.4 -> 9.0 8.3 -> 9.0 Surprisingly, it is 8.3 -> 8.4 that has the most restrictions because it doesn't have access to the features we added in Postgres 9.0. Tom is right that the code could be cleaned up if we removed 8.3 -> 8.4, but more importantly the documentation would be clearer. I think it's extremely valuable that either 8.3 or 8.4 can be upgraded to 9.0. But let's face it: in the long run, the number of people who are going to use pg_migrator for a 8.3->8.4 migration, but that's haven't already done so, is a small user base. The feature set improvement in 8.4 had a lot of great stuff, but few that were compelling from a "now I can do something completely impossible before!" standpoint. As was noted recently during the "Native DB replication for PG" discussion over on pgsql-general last week, there are plenty of people happily running a stable 8.3 who just ignore 8.4 altogether for that reason. The replication features in 9.0 are compelling in that way though, and I expect to see plenty of upgrades to that version from both 8.3 and 8.4 installs. If that works fine right now, I would prefer to see that documented as a special case two-versions at once situation that people shouldn't necessarily expect in the future, but certainly valuable to keep going if the maintenance burden isn't so bad. Balancing out development reality with the ideal situation from the perspective of [potential|current] customers that I deal with every day, what I would prefer to see here is: 1) Commit a streamlined pg_migrator that only handles conversions with 9.0 as a target into contrib, and ship it with 9.0. Like Bruce, I had presumed that the discussion about whether that was going to happen would happen in parallel with beta (read: right now), rather than its already being too late to even consider. I think it's completely bizarre from an advocacy standpoint to even consider that you wouldn't ship such a tool with the core database, now that it's been around for long enough to have a positive track record. 2) Deprecate the pg_migrator hosted on pg_foundry as only being recommended for limited 8.3->8.4 upgrades. Essentially stop active development on the version there, and focus on the one in contrib/ instead. People who want an improved 8.3->8.4 tool can always contract with someone to backport fixes needed for their particular use case. I think we're past the point where the community at large (meaning: mainly Bruce right now) should be expected to do that, now that 9.0 is coming out, so long as 8.3 to 9.0 conversions are available too. I can't imagine suggesting to anyone that they upgrade in-place from 8.3 to 8.4 right now. Everybody I talk to who isn't already on 8.4 is delaying upgrades in anticipation of 9.0 later this year or early next. My main issues with this project continuing to be hosted in pgfoundry are: 1) Perceived lack of confidence and/or legitimacy for it as an in-place upgrade solution, which would be a terrible PR move. When people ask about in-place upgrades and I tell them "there's a tool you can download for that", they look at me in terror and ask if I'm serious that it isn't just included in the core code. The improvement between answering that way and saying "yes, the tool for 8.3 and 8.4 is included with the core distribution", from the perspective of selling people on adopting PostgreSQL, cannot be overstated. 2) Anyone who looks at pgfoundry for more than a few minutes walks away covered with the scent of dead projects. One reason for that is that related to how painful it is to develop there. I don't want to reignite a full anti-pgfoundry discussion here. Suffice it to say that there are many of us who just can't bear working with CVS anymore who have just given up on doing anything useful to projects hosted there. Something that's in core (and therefore included in the git conversion already being published) is much easier to work with and submit patches against. I'm already dumping git clones of the PG repo on every system I do serious work on. If each of those were then capable of generating pg_migrator patches I could submit, I would actually do that each time I use the tool for an upgrade and notice how it could be improved. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Mon, 2010-05-03 at 16:12 -0400, Robert Haas wrote: > On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine > wrote: > > Now you tell me how awful this idea really is :) > > I'm not sure I can count that high. :-) You don't have to... NaN Joshua D. Drake > > ...Robert > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine wrote: > Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Andrew Dunstan writes: > We need to be thinking more now about such a contingency. Postgres use in > very large installations is now at such a level that requiring a > pg_dump/pg_restore is really not an option for many users. If pg_migrator is > not always going to work then we need to be addressing that now, or else it > is at best a stop-gap. ISTM some sort of page layout versioning scheme might > be at least part of what we'll need in the medium term. Would it be on the same complexity level to support recovering WALs from previous version? On the code maintenance alone it sounds bad enough, but apart from that? The idea of course would be then to add an Hot-Standby server running next PostgreSQL version and fed from current production server. The base backup would have to either be processed by pg_migrator or we'd have to open the possibility of starting a slave from a pg_dump, which has been talked about already. The change here would be that this initial step would not be done as part of the maintenance window. Now you tell me how awful this idea really is :) Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Andrew Dunstan wrote: > > > Bruce Momjian wrote: > > For example, I assume there > > will be some major version of Postgres where pg_migrator will not work > > at all. > > > > > > We need to be thinking more now about such a contingency. Postgres use > in very large installations is now at such a level that requiring a > pg_dump/pg_restore is really not an option for many users. If > pg_migrator is not always going to work then we need to be addressing > that now, or else it is at best a stop-gap. ISTM some sort of page > layout versioning scheme might be at least part of what we'll need in > the medium term. Yes, my bet is that when we do need to change the page format, we will either use a conversion-on-read process or some external tool to do the conversion. pg_migrator does have code to do page conversions but it has never been used. If you are using pg_migrator in copy mode, we might be able to do page conversion during the copy, but for link mode, that is not possible. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: For example, I assume there will be some major version of Postgres where pg_migrator will not work at all. We need to be thinking more now about such a contingency. Postgres use in very large installations is now at such a level that requiring a pg_dump/pg_restore is really not an option for many users. If pg_migrator is not always going to work then we need to be addressing that now, or else it is at best a stop-gap. ISTM some sort of page layout versioning scheme might be at least part of what we'll need in the medium term. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: > Well, that is going to make the documentation more complicated than it > already is. Why mention a process in 9.0 that no one needs to use? I > am not writing the docs for some hypothetical release, but for 9.0. > When we have some restriction, we can document that. > > My guess is that 9.2 will be able to upgrade from 8.3, 8.4, 9.0, and > 9.1, but that going to 9.5 will require you to go to 9.2 first, run some > script, then upgrade to 9.5; again all hypothetical. I think we can > require people using pg_migrator to read matching documentation for that > major version of pg_migrator, and pg_migrator will enforce any > restrictions we come up with in the future. For example, I assume there > will be some major version of Postgres where pg_migrator will not work > at all. I do think we are going to have to mention new pg_migrator restrictions in the major release notes for future versions. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Peter Eisentraut wrote: > On l?r, 2010-05-01 at 17:26 -0400, Bruce Momjian wrote: > > I am unclear why it would be in /bin if it requires 15 steps to run > > and is run only once by only some users. It seems natural > > for /contrib, like pgcrypto. > > Well, pg_resetxlog is also rarely run by most users. It started in > contrib but was later moved to bin in order to show that it is fully > supported. Yea, it is like pg_resetxlog. The only counter-argument I can think of is that we decided that everyone should have the pg_resetxlog binary available, and not have to scramble around to install it in case of a problem, but I will admit that is a thin argument. > Also, I think the 15 steps are a bit inflated. Several of those steps > are about building and installing various pieces of software. If you > count that way, using PostgreSQL itself might also require about 12 > steps. In a packaged environment that allows side-by-side installation > of major versions (such as Debian or Windows), you need about 4 or 5 > manual steps, and with a small script layer you need only 1 or 0. Well, that is good news. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Robert Haas writes: > > Yeah. It's not uncommon to want to upgrade by more than one release at > > a time, and I haven't heard any reason why we should arbitrarily > > refuse to support that. Of course we may need to do that eventually > > for some specific reason, but it seems like we should only consider > > imposing such a restriction in the face of a tangible need. > > I wasn't suggesting that we should arbitrarily refuse to support cases > that would work otherwise. What I *was* saying is that the community > has not bought into doing any extra work to support > cross-multiple-version migrations, and I don't think it will do so. > It would be a mistake to give users the impression that such cases can > be expected to work in future. In particular we should not provide a > documentation table that is set up to give the impression that > multi-version upgrades are going to be supported. The docs should be > written to make it clear that one-version-at-a-time is the normal case, > and any cases where you can take a shortcut should be documented as > exceptions. Well, that is going to make the documentation more complicated than it already is. Why mention a process in 9.0 that no one needs to use? I am not writing the docs for some hypothetical release, but for 9.0. When we have some restriction, we can document that. My guess is that 9.2 will be able to upgrade from 8.3, 8.4, 9.0, and 9.1, but that going to 9.5 will require you to go to 9.2 first, run some script, then upgrade to 9.5; again all hypothetical. I think we can require people using pg_migrator to read matching documentation for that major version of pg_migrator, and pg_migrator will enforce any restrictions we come up with in the future. For example, I assume there will be some major version of Postgres where pg_migrator will not work at all. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas writes: > Yeah. It's not uncommon to want to upgrade by more than one release at > a time, and I haven't heard any reason why we should arbitrarily > refuse to support that. Of course we may need to do that eventually > for some specific reason, but it seems like we should only consider > imposing such a restriction in the face of a tangible need. I wasn't suggesting that we should arbitrarily refuse to support cases that would work otherwise. What I *was* saying is that the community has not bought into doing any extra work to support cross-multiple-version migrations, and I don't think it will do so. It would be a mistake to give users the impression that such cases can be expected to work in future. In particular we should not provide a documentation table that is set up to give the impression that multi-version upgrades are going to be supported. The docs should be written to make it clear that one-version-at-a-time is the normal case, and any cases where you can take a shortcut should be documented as exceptions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: > > My guess is that when that happens we would just document/enforce it > > in > > pg_migrator, but I don't see why we would arbitrarily restrict > > pg_migrator at this time. > > Yeah. It's not uncommon to want to upgrade by more than one release at > a time, and I haven't heard any reason why we should arbitrarily > refuse to support that. Of course we may need to do that eventually > for some specific reason, but it seems like we should only consider > imposing such a restriction in the face of a tangible need. Yea, we will need it one day, and if pg_migrator was more tied to system table changes and such, it would be smart to do it now, but if you look at the C code you will see that most of the effort is related to compatibility with the _target_ major version, not the _source_ major version, and by definition, the source major version never changes as we release new major versions. (Remember, pg_dump already does the heavy lifting of moving our database schema to the new major version.) A lot of understanding pg_migrator is understanding the source/target matrix of compatibility --- something we as a community have not thought about very much at this level. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On May 2, 2010, at 12:01 PM, Bruce Momjian wrote: > Tom Lane wrote: >> Bruce Momjian writes: >>> Andrew Dunstan wrote: I thought the idea was just to support migration from version N to version N+1. >> >>> Oh, I will also support many older _source_ versions, like 8.3 and >>> 8.4. >> >> Really? Nobody else has bought into that, and it's not only >> pg_migrator >> that would have to go out of its way to support such cases. You're >> talking about cross-multi-version compatibility of on-disk formats >> too. > > Well, it works. I have a test suite that I run regularly. Because of > the way pg_migrator works it is pretty painless to support multiple > _source_ major versions. > > The binary format issue is relevant, but until we have a way to remove > the old binary format, I don't see much value in supporting just one > source version. For example, we don't have any system now to remove > the > HEAP_MOVED_OFF and HEAP_MOVED_IN heap bits so effectively major > versions > have to support them forever. Now, if we develop a system where a > version would _remove_ the old data format, we would then specify that > pg_migrator can only migrate _from_ one major version, and you would > have to run a script to remove the old data format. For example, > migrating from 9.0 to 9.2 would requiring migrating from 9.0 to 9.1 > with > pg_migrator, updating the data pages to 9.1 format, then using > pg_migrator again to migrate from 9.1 to 9.2, but of course, we are > not > there yet. > > My guess is that when that happens we would just document/enforce it > in > pg_migrator, but I don't see why we would arbitrarily restrict > pg_migrator at this time. Yeah. It's not uncommon to want to upgrade by more than one release at a time, and I haven't heard any reason why we should arbitrarily refuse to support that. Of course we may need to do that eventually for some specific reason, but it seems like we should only consider imposing such a restriction in the face of a tangible need. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Bruce Momjian writes: > > Andrew Dunstan wrote: > >> I thought the idea was just to support migration from version N to > >> version N+1. > > > Oh, I will also support many older _source_ versions, like 8.3 and 8.4. > > Really? Nobody else has bought into that, and it's not only pg_migrator > that would have to go out of its way to support such cases. You're > talking about cross-multi-version compatibility of on-disk formats too. Well, it works. I have a test suite that I run regularly. Because of the way pg_migrator works it is pretty painless to support multiple _source_ major versions. The binary format issue is relevant, but until we have a way to remove the old binary format, I don't see much value in supporting just one source version. For example, we don't have any system now to remove the HEAP_MOVED_OFF and HEAP_MOVED_IN heap bits so effectively major versions have to support them forever. Now, if we develop a system where a version would _remove_ the old data format, we would then specify that pg_migrator can only migrate _from_ one major version, and you would have to run a script to remove the old data format. For example, migrating from 9.0 to 9.2 would requiring migrating from 9.0 to 9.1 with pg_migrator, updating the data pages to 9.1 format, then using pg_migrator again to migrate from 9.1 to 9.2, but of course, we are not there yet. My guess is that when that happens we would just document/enforce it in pg_migrator, but I don't see why we would arbitrarily restrict pg_migrator at this time. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Andrew Dunstan writes: > > Robert Haas wrote: > >> I don't think it's going > >> to be practical to retain all the migration code for every pair of > >> versions forever, > > > I thought the idea was just to support migration from version N to > > version N+1. > > Yeah. I think trying to do more than that is just going to make things > messy. For example, we added features to pg_dump and the core server > since 8.4 to help pg_migrator do its thing. Trying to make the same > pg_migrator code support cases with and without those features available > is going to complicate the code, not to mention the documentation, > enormously. As a summary, let me list the migrations pg_migrator supports: 8.3 -> 8.4 8.4 -> 9.0 8.3 -> 9.0 Surprisingly, it is 8.3 -> 8.4 that has the most restrictions because it doesn't have access to the features we added in Postgres 9.0. Tom is right that the code could be cleaned up if we removed 8.3 -> 8.4, but more importantly the documentation would be clearer. > To the extent that future bug fixes are relevant to multiple versions > of pg_migrator, we could just apply them to multiple branches, same as > we manage such fixes for the core code. I don't see that trying to > have a single version of pg_migrator is going to make things easier > anywhere. Yea, that is probably right. We can enforce in pg_migrator that you can only migrate to the matching major version you are using in /contrib. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: > On Sat, May 1, 2010 at 11:31 PM, Andrew Dunstan wrote: > > Robert Haas wrote: > >> > >> ?I don't think it's going > >> to be practical to retain all the migration code for every pair of > >> versions forever, > > > > I thought the idea was just to support migration from version N to version > > N+1. > > I think that would be "good enough". But right now we can do better. > No reason to rip it out, is there? Again, we are talking about removing _target_ support for 8.4 in pg_migrator 9.0. It is hard to see why someone would use 9.0 /contrib pg_migrator to migrate from 8.3 to 8.4. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian writes: > Andrew Dunstan wrote: >> I thought the idea was just to support migration from version N to >> version N+1. > Oh, I will also support many older _source_ versions, like 8.3 and 8.4. Really? Nobody else has bought into that, and it's not only pg_migrator that would have to go out of its way to support such cases. You're talking about cross-multi-version compatibility of on-disk formats too. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Andrew Dunstan wrote: > > > Robert Haas wrote: > > I don't think it's going > > to be practical to retain all the migration code for every pair of > > versions forever, > > > > I thought the idea was just to support migration from version N to > version N+1. Oh, I will also support many older _source_ versions, like 8.3 and 8.4. The question is whether a pg_migrator in /contrib should support multiple _target_ versions. (Again, I assumed this discussion would be necessary, and is best done during beta.) -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: > On Sat, May 1, 2010 at 5:46 PM, Bruce Momjian wrote: > >> > Agreed, we're not holding up 9.0 for it. ?I think the main bit of work > >> > that would be needed to put it into contrib would be to SGML-ize the > >> > docs. ?Don't know if Bruce has got the time to get that done. > >> > >> Creating the SGML docs is trivial, especially compared to the 9.0 > >> release notes SGML. ?;-) ?It will take only an hour --- I am basically > >> going to merge the README and the INSTALL file, remove mentions about > >> migrating to < 9.0, and add SGML markup. ?I labored on README and the > >> INSTALL files for a long time and can't figure out how to improve them. > > > > Oh, and I will remove the C code that was used to migrate _to_ pre-9.0 > > databases. ?People can use the pgfoundry version for such cases. I have > > specifically not created a pgfoundry release of pg_migrator that > > migrates to 9.0. ?(It worked for the 8.5 numbering.) > > I wonder if this is just going to lead to us maintaining two versions > of pg_migrator, which wouldn't be awesome. I don't think it's going > to be practical to retain all the migration code for every pair of > versions forever, but I'm reluctant to start changing things just > because we're sucking the thing into our main tree. Especially things > that sound suspiciously like features. Tom's idea basically was that each version of pg_migrator would only support it current major version as a _target_. We would have to backpatch fixes to pg_migrator to previous major versions, and to pgfoundry if necessary. However, there isn't much code churn in pg_migrator anymore (as there was months before), so we should be OK. We already do such backpatching for all our other core code. I can easily keep all the code in each version of pg_migrator. However, pg_dump only supports loads into the _target_ major version, just like pg_migrator would do. I am unclear on which direction to take, but both are easy for me. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On lör, 2010-05-01 at 17:26 -0400, Bruce Momjian wrote: > I am unclear why it would be in /bin if it requires 15 steps to run > and is run only once by only some users. It seems natural > for /contrib, like pgcrypto. Well, pg_resetxlog is also rarely run by most users. It started in contrib but was later moved to bin in order to show that it is fully supported. Also, I think the 15 steps are a bit inflated. Several of those steps are about building and installing various pieces of software. If you count that way, using PostgreSQL itself might also require about 12 steps. In a packaged environment that allows side-by-side installation of major versions (such as Debian or Windows), you need about 4 or 5 manual steps, and with a small script layer you need only 1 or 0. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas writes: > On Sat, May 1, 2010 at 11:38 PM, Tom Lane wrote: >> To the extent that future bug fixes are relevant to multiple versions >> of pg_migrator, we could just apply them to multiple branches, same as >> we manage such fixes for the core code. I don't see that trying to >> have a single version of pg_migrator is going to make things easier >> anywhere. > That might be OK if by "multiple versions" we mean "multiple branches > within the repository". But I'm not so sure about having a version in > the core repository and another version on pgfoundry, or something of > that sort. True. If we push it into contrib as of 9.0, I think this won't be too bad, because the pgfoundry version will only be covering 8.3->8.4. Given all the limitations in that version, it's not going to be too interesting and will probably die on the vine pretty quickly. If we hold off merging it till 9.1, though, the pgfoundry version probably will be useful and in need of maintenance for a while to come. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 11:38 PM, Tom Lane wrote: > To the extent that future bug fixes are relevant to multiple versions > of pg_migrator, we could just apply them to multiple branches, same as > we manage such fixes for the core code. I don't see that trying to > have a single version of pg_migrator is going to make things easier > anywhere. That might be OK if by "multiple versions" we mean "multiple branches within the repository". But I'm not so sure about having a version in the core repository and another version on pgfoundry, or something of that sort. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Andrew Dunstan writes: > Robert Haas wrote: >> I don't think it's going >> to be practical to retain all the migration code for every pair of >> versions forever, > I thought the idea was just to support migration from version N to > version N+1. Yeah. I think trying to do more than that is just going to make things messy. For example, we added features to pg_dump and the core server since 8.4 to help pg_migrator do its thing. Trying to make the same pg_migrator code support cases with and without those features available is going to complicate the code, not to mention the documentation, enormously. To the extent that future bug fixes are relevant to multiple versions of pg_migrator, we could just apply them to multiple branches, same as we manage such fixes for the core code. I don't see that trying to have a single version of pg_migrator is going to make things easier anywhere. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 11:31 PM, Andrew Dunstan wrote: > Robert Haas wrote: >> >> I don't think it's going >> to be practical to retain all the migration code for every pair of >> versions forever, > > I thought the idea was just to support migration from version N to version > N+1. I think that would be "good enough". But right now we can do better. No reason to rip it out, is there? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: I don't think it's going to be practical to retain all the migration code for every pair of versions forever, I thought the idea was just to support migration from version N to version N+1. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 5:46 PM, Bruce Momjian wrote: >> > Agreed, we're not holding up 9.0 for it. I think the main bit of work >> > that would be needed to put it into contrib would be to SGML-ize the >> > docs. Don't know if Bruce has got the time to get that done. >> >> Creating the SGML docs is trivial, especially compared to the 9.0 >> release notes SGML. ;-) It will take only an hour --- I am basically >> going to merge the README and the INSTALL file, remove mentions about >> migrating to < 9.0, and add SGML markup. I labored on README and the >> INSTALL files for a long time and can't figure out how to improve them. > > Oh, and I will remove the C code that was used to migrate _to_ pre-9.0 > databases. People can use the pgfoundry version for such cases. I have > specifically not created a pgfoundry release of pg_migrator that > migrates to 9.0. (It worked for the 8.5 numbering.) I wonder if this is just going to lead to us maintaining two versions of pg_migrator, which wouldn't be awesome. I don't think it's going to be practical to retain all the migration code for every pair of versions forever, but I'm reluctant to start changing things just because we're sucking the thing into our main tree. Especially things that sound suspiciously like features. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
C?dric Villemain wrote: > 2010/5/1 Bruce Momjian : > > Tom Lane wrote: > >> Bruce Momjian writes: > >> > While most of the limitations in previous versions of pg_migrator are > >> > gone, there are still issues with migrating /contrib modules, and there > >> > are many steps to its use. > >> > >> > I think to attain mass usage of pg_migrator, some type of one-click > >> > installer has to be built that can access the operating system and make > >> > the migration process simple, though that is probably beyond what we as > >> > a community are going to do. > >> > >> While the above are true statements, IMO the real gating factor right now > >> for pg_migrator is that people don't know whether they can trust it. > >> It won't get over that basic hump without a lot more real-world testing; > >> and as long as it's a separately distributed project I don't think it'll > >> get the necessary level of testing. ?That's why I feel it needs some > > > > Agreed. > > > >> time in contrib --- and why I don't have a warm fuzzy feeling about > >> Peter's proposal to push it directly into the core project. > > > > I am unclear why it would be in /bin if it requires 15 steps to run and > > is run only once by only some users. ?It seems natural for /contrib, > > like pgcrypto. > > Do we already have a process to upgrade postgres + HotStandby > (9.0->9.1 for example) ? No because when you create the file system backup and restore it to the new server, it is still running the same Postgres major version. There is no facility to handle major-version-changed master/slave setups. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
2010/5/1 Bruce Momjian : > Tom Lane wrote: >> Bruce Momjian writes: >> > While most of the limitations in previous versions of pg_migrator are >> > gone, there are still issues with migrating /contrib modules, and there >> > are many steps to its use. >> >> > I think to attain mass usage of pg_migrator, some type of one-click >> > installer has to be built that can access the operating system and make >> > the migration process simple, though that is probably beyond what we as >> > a community are going to do. >> >> While the above are true statements, IMO the real gating factor right now >> for pg_migrator is that people don't know whether they can trust it. >> It won't get over that basic hump without a lot more real-world testing; >> and as long as it's a separately distributed project I don't think it'll >> get the necessary level of testing. That's why I feel it needs some > > Agreed. > >> time in contrib --- and why I don't have a warm fuzzy feeling about >> Peter's proposal to push it directly into the core project. > > I am unclear why it would be in /bin if it requires 15 steps to run and > is run only once by only some users. It seems natural for /contrib, > like pgcrypto. Do we already have a process to upgrade postgres + HotStandby (9.0->9.1 for example) ? > >> As for the "one click" aspect, I think that that's largely on the heads >> of packagers to provide some convenient method of using it. For the RPM >> packages, I envision eventually having a "postgresql-migrate" package >> that contains pg_migrator, a copy of the version-N-minus-1 postmaster, >> and some sort of frontend script. Install, run the script, remove. >> But we're a long way from that yet. > > Yes, that is what is needed eventually. > > -- > Bruce Momjian http://momjian.us > EnterpriseDB http://enterprisedb.com > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Cédric Villemain -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
> > Agreed, we're not holding up 9.0 for it. I think the main bit of work > > that would be needed to put it into contrib would be to SGML-ize the > > docs. Don't know if Bruce has got the time to get that done. > > Creating the SGML docs is trivial, especially compared to the 9.0 > release notes SGML. ;-) It will take only an hour --- I am basically > going to merge the README and the INSTALL file, remove mentions about > migrating to < 9.0, and add SGML markup. I labored on README and the > INSTALL files for a long time and can't figure out how to improve them. Oh, and I will remove the C code that was used to migrate _to_ pre-9.0 databases. People can use the pgfoundry version for such cases. I have specifically not created a pgfoundry release of pg_migrator that migrates to 9.0. (It worked for the 8.5 numbering.) -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Robert Haas writes: > > On Sat, May 1, 2010 at 11:42 AM, Tom Lane wrote: > >> I think that having it in contrib for a release cycle or so would be > >> exactly the right approach, actually. ?Peter's position that it should > >> be in /bin is fine *once the bugs are out*. ?Just dropping it there > >> doesn't make the bugs go away. > > > I think in the previous iteration of this discussion I had the > > impression that you felt that it wasn't really to the point where it > > even met our standards for /contrib (although, admittedly, it seems > > those are pretty darn low, at least as far as the stuff that's already > > in there goes). If I misunderstood or if that that's no longer your > > feeling then maybe it makes sense. > > In the 8.3->8.4 cycle I did think it was pretty half-baked. We've > stomped many of the worst limitations since then, so I think that for > 8.4->9.0 it might be a credible solution. We won't really know unless > we try. True. I do see this as a much-requested feature (like built-in replication). > > But I don't think we should do it > > at this point unless it's as simple as "check it in and ship it". If > > doing this seems likely to make 9.0 take longer to get out the door, > > then I think we should wait and do it in 9.1 instead. > > Agreed, we're not holding up 9.0 for it. I think the main bit of work > that would be needed to put it into contrib would be to SGML-ize the > docs. Don't know if Bruce has got the time to get that done. Creating the SGML docs is trivial, especially compared to the 9.0 release notes SGML. ;-) It will take only an hour --- I am basically going to merge the README and the INSTALL file, remove mentions about migrating to < 9.0, and add SGML markup. I labored on README and the INSTALL files for a long time and can't figure out how to improve them. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Bruce Momjian writes: > > While most of the limitations in previous versions of pg_migrator are > > gone, there are still issues with migrating /contrib modules, and there > > are many steps to its use. > > > I think to attain mass usage of pg_migrator, some type of one-click > > installer has to be built that can access the operating system and make > > the migration process simple, though that is probably beyond what we as > > a community are going to do. > > While the above are true statements, IMO the real gating factor right now > for pg_migrator is that people don't know whether they can trust it. > It won't get over that basic hump without a lot more real-world testing; > and as long as it's a separately distributed project I don't think it'll > get the necessary level of testing. That's why I feel it needs some Agreed. > time in contrib --- and why I don't have a warm fuzzy feeling about > Peter's proposal to push it directly into the core project. I am unclear why it would be in /bin if it requires 15 steps to run and is run only once by only some users. It seems natural for /contrib, like pgcrypto. > As for the "one click" aspect, I think that that's largely on the heads > of packagers to provide some convenient method of using it. For the RPM > packages, I envision eventually having a "postgresql-migrate" package > that contains pg_migrator, a copy of the version-N-minus-1 postmaster, > and some sort of frontend script. Install, run the script, remove. > But we're a long way from that yet. Yes, that is what is needed eventually. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas writes: > On Sat, May 1, 2010 at 11:42 AM, Tom Lane wrote: >> I think that having it in contrib for a release cycle or so would be >> exactly the right approach, actually. Peter's position that it should >> be in /bin is fine *once the bugs are out*. Just dropping it there >> doesn't make the bugs go away. > I think in the previous iteration of this discussion I had the > impression that you felt that it wasn't really to the point where it > even met our standards for /contrib (although, admittedly, it seems > those are pretty darn low, at least as far as the stuff that's already > in there goes). If I misunderstood or if that that's no longer your > feeling then maybe it makes sense. In the 8.3->8.4 cycle I did think it was pretty half-baked. We've stomped many of the worst limitations since then, so I think that for 8.4->9.0 it might be a credible solution. We won't really know unless we try. > But I don't think we should do it > at this point unless it's as simple as "check it in and ship it". If > doing this seems likely to make 9.0 take longer to get out the door, > then I think we should wait and do it in 9.1 instead. Agreed, we're not holding up 9.0 for it. I think the main bit of work that would be needed to put it into contrib would be to SGML-ize the docs. Don't know if Bruce has got the time to get that done. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian writes: > While most of the limitations in previous versions of pg_migrator are > gone, there are still issues with migrating /contrib modules, and there > are many steps to its use. > I think to attain mass usage of pg_migrator, some type of one-click > installer has to be built that can access the operating system and make > the migration process simple, though that is probably beyond what we as > a community are going to do. While the above are true statements, IMO the real gating factor right now for pg_migrator is that people don't know whether they can trust it. It won't get over that basic hump without a lot more real-world testing; and as long as it's a separately distributed project I don't think it'll get the necessary level of testing. That's why I feel it needs some time in contrib --- and why I don't have a warm fuzzy feeling about Peter's proposal to push it directly into the core project. As for the "one click" aspect, I think that that's largely on the heads of packagers to provide some convenient method of using it. For the RPM packages, I envision eventually having a "postgresql-migrate" package that contains pg_migrator, a copy of the version-N-minus-1 postmaster, and some sort of frontend script. Install, run the script, remove. But we're a long way from that yet. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
> I think the big question is whether we want to provide a binary upgrade > facility for Postgres. If so, pg_migrator is the only facility > currently available, and I can't imagine another option appearing. I > would love for pg_migrator to be easier to use, but I can't figure out > how that can happen without an external OS-specific installer. FYI, the length of this thread is kind of why I waited for beta to talk about pg_migrator. pg_migrator is really external to the backend code and now we have the leisure time to talk about it, understand what /contrib is useful for, and make a decision. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: > On Sat, May 1, 2010 at 11:42 AM, Tom Lane wrote: > > Robert Haas writes: > >> On Sat, May 1, 2010 at 3:32 AM, Magnus Hagander > >> wrote: > >>> A lot of people are not willing to put stuff labeled "contrib" on > >>> their production boxes. > >>> > >>> And as Tom says, even we *ourselves* acknowledge that things in > >>> /contrib are held to a lower standard. If we put that label on > >>> pg_migrator, it doesn't exactly signal people that this is something > >>> they should use on their critical database. > > > >> Well, I guess that begs the question... ?IS this something they should > >> use on their critical database? > > > > Not unless it gets some serious testing during the 9.0 beta cycle. > > Which it surely won't get if it's not included in the core tarball. > > > > I think that having it in contrib for a release cycle or so would be > > exactly the right approach, actually. ?Peter's position that it should > > be in /bin is fine *once the bugs are out*. ?Just dropping it there > > doesn't make the bugs go away. > > I think in the previous iteration of this discussion I had the > impression that you felt that it wasn't really to the point where it > even met our standards for /contrib (although, admittedly, it seems > those are pretty darn low, at least as far as the stuff that's already > in there goes). If I misunderstood or if that that's no longer your > feeling then maybe it makes sense. But I don't think we should do it Well, the original suggestion to move pg_migrator to /contrib was that it would be easier to do per-major-version distributions of pg_migrator. However, I have code that is pretty clear about what version it is dealing with, so I am not worried about that. One concern I had that it would be harder to make fixes to pg_migrator if it was tied to Postgres releases. However, I am no longer worried about that because I have not made changes to pg_migrator for months. (Six months ago I was making major changes to pg_migrator regularly.) > at this point unless it's as simple as "check it in and ship it". If > doing this seems likely to make 9.0 take longer to get out the door, > then I think we should wait and do it in 9.1 instead. I can't imagine why pg_migrator would ever delay 9.0 -- it is in /contrib and everything it needs is already in the backend code. We could always rip it back out of /contrib if we wanted to, or disable it. That's the beauty of /contrib --- it is isolated from the rest of the system. I think the big question is whether we want to provide a binary upgrade facility for Postgres. If so, pg_migrator is the only facility currently available, and I can't imagine another option appearing. I would love for pg_migrator to be easier to use, but I can't figure out how that can happen without an external OS-specific installer. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 11:42 AM, Tom Lane wrote: > Robert Haas writes: >> On Sat, May 1, 2010 at 3:32 AM, Magnus Hagander wrote: >>> A lot of people are not willing to put stuff labeled "contrib" on >>> their production boxes. >>> >>> And as Tom says, even we *ourselves* acknowledge that things in >>> /contrib are held to a lower standard. If we put that label on >>> pg_migrator, it doesn't exactly signal people that this is something >>> they should use on their critical database. > >> Well, I guess that begs the question... IS this something they should >> use on their critical database? > > Not unless it gets some serious testing during the 9.0 beta cycle. > Which it surely won't get if it's not included in the core tarball. > > I think that having it in contrib for a release cycle or so would be > exactly the right approach, actually. Peter's position that it should > be in /bin is fine *once the bugs are out*. Just dropping it there > doesn't make the bugs go away. I think in the previous iteration of this discussion I had the impression that you felt that it wasn't really to the point where it even met our standards for /contrib (although, admittedly, it seems those are pretty darn low, at least as far as the stuff that's already in there goes). If I misunderstood or if that that's no longer your feeling then maybe it makes sense. But I don't think we should do it at this point unless it's as simple as "check it in and ship it". If doing this seems likely to make 9.0 take longer to get out the door, then I think we should wait and do it in 9.1 instead. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Robert Haas writes: > > On Sat, May 1, 2010 at 3:32 AM, Magnus Hagander wrote: > >> A lot of people are not willing to put stuff labeled "contrib" on > >> their production boxes. > >> > >> And as Tom says, even we *ourselves* acknowledge that things in > >> /contrib are held to a lower standard. If we put that label on > >> pg_migrator, it doesn't exactly signal people that this is something > >> they should use on their critical database. > > > Well, I guess that begs the question... IS this something they should > > use on their critical database? > > Not unless it gets some serious testing during the 9.0 beta cycle. > Which it surely won't get if it's not included in the core tarball. > > I think that having it in contrib for a release cycle or so would be > exactly the right approach, actually. Peter's position that it should > be in /bin is fine *once the bugs are out*. Just dropping it there > doesn't make the bugs go away. I think one aspect we might be missing is that /contrib has uses beyond experimental stuff. For example, I don't believe anyone thinks /contrib/pgcrypto is going to get more stable than it is now, but it is in /contrib because it has functionality that is useful to a limited number of users. I think these /contrib modules fall into a similar category: auto_explain/ fuzzystrmatch/ hstore/ isn/ oid2name/ pageinspect/ pg_buffercache/ pg_freespacemap/ pg_standby/ pg_stat_statements/ pgbench/ pgrowlocks/ pgstattuple/ sslinfo/ unaccent/ That is over a third of the /contrib modules. I think pg_migrator falls into that category too --- it is only of use to people wanting to do a binary upgrade, and even then, they only run it once. And it is not something you are going to just fire up like psql. Here is the pg_migrator README: http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/~checkout~/pg-migrator/pg_migrator/README?rev=1.72&content-type=text/plain and the 15-step INSTALL file: http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/~checkout~/pg-migrator/pg_migrator/INSTALL?rev=1.56&content-type=text/plain While most of the limitations in previous versions of pg_migrator are gone, there are still issues with migrating /contrib modules, and there are many steps to its use. I think to attain mass usage of pg_migrator, some type of one-click installer has to be built that can access the operating system and make the migration process simple, though that is probably beyond what we as a community are going to do. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas writes: > On Sat, May 1, 2010 at 3:32 AM, Magnus Hagander wrote: >> A lot of people are not willing to put stuff labeled "contrib" on >> their production boxes. >> >> And as Tom says, even we *ourselves* acknowledge that things in >> /contrib are held to a lower standard. If we put that label on >> pg_migrator, it doesn't exactly signal people that this is something >> they should use on their critical database. > Well, I guess that begs the question... IS this something they should > use on their critical database? Not unless it gets some serious testing during the 9.0 beta cycle. Which it surely won't get if it's not included in the core tarball. I think that having it in contrib for a release cycle or so would be exactly the right approach, actually. Peter's position that it should be in /bin is fine *once the bugs are out*. Just dropping it there doesn't make the bugs go away. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 3:32 AM, Magnus Hagander wrote: > On Sat, May 1, 2010 at 02:23, Bruce Momjian wrote: >> Tom Lane wrote: >>> Dimitri Fontaine writes: >>> > Peter Eisentraut writes: >>> >> I also think that the standards for contrib should not be so lax that a >>> >> completely new module can be added after beta. (This is mostly informed >>> >> by the feeling that contrib should go away entirely.) >>> >>> > +1 >>> >>> > For the record, the contrib replacement would look like proper Extension >>> > handling in dump&restore, PGXS support for windows, and PGAN for source >>> > level archive distribution. We'd still rely on distributions support for >>> > binaries. >>> >>> Both of you are living in some fantasy land. The reason contrib is held >>> to a lower standard than core is that nobody is willing to put the same >>> level of effort into contrib. There are modules in there (most of them, >>> in fact) that haven't been touched for years, other than as part of >>> system-wide search-and-replace patches. Extension support is not going >>> to magically fix that and cause maintenance effort to appear from >>> nowhere. >>> >>> In the end, the main useful function that contrib serves is to provide >>> examples of how to write Postgres extensions. Because of that, removing >>> it as Peter suggests doesn't seem like a good idea to me. >> >> So what do people want to do with pg_migrator? I don't think calling >> pg_migrator a major features requires it to be in /bin. We added full >> text search to /contrib years ago and that was a major feature. > > > There is a reason people said that "8.3 comes with full text search" - > that's when it really became a major feature to the outside world. > Before that, it wasn't included in most comparisons. It was a PITA to > install (well, not install, but upgrading when you had it, etc). (once > you knew the insides, it was a major feature yes, but people didn't > know about that) > > A lot of people are not willing to put stuff labeled "contrib" on > their production boxes. > > And as Tom says, even we *ourselves* acknowledge that things in > /contrib are held to a lower standard. If we put that label on > pg_migrator, it doesn't exactly signal people that this is something > they should use on their critical database. Well, I guess that begs the question... IS this something they should use on their critical database? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sat, May 1, 2010 at 02:23, Bruce Momjian wrote: > Tom Lane wrote: >> Dimitri Fontaine writes: >> > Peter Eisentraut writes: >> >> I also think that the standards for contrib should not be so lax that a >> >> completely new module can be added after beta. (This is mostly informed >> >> by the feeling that contrib should go away entirely.) >> >> > +1 >> >> > For the record, the contrib replacement would look like proper Extension >> > handling in dump&restore, PGXS support for windows, and PGAN for source >> > level archive distribution. We'd still rely on distributions support for >> > binaries. >> >> Both of you are living in some fantasy land. The reason contrib is held >> to a lower standard than core is that nobody is willing to put the same >> level of effort into contrib. There are modules in there (most of them, >> in fact) that haven't been touched for years, other than as part of >> system-wide search-and-replace patches. Extension support is not going >> to magically fix that and cause maintenance effort to appear from >> nowhere. >> >> In the end, the main useful function that contrib serves is to provide >> examples of how to write Postgres extensions. Because of that, removing >> it as Peter suggests doesn't seem like a good idea to me. > > So what do people want to do with pg_migrator? I don't think calling > pg_migrator a major features requires it to be in /bin. We added full > text search to /contrib years ago and that was a major feature. There is a reason people said that "8.3 comes with full text search" - that's when it really became a major feature to the outside world. Before that, it wasn't included in most comparisons. It was a PITA to install (well, not install, but upgrading when you had it, etc). (once you knew the insides, it was a major feature yes, but people didn't know about that) A lot of people are not willing to put stuff labeled "contrib" on their production boxes. And as Tom says, even we *ourselves* acknowledge that things in /contrib are held to a lower standard. If we put that label on pg_migrator, it doesn't exactly signal people that this is something they should use on their critical database. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Dimitri Fontaine writes: > > Peter Eisentraut writes: > >> I also think that the standards for contrib should not be so lax that a > >> completely new module can be added after beta. (This is mostly informed > >> by the feeling that contrib should go away entirely.) > > > +1 > > > For the record, the contrib replacement would look like proper Extension > > handling in dump&restore, PGXS support for windows, and PGAN for source > > level archive distribution. We'd still rely on distributions support for > > binaries. > > Both of you are living in some fantasy land. The reason contrib is held > to a lower standard than core is that nobody is willing to put the same > level of effort into contrib. There are modules in there (most of them, > in fact) that haven't been touched for years, other than as part of > system-wide search-and-replace patches. Extension support is not going > to magically fix that and cause maintenance effort to appear from > nowhere. > > In the end, the main useful function that contrib serves is to provide > examples of how to write Postgres extensions. Because of that, removing > it as Peter suggests doesn't seem like a good idea to me. So what do people want to do with pg_migrator? I don't think calling pg_migrator a major features requires it to be in /bin. We added full text search to /contrib years ago and that was a major feature. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Andrew Dunstan writes: > Quite so. Getting a better extensions mechanism doesn't mean we should > abandon what we currently have, IMNSHO. Yeah, agreed. Exactly what I proposed. The only change is the distribution mean. Either we keep things as they are now exactly, or we use the new Archive Network facilities, in the spirit of being sure they get exercised, requiring that commiters gets to use them and maybe use a separate code repository for contribs. Or simply an adjusted Makefile. Summary: the only proposed change is how to do it, not what we do. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On fre, 2010-04-30 at 10:45 -0400, Tom Lane wrote: > In the end, the main useful function that contrib serves is to provide > examples of how to write Postgres extensions. Maybe, but pg_migrator surely doesn't fit that. And neither does about a third of the other contrib modules, IMO. > Because of that, removing > it as Peter suggests doesn't seem like a good idea to me. contrib means many things to many people, and that's exactly the problem in my mind: It doesn't mean anything in particular. If we were to separate it into - examples - production-quality add-ons with small user base - production-quality add-ons that everyone wants, but we keep them as plugins because plugins are cool - experimental code that we wanted to ship anyway - (historically) differently licensed code then these discussions would be much simpler. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: Dimitri Fontaine writes: Peter Eisentraut writes: I also think that the standards for contrib should not be so lax that a completely new module can be added after beta. (This is mostly informed by the feeling that contrib should go away entirely.) +1 For the record, the contrib replacement would look like proper Extension handling in dump&restore, PGXS support for windows, and PGAN for source level archive distribution. We'd still rely on distributions support for binaries. Both of you are living in some fantasy land. The reason contrib is held to a lower standard than core is that nobody is willing to put the same level of effort into contrib. There are modules in there (most of them, in fact) that haven't been touched for years, other than as part of system-wide search-and-replace patches. Extension support is not going to magically fix that and cause maintenance effort to appear from nowhere. In the end, the main useful function that contrib serves is to provide examples of how to write Postgres extensions. Because of that, removing it as Peter suggests doesn't seem like a good idea to me. Quite so. Getting a better extensions mechanism doesn't mean we should abandon what we currently have, IMNSHO. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Dimitri Fontaine writes: > Peter Eisentraut writes: >> I also think that the standards for contrib should not be so lax that a >> completely new module can be added after beta. (This is mostly informed >> by the feeling that contrib should go away entirely.) > +1 > For the record, the contrib replacement would look like proper Extension > handling in dump&restore, PGXS support for windows, and PGAN for source > level archive distribution. We'd still rely on distributions support for > binaries. Both of you are living in some fantasy land. The reason contrib is held to a lower standard than core is that nobody is willing to put the same level of effort into contrib. There are modules in there (most of them, in fact) that haven't been touched for years, other than as part of system-wide search-and-replace patches. Extension support is not going to magically fix that and cause maintenance effort to appear from nowhere. In the end, the main useful function that contrib serves is to provide examples of how to write Postgres extensions. Because of that, removing it as Peter suggests doesn't seem like a good idea to me. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Peter Eisentraut writes: > My personal feeling is that pg_migrator should be fully integrated, but > it's too late for that, obviously. Let's do it for 9.1. +1 > I also think that the standards for contrib should not be so lax that a > completely new module can be added after beta. (This is mostly informed > by the feeling that contrib should go away entirely.) +1 For the record, the contrib replacement would look like proper Extension handling in dump&restore, PGXS support for windows, and PGAN for source level archive distribution. We'd still rely on distributions support for binaries. Those are the technical layers we need, then we'd have a PGAN entry for replacing contrib, and a host of other ones. The contrib Archive Network would contain -core reviewed (and maintained?) extensions, the other ones are on their own. Maybe the main other one would be (could be?) a new component of pgfoundry. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On tor, 2010-04-29 at 17:34 -0400, Bruce Momjian wrote: > I talked to a few people personally about this, and it seems there was a > misunderstanding. I was not asking if pg_migrator should be in 9.0 > beta1. I was asking if we should think about putting it into a later > 9.0 beta, like 9.0 beta3. It would be another major 9.0 feature. If it's supposed to be a major feature, then it should be in src/bin, and fully integrated in the install, the documentation, etc. If you want to put it in contrib because the standards are more lax there, then by definition it's not really a major feature, it's just a random feature that was snuck in. You can't have it both ways. My personal feeling is that pg_migrator should be fully integrated, but it's too late for that, obviously. Let's do it for 9.1. I also think that the standards for contrib should not be so lax that a completely new module can be added after beta. (This is mostly informed by the feeling that contrib should go away entirely.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Mark Kirkwood wrote: > Bruce Momjian wrote: > > > > and most of the limitations of pg_migrator are gone when migrating to > > 9.0, even from Postgres 8.3. This could help beta testers move their > > data to 9.0 as well. > > > > > Wouldn't this help even for beta1? It would, but there is so much work going into getting other features done that there just isn't enough time. We don't want pg_migrator delaying beta1. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Mark Kirkwood writes: > Bruce Momjian wrote: >> and most of the limitations of pg_migrator are gone when migrating to >> 9.0, even from Postgres 8.3. This could help beta testers move their >> data to 9.0 as well. > Wouldn't this help even for beta1? It's too late for beta1. It probably should have been proposed when there was still time ... but it wasn't. I'm not necessarily averse to shoving it in as of beta2 or beta3 or so; we've always been laxer about contrib than the core server. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: and most of the limitations of pg_migrator are gone when migrating to 9.0, even from Postgres 8.3. This could help beta testers move their data to 9.0 as well. Wouldn't this help even for beta1? Cheers Mark -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_migrator to /contrib in a later 9.0 beta
Tom Lane wrote: > Robert Haas writes: > > On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: > >> There was talk of including pg_migrator in Postgres 9.0 in /contrib. ?Do > >> we still want to do that? > > > I think you articulated some pretty good reasons previously for > > keeping it separate and, at any rate, I'm not eager to do it at the > > 11th hour without due consideration and adequate engineering time. > > I concur; it's about a month too late to propose this. I talked to a few people personally about this, and it seems there was a misunderstanding. I was not asking if pg_migrator should be in 9.0 beta1. I was asking if we should think about putting it into a later 9.0 beta, like 9.0 beta3. It would be another major 9.0 feature. Because pg_migrator is an external project, it seemed best to do it after beta1, while we are just waiting for bug reports, and not during our main push to beta1. FYI, here was the last discussion about having pg_migrator in /contrib in December 2009: http://archives.postgresql.org/pgsql-hackers/2009-12/msg01787.php and most of the limitations of pg_migrator are gone when migrating to 9.0, even from Postgres 8.3. This could help beta testers move their data to 9.0 as well. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
Andrew Dunstan wrote: > > > Bruce Momjian wrote: > >> I concur; it's about a month too late to propose this. > >> > > > > I am confused why it is late. We add to /contrib even during beta, and > > I didn't bring it up earlier because I didn't want to be pushing my own > > software. Was someone else supposed to suggest it a month ago? > > > > > > Bruce, > > you're not usually such a shrinking violet. If you don't push your > project it's less likely others will, IMNSHO. Well, I am sensitive to be pushing my external project into /contrib when I am someone who puts other stuff into contrib, and I have been working on pg_migrator for only one year. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
Bruce Momjian wrote: I concur; it's about a month too late to propose this. I am confused why it is late. We add to /contrib even during beta, and I didn't bring it up earlier because I didn't want to be pushing my own software. Was someone else supposed to suggest it a month ago? Bruce, you're not usually such a shrinking violet. If you don't push your project it's less likely others will, IMNSHO. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
Tom Lane wrote: > Robert Haas writes: > > On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: > >> There was talk of including pg_migrator in Postgres 9.0 in /contrib. ?Do > >> we still want to do that? > > > I think you articulated some pretty good reasons previously for > > keeping it separate and, at any rate, I'm not eager to do it at the > > 11th hour without due consideration and adequate engineering time. > > I concur; it's about a month too late to propose this. I am confused why it is late. We add to /contrib even during beta, and I didn't bring it up earlier because I didn't want to be pushing my own software. Was someone else supposed to suggest it a month ago? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
Robert Haas writes: > On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: >> There was talk of including pg_migrator in Postgres 9.0 in /contrib. Do >> we still want to do that? > I think you articulated some pretty good reasons previously for > keeping it separate and, at any rate, I'm not eager to do it at the > 11th hour without due consideration and adequate engineering time. I concur; it's about a month too late to propose this. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
On Mon, Apr 26, 2010 at 9:46 PM, Bruce Momjian wrote: > Robert Haas wrote: >> On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: >> > There was talk of including pg_migrator in Postgres 9.0 in /contrib. ?Do >> > we still want to do that? >> >> I think you articulated some pretty good reasons previously for >> keeping it separate and, at any rate, I'm not eager to do it at the >> 11th hour without due consideration and adequate engineering time. So >> I vote for holding off for this release and possibly revisiting at >> some point down the road. > > You might also remember I was outvoted. It will not be hard to put it > in /contrib as that is already a valid build option for pg_migrator. [shrug...] If that's the consensus I'll go along with it, but I'm not excited about adding more things to our to-do list at this point, even apparently simple ones. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
Robert Haas wrote: > On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: > > There was talk of including pg_migrator in Postgres 9.0 in /contrib. ?Do > > we still want to do that? > > I think you articulated some pretty good reasons previously for > keeping it separate and, at any rate, I'm not eager to do it at the > 11th hour without due consideration and adequate engineering time. So > I vote for holding off for this release and possibly revisiting at > some point down the road. You might also remember I was outvoted. It will not be hard to put it in /contrib as that is already a valid build option for pg_migrator. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator
On Mon, Apr 26, 2010 at 9:26 PM, Bruce Momjian wrote: > There was talk of including pg_migrator in Postgres 9.0 in /contrib. Do > we still want to do that? I think you articulated some pretty good reasons previously for keeping it separate and, at any rate, I'm not eager to do it at the 11th hour without due consideration and adequate engineering time. So I vote for holding off for this release and possibly revisiting at some point down the road. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_migrator
There was talk of including pg_migrator in Postgres 9.0 in /contrib. Do we still want to do that? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
FYI, I consider all the issues below to be addressed (we did all but #4), and pg_migrator will take advantage of these new facilities for 8.5. --- Bruce Momjian wrote: > pg_migrator has become more popular recently, so it seems time to look > at some enhancements that would improve pg_migrator. None of these are > required, but rather changes that would be nice to have: > > 1) Right now pg_migrator preserves relfilenodes for TOAST files because > this is required for proper migration. Now that we have shown that > strategically-placed global variables with a server-side function to set > them is a viable solution, it would be nice to preserve all relfilenodes > from the old server. This would simplify pg_migrator by no long > requiring place-holder relfilenodes or the renaming of TOAST files. A > simpler solution would just be to allow TOAST table creation to > automatically remove placeholder files and create specified relfilenodes > via global variables. > > 2) Right now pg_migrator renames old tablespaces to .old, which fails > if the tablespaces are on mount points. I have already received a > report of such a failure. $PGDATA also has that issue, but that > renaming has to be done by the user before pg_migrator is run, and only > if they want to keep the same $PGDATA value after migration, i.e. no > version-specific directory path. One idea we floated around was to have > tablespaces use major version directory names under the tablespace > directory so renaming would not be necessary. I could implement a > pg_migrator --delete-old flag to cleanly delete the old 8.4 server files > which are not in a version-specific subdirectory. > > 3) There is no easy way to analyze all databases. vacuumdb --analyze > does analyze _and_ vacuum, which for an 8.4 to 8.5 migration does an > unnecessary vacuum. Right now I recommend ANALYZE in every database, > but it would be nice if there were a single command which did this. > > 4) I have implemented the ability to run pg_migrator --check on a live > old server. However, pg_migrator uses information from controldata to > check things, and it also needs xid information that is only available > via pg_resetxlog -n(no update) to perform the migration. Unfortunately, > pg_resetxlog -n cannot be run on a live server, so pg_migrator runs > pg_controldata for --check and pg_resetxlog -n for real upgrades. It > would simplify pg_migrator if I would run pg_resetxlog -n on a live > server, but I can understand if people don't want to do that because the > xid information reported on a live server is inaccurate. > > Comments? > > -- > Bruce Momjian http://momjian.us > EnterpriseDB http://enterprisedb.com > > + If your life is a hard drive, Christ can be your backup. + > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Bruce Momjian wrote: > Bruce Momjian wrote: > > 2) Right now pg_migrator renames old tablespaces to .old, which fails > > if the tablespaces are on mount points. I have already received a > > report of such a failure. $PGDATA also has that issue, but that > > renaming has to be done by the user before pg_migrator is run, and only > > if they want to keep the same $PGDATA value after migration, i.e. no > > version-specific directory path. One idea we floated around was to have > > tablespaces use major version directory names under the tablespace > > directory so renaming would not be necessary. I could implement a > > pg_migrator --delete-old flag to cleanly delete the old 8.4 server files > > which are not in a version-specific subdirectory. > > I have created a patch to implement per-cluster directories in > tablespaces. This is for use by pg_migrator so it doesn't have to > rename the tablespaces during the migration. Users still need to remove > the old cluster's tablespace subdirectory, and I can add a --delete-old > option to pg_migrator to do that. > > The old code used a symlink from pg_tblspc/ to the location > directory specified in CREATE TABLESPACE. During CREATE TABLESPACE, a > PG_VERSION file is created containing the major version number. Anytime > a database object is created in the tablespace, a per-database directory > is created. > > With the new code in this patch, pg_tblspc/ points to the CREATE > TABLESPACE directory just like before, but a new directory, PG_ + > major_version + catalog_version, e.g. PG_8.5_201001061, is created and > all per-database directories are created under that directory. This > directory has the same purpose as the old PG_VERSION file. One > disadvantage of this approach is that functions that need to look inside > tablespaces must now also specify the version directory, e.g. > pg_tablespace_databases(). > > An alternative approach would be for the pg_tblspc/ symbolic link to > point to the new version directory, PG_*, but that makes removal of the > version directory complicated, particularly during WAL replay where we > don't have access to the system catalogs, and readlink() to read the > symbolic link target is not supported on all operating systems > (particularly Win32). > > I used the version directory pattern "PG_8.5_201001061" because "PG_" > helps people realize the directory is for the use of Postgres > (PG_VERSION is gone in tablespaces), and the catalog version number > enables alpha migrations. The major version number is not necessary but > probably useful for administrators. > > pg_migrator is going to need to know about the version directory too, > and it can't use the C macro --- it has to construct the directory > pattern based on the contents of pg_control from the old and new > servers. And, it is going to be difficult to run pg_control on the old > server for pg_migrator --delete-old after migration because it is > renamed to pg_control.old --- I will need to create a symbolic link > during the time I run pg_controldata. Also, the contents of the > tablespace directory for an 8.4 to 8.5 migration is going to be ugly > because there will be many numeric directories (for databases), and > PG_VERSION (for 8.4), and the PG_8.5_201001061 directory which should > not be touched. > > Can someone explain why TablespaceCreateDbspace() creates a non-symlink > directory during recovery if the symlink is missing? Is it just for > robustness? I would like to document that more clearly. Applied. FYI, I decide to create a pg_migrator_remove_old_cluster.sh/.bat file that can be run by the user after the upgrade, instead of adding a --delete-old-cluster option to pg_migrator. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Bruce Momjian wrote:
> 2) Right now pg_migrator renames old tablespaces to .old, which fails
> if the tablespaces are on mount points. I have already received a
> report of such a failure. $PGDATA also has that issue, but that
> renaming has to be done by the user before pg_migrator is run, and only
> if they want to keep the same $PGDATA value after migration, i.e. no
> version-specific directory path. One idea we floated around was to have
> tablespaces use major version directory names under the tablespace
> directory so renaming would not be necessary. I could implement a
> pg_migrator --delete-old flag to cleanly delete the old 8.4 server files
> which are not in a version-specific subdirectory.
I have created a patch to implement per-cluster directories in
tablespaces. This is for use by pg_migrator so it doesn't have to
rename the tablespaces during the migration. Users still need to remove
the old cluster's tablespace subdirectory, and I can add a --delete-old
option to pg_migrator to do that.
The old code used a symlink from pg_tblspc/ to the location
directory specified in CREATE TABLESPACE. During CREATE TABLESPACE, a
PG_VERSION file is created containing the major version number. Anytime
a database object is created in the tablespace, a per-database directory
is created.
With the new code in this patch, pg_tblspc/ points to the CREATE
TABLESPACE directory just like before, but a new directory, PG_ +
major_version + catalog_version, e.g. PG_8.5_201001061, is created and
all per-database directories are created under that directory. This
directory has the same purpose as the old PG_VERSION file. One
disadvantage of this approach is that functions that need to look inside
tablespaces must now also specify the version directory, e.g.
pg_tablespace_databases().
An alternative approach would be for the pg_tblspc/ symbolic link to
point to the new version directory, PG_*, but that makes removal of the
version directory complicated, particularly during WAL replay where we
don't have access to the system catalogs, and readlink() to read the
symbolic link target is not supported on all operating systems
(particularly Win32).
I used the version directory pattern "PG_8.5_201001061" because "PG_"
helps people realize the directory is for the use of Postgres
(PG_VERSION is gone in tablespaces), and the catalog version number
enables alpha migrations. The major version number is not necessary but
probably useful for administrators.
pg_migrator is going to need to know about the version directory too,
and it can't use the C macro --- it has to construct the directory
pattern based on the contents of pg_control from the old and new
servers. And, it is going to be difficult to run pg_control on the old
server for pg_migrator --delete-old after migration because it is
renamed to pg_control.old --- I will need to create a symbolic link
during the time I run pg_controldata. Also, the contents of the
tablespace directory for an 8.4 to 8.5 migration is going to be ugly
because there will be many numeric directories (for databases), and
PG_VERSION (for 8.4), and the PG_8.5_201001061 directory which should
not be touched.
Can someone explain why TablespaceCreateDbspace() creates a non-symlink
directory during recovery if the symlink is missing? Is it just for
robustness? I would like to document that more clearly.
Comments?
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: src/backend/catalog/catalog.c
===
RCS file: /cvsroot/pgsql/src/backend/catalog/catalog.c,v
retrieving revision 1.86
diff -c -c -r1.86 catalog.c
*** src/backend/catalog/catalog.c 6 Jan 2010 02:41:37 - 1.86
--- src/backend/catalog/catalog.c 8 Jan 2010 01:07:41 -
***
*** 115,130
else
{
/* All other tablespaces are accessed via symlinks */
! pathlen = 10 + OIDCHARS + 1 + OIDCHARS + 1 + OIDCHARS + 1
! + FORKNAMECHARS + 1;
path = (char *) palloc(pathlen);
if (forknum != MAIN_FORKNUM)
! snprintf(path, pathlen, "pg_tblspc/%u/%u/%u_%s",
! rnode.spcNode, rnode.dbNode, rnode.relNode,
! forkNames[forknum]);
else
! snprintf(path, pathlen, "pg_tblspc/%u/%u/%u",
! rnode.spcNode, rnode.dbNode, rnode.relNode);
}
return path;
}
--- 115,131
else
{
/* All other tablespaces are accessed via symlinks */
! pathlen = 9 + 1 + OIDCHARS + 1 + strlen(TABLESPACE_VERSION_DIRECTORY) +
! 1 + OIDCHARS + 1 + OIDCHARS + 1 + FORKNAMECHARS + 1;
path = (char *) palloc(pathlen);
if (forknum != MAIN_FORKNUM)
! snprintf(path, pathlen, "pg_tblspc/%u/%s/%u/%u_%s",
! rnode.spcNode, TABLESPACE_VERSION_DIRECTORY,
! rnode.dbNode, rnode.relNode, forkNames[forknum]);
else
! snprintf(path, pathlen, "pg_tblspc/%u/%s/%u/%u",
! rnode.
Re: [HACKERS] pg_migrator issues
Alvaro Herrera wrote:
> Bruce Momjian escribi?:
>
> > > Oh, interesting about pg_dump. Let's just go with --analyze-only.
> > > --only-analyze is feeling odd to me too.
> >
> > Done, attached and applied.
>
>
> Why -o and not, say, -Z? I imagine you picked -o for "only" but it
> seems strange.
>
Hmmm, sure -Z makes sense. Change applied with attached patch. Thanks.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/ref/vacuumdb.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/ref/vacuumdb.sgml,v
retrieving revision 1.48
diff -c -c -r1.48 vacuumdb.sgml
*** doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 12:38:55 - 1.48
--- doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 14:34:39 -
***
*** 28,34
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-o
--table | -t table
( column [,...] )
--- 28,34
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-Z
--table | -t table
( column [,...] )
***
*** 42,48
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-o
--- 42,48
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-Z
***
*** 142,148
! -o
--analyze-only
--- 142,148
! -Z
--analyze-only
Index: src/bin/scripts/vacuumdb.c
===
RCS file: /cvsroot/pgsql/src/bin/scripts/vacuumdb.c,v
retrieving revision 1.32
diff -c -c -r1.32 vacuumdb.c
*** src/bin/scripts/vacuumdb.c 7 Jan 2010 12:38:55 - 1.32
--- src/bin/scripts/vacuumdb.c 7 Jan 2010 14:34:44 -
***
*** 41,47
{"quiet", no_argument, NULL, 'q'},
{"dbname", required_argument, NULL, 'd'},
{"analyze", no_argument, NULL, 'z'},
! {"analyze-only", no_argument, NULL, 'o'},
{"freeze", no_argument, NULL, 'F'},
{"all", no_argument, NULL, 'a'},
{"table", required_argument, NULL, 't'},
--- 41,47
{"quiet", no_argument, NULL, 'q'},
{"dbname", required_argument, NULL, 'd'},
{"analyze", no_argument, NULL, 'z'},
! {"analyze-only", no_argument, NULL, 'Z'},
{"freeze", no_argument, NULL, 'F'},
{"all", no_argument, NULL, 'a'},
{"table", required_argument, NULL, 't'},
***
*** 107,113
case 'z':
and_analyze = true;
break;
! case 'o':
analyze_only = true;
break;
case 'F':
--- 107,113
case 'z':
and_analyze = true;
break;
! case 'Z':
analyze_only = true;
break;
case 'F':
***
*** 351,361
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freezefreeze row transaction information\n"));
printf(_(" -i, --inplace do full inplace vacuuming\n"));
- printf(_(" -o, --analyze-only only update optimizer hints\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table only\n"));
printf(_(" -v, --verbose write a lot of output\n"));
printf(_(" -z, --analyze update optimizer hints\n"));
printf(_(" --help show this help, then exit\n"));
printf(_(" --version output version information, then exit\n"));
printf(_("\nConnection options:\n"));
--- 351,361
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freezefreeze row transaction information\n"));
printf(_(" -i, --inplace do full inplace vacuuming\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table only\n"));
printf(_(" -v, --verbose write a lot of output\n"));
printf(_(" -z, --analyze update optimizer hints\n"));
+ printf(_(" -Z, --analyze-only only update optimizer hints\n"));
printf(_(" --help show this help, then exit\n"));
printf(_(" --version output version information, then exit\n"));
printf(_("\nConnection options:\n"));
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Bruce Momjian escribió: > > Oh, interesting about pg_dump. Let's just go with --analyze-only. > > --only-analyze is feeling odd to me too. > > Done, attached and applied. Why -o and not, say, -Z? I imagine you picked -o for "only" but it seems strange. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
> Tom Lane wrote:
> > Peter Eisentraut writes:
> > > On mn, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote:
> > >> Yea, I am not excited about having vacuumdb do only analyze, but it
> > >> seems the most minimal solution. I spelled it --only-analyze and just
> > >> posted the reason and patch.
> >
> > > I can't find the patch and the reason, but note that we already have
> > > other options like --data-only, --schema-only, --tuples-only. I
> > > personally don't like the spelling of --only-analyze.
> >
> > In particular note that pg_dump has options --schema and --schema-only,
> > and nobody has complained about that. I concur with Peter that this
> > spelling is gratuitously unlike everyplace else.
>
> Oh, interesting about pg_dump. Let's just go with --analyze-only.
> --only-analyze is feeling odd to me too.
Done, attached and applied.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/ref/vacuumdb.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/ref/vacuumdb.sgml,v
retrieving revision 1.47
diff -c -c -r1.47 vacuumdb.sgml
*** doc/src/sgml/ref/vacuumdb.sgml 6 Jan 2010 05:31:13 - 1.47
--- doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 12:37:08 -
***
*** 28,34
--freeze-F
--verbose-v
--analyze-z
!--only-analyze-o
--table | -t table
( column [,...] )
--- 28,34
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-o
--table | -t table
( column [,...] )
***
*** 42,48
--freeze-F
--verbose-v
--analyze-z
!--only-analyze-o
--- 42,48
--freeze-F
--verbose-v
--analyze-z
!--analyze-only-o
***
*** 143,149
-o
! --only-analyze
Only calculate statistics for use by the optimizer (no vacuum).
--- 143,149
-o
! --analyze-only
Only calculate statistics for use by the optimizer (no vacuum).
***
*** 168,174
Clean or analyze table only.
Column names can be specified only in conjunction with
! the --analyze or --only-analyze options.
--- 168,174
Clean or analyze table only.
Column names can be specified only in conjunction with
! the --analyze or --analyze-only options.
Index: src/bin/scripts/vacuumdb.c
===
RCS file: /cvsroot/pgsql/src/bin/scripts/vacuumdb.c,v
retrieving revision 1.31
diff -c -c -r1.31 vacuumdb.c
*** src/bin/scripts/vacuumdb.c 6 Jan 2010 16:04:05 - 1.31
--- src/bin/scripts/vacuumdb.c 7 Jan 2010 12:37:08 -
***
*** 15,26
static void vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose,
! bool and_analyze, bool only_analyze, bool freeze,
const char *table, const char *host, const char *port,
const char *username, enum trivalue prompt_password,
const char *progname, bool echo);
static void vacuum_all_databases(bool full, bool inplace, bool verbose, bool and_analyze,
! bool only_analyze, bool freeze,
const char *host, const char *port,
const char *username, enum trivalue prompt_password,
const char *progname, bool echo, bool quiet);
--- 15,26
static void vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose,
! bool and_analyze, bool analyze_only, bool freeze,
const char *table, const char *host, const char *port,
const char *username, enum trivalue prompt_password,
const char *progname, bool echo);
static void vacuum_all_databases(bool full, bool inplace, bool verbose, bool and_analyze,
! bool analyze_only, bool freeze,
const char *host, const char *port,
const char *username, enum trivalue prompt_password,
const char *progname, bool echo, bool quiet);
***
*** 41,47
{"quiet", no_argument, NULL, 'q'},
{"dbname", required_argument, NULL, 'd'},
{"analyze", no_argument, NULL, 'z'},
! {"only-analyze", no_argument, NULL, 'o'},
{"freeze", no_argument, NULL, 'F'},
{"all", no_argument, NULL, 'a'},
{"table", required_argument, NULL, 't'},
--- 41,47
{"quiet", no_argument, NULL, 'q'},
{"dbname", required_argument, NULL, 'd'},
{"analyze", no_argument, NULL, 'z'},
! {"analyze-only", no_argument, NULL, 'o'},
{"freeze", no_argument, NULL, 'F'},
{"all", no_argument, NULL, 'a'},
{"table", required_argument, NULL, 't'},
***
*** 63,69
bool echo = false;
bool quiet = false;
bool and_analyze = fa
Re: [HACKERS] pg_migrator issues
Tom Lane wrote: > Peter Eisentraut writes: > > On m??n, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote: > >> Yea, I am not excited about having vacuumdb do only analyze, but it > >> seems the most minimal solution. I spelled it --only-analyze and just > >> posted the reason and patch. > > > I can't find the patch and the reason, but note that we already have > > other options like --data-only, --schema-only, --tuples-only. I > > personally don't like the spelling of --only-analyze. > > In particular note that pg_dump has options --schema and --schema-only, > and nobody has complained about that. I concur with Peter that this > spelling is gratuitously unlike everyplace else. Oh, interesting about pg_dump. Let's just go with --analyze-only. --only-analyze is feeling odd to me too. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Peter Eisentraut writes: > On mån, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote: >> Yea, I am not excited about having vacuumdb do only analyze, but it >> seems the most minimal solution. I spelled it --only-analyze and just >> posted the reason and patch. > I can't find the patch and the reason, but note that we already have > other options like --data-only, --schema-only, --tuples-only. I > personally don't like the spelling of --only-analyze. In particular note that pg_dump has options --schema and --schema-only, and nobody has complained about that. I concur with Peter that this spelling is gratuitously unlike everyplace else. Perhaps we could compromise by calling it "--no-vacuum", instead? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
On mån, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote: > Yea, I am not excited about having vacuumdb do only analyze, but it > seems the most minimal solution. I spelled it --only-analyze and just > posted the reason and patch. I can't find the patch and the reason, but note that we already have other options like --data-only, --schema-only, --tuples-only. I personally don't like the spelling of --only-analyze. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Bruce Momjian wrote: > 2) Right now pg_migrator renames old tablespaces to .old, which fails > if the tablespaces are on mount points. I have already received a > report of such a failure. $PGDATA also has that issue, but that > renaming has to be done by the user before pg_migrator is run, and only > if they want to keep the same $PGDATA value after migration, i.e. no > version-specific directory path. One idea we floated around was to have > tablespaces use major version directory names under the tablespace > directory so renaming would not be necessary. I could implement a > pg_migrator --delete-old flag to cleanly delete the old 8.4 server files > which are not in a version-specific subdirectory. FYI, pg_migrator CVS now uses the relfilenode preservation ability in 8.5 to avoid the creation of placeholder relfilenodes and renaming. It also recommends using vacuumdb --only-analyze. To simplify pg_migrator, you can now only migrate _to_ 8.5 as of today's CVS, not earlier 8.5 CVS trees. One interesting aspect of pg_migrator is that while it has to carry around support for upgrading _from_ many old releases, the target/to version support can stay limited, i.e. I doubt anyone is going to want to use pg_migrator to migrate to 8.4 in a year or two --- they will be migrating to 8.5. We can also consider removing 8.4 target migration support in pg_migrator 8.5 and just tell people they have to use pg_migrator 8.4.X for a migration to PG 8.4.X. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Zdenek Kotala wrote: > Dne 4.01.10 19:28, Alvaro Herrera napsal(a): > > Bruce Momjian escribi?: > > > >> I considered that but realize that pg_migrator has to read > >> pg_controldata in both the old and new servers, meaning it would need > >> access to both C structures, and considering they both have the same > >> structure names, that would require some odd C tricks. Add to that you > >> don't know which version of Postgres you are migrating from/to during > >> compile and the idea of using C becomes even less attractive. > > > > However, keep in mind that this might not be the last time on which we > > will want to read something from a C struct, so perhaps it would be good > > to bite the bullet and write the odd tricks. Does it already have > > access (at compile time) to the old and new source trees? > > I have some proof of concept when each control data struct version > version have one header file like pg_control_843.h and structure like > ControlFileData has name ControlFileData_843. The main pg_control.h > defines types without version like > > typedef ControlFileData_843 ControlFileData; > > I planed to do it for 8.5 but unfortunately no time :( commit fest is > too close. Yea, I think the confusion of doing it isn't worth it. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Dne 4.01.10 19:28, Alvaro Herrera napsal(a): Bruce Momjian escribió: I considered that but realize that pg_migrator has to read pg_controldata in both the old and new servers, meaning it would need access to both C structures, and considering they both have the same structure names, that would require some odd C tricks. Add to that you don't know which version of Postgres you are migrating from/to during compile and the idea of using C becomes even less attractive. However, keep in mind that this might not be the last time on which we will want to read something from a C struct, so perhaps it would be good to bite the bullet and write the odd tricks. Does it already have access (at compile time) to the old and new source trees? I have some proof of concept when each control data struct version version have one header file like pg_control_843.h and structure like ControlFileData has name ControlFileData_843. The main pg_control.h defines types without version like typedef ControlFileData_843 ControlFileData; I planed to do it for 8.5 but unfortunately no time :( commit fest is too close. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
On Jan 6, 2010, at 1:52 AM, decibel wrote: > On Dec 30, 2009, at 9:50 PM, Bruce Momjian wrote: >> 3) There is no easy way to analyze all databases. vacuumdb --analyze >> does analyze _and_ vacuum, which for an 8.4 to 8.5 migration does an >> unnecessary vacuum. Right now I recommend ANALYZE in every database, >> but it would be nice if there were a single command which did this. > > I actually started on a patch for this > (http://lnk.nu/archives.postgresql.org/14rm.php). IIRC it's pretty close, I > just haven't had time to come back to it for final cleanup and changing the > docs as needed. Crap, I see I should have read the whole thread before posting. Sorry for the noise (and not getting the patch completed). :( -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
On Dec 30, 2009, at 9:50 PM, Bruce Momjian wrote: > 3) There is no easy way to analyze all databases. vacuumdb --analyze > does analyze _and_ vacuum, which for an 8.4 to 8.5 migration does an > unnecessary vacuum. Right now I recommend ANALYZE in every database, > but it would be nice if there were a single command which did this. I actually started on a patch for this (http://lnk.nu/archives.postgresql.org/14rm.php). IIRC it's pretty close, I just haven't had time to come back to it for final cleanup and changing the docs as needed. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
