Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-07 Thread Bruce Momjian
Bruce Momjian wrote: > Tom Lane wrote: > > hubert depesz lubaczewski writes: > > > Worked a bit to get the ltree problem down to smallest possible, > > > repeatable, situation. > > > > I looked at this again and verified that indeed, commit > > 8eee65c996048848c20f6637c1d12b319a4ce244 introduced

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-07 Thread hubert depesz lubaczewski
On Tue, Sep 06, 2011 at 09:21:02PM -0400, Bruce Momjian wrote: > Tom Lane wrote: > > hubert depesz lubaczewski writes: > > > Worked a bit to get the ltree problem down to smallest possible, > > > repeatable, situation. > > > > I looked at this again and verified that indeed, commit > > 8eee65c99

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread Bruce Momjian
Tom Lane wrote: > hubert depesz lubaczewski writes: > > Worked a bit to get the ltree problem down to smallest possible, > > repeatable, situation. > > I looked at this again and verified that indeed, commit > 8eee65c996048848c20f6637c1d12b319a4ce244 introduced an incompatible > change into the

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread Tom Lane
hubert depesz lubaczewski writes: > Worked a bit to get the ltree problem down to smallest possible, repeatable, > situation. I looked at this again and verified that indeed, commit 8eee65c996048848c20f6637c1d12b319a4ce244 introduced an incompatible change into the on-disk format of ltree column

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread hubert depesz lubaczewski
Hi, Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. Initial setup: 1. PostgreSQL 8.3.11, configured with: ./configure\ --prefix=/opt/pgsql-8.3.11-int \ --disable-rpath\ --without-perl

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 05:26:00PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > Odd it is dying in the memory freeing at executor close --- not in the > > ltree code. > > Doesn't seem odd. The glibc complaint previously shown already > indicates this is a memory stomp problem. > > --enabl

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Peter Eisentraut
On mån, 2011-09-05 at 16:53 -0400, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > > Good. Is it possible to compile with debug symbols, -g? Odd you are > > > crashing in libc. > > > > this had debug: > > > > ./configure \ > > --prefix=/opt/pgsql-9.0.5a-int \ > > --e

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
daveg wrote: > > > As far as I can tell pg_upgrade never copied any pg_clog files from the > > > old cluster to the new cluster. I wish I had detected that before running > > > the remove_old_cluster.sh script. > > > > Wow, no clogs? That would make the system very confused. You can pull > > the

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread daveg
On Mon, Sep 05, 2011 at 08:19:21PM -0400, Bruce Momjian wrote: > daveg wrote: > > > Can you tell me what table is showing this error? Does it happen during > > > vacuum? Can you run a vacuum verbose to see what it is throwing the > > > error on? Thanks. > > > > This was upgrading from 8.4.8 to

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
daveg wrote: > > Can you tell me what table is showing this error? Does it happen during > > vacuum? Can you run a vacuum verbose to see what it is throwing the > > error on? Thanks. > > This was upgrading from 8.4.8 to 9.0.4. I don't have the running cluster > anymore, but I do have tar.gz arc

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread daveg
Sorry I missed your reply, catching up now. On Wed, Aug 31, 2011 at 09:56:59PM -0400, Bruce Momjian wrote: > daveg wrote: > > On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: > > > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > > > vacuumdb:

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Tom Lane
Bruce Momjian writes: > Odd it is dying in the memory freeing at executor close --- not in the > ltree code. Doesn't seem odd. The glibc complaint previously shown already indicates this is a memory stomp problem. --enable-cassert might (or might not) provide additional help.

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: > > hubert depesz lubaczewski wrote: > > > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > > > > hubert depesz lubaczewski wrote: > > > > > I'm not sure if it's upgrade thing, or is it

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > > Good. Is it possible to compile with debug symbols, -g? Odd you are > > crashing in libc. > > this had debug: > > ./configure \ > --prefix=/opt/pgsql-9.0.5a-int \ > --enable-debug \ > --disable-rpath \ > --without-perl \ >

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 04:43:47PM -0400, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: > > > hubert depesz lubaczewski wrote: > > > > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > > > > > hubert depesz

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: > > hubert depesz lubaczewski wrote: > > > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > > > > hubert depesz lubaczewski wrote: > > > > > I'm not sure if it's upgrade thing, or is it

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > > > hubert depesz lubaczewski wrote: > > > > I'm not sure if it's upgrade thing, or is it because of error in > > > > ltree/compilat

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Tom Lane
hubert depesz lubaczewski writes: > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: >> If I had to take a guess, it would be that there is some ltree >> incompatibility from PG 8.3 that we didn't know about. > it's possible. [ checks the git history... ] This 8.4 commit: http://g

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > > hubert depesz lubaczewski wrote: > > > I'm not sure if it's upgrade thing, or is it because of error in > > > ltree/compilation, but it looks bad. > > > > > > Is there any more info I could show/g

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > I'm not sure if it's upgrade thing, or is it because of error in > > ltree/compilation, but it looks bad. > > > > Is there any more info I could show/gather to help debug the issue? > > I am conf

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > I'm not sure if it's upgrade thing, or is it because of error in > ltree/compilation, but it looks bad. > > Is there any more info I could show/gather to help debug the issue? I am confused by the error --- is it not loading, or can you get a backtrace of the cr

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: > > > Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap > > > tables involved? > > > > Sure: > > > > =# select oid::regclass, relfrozenxid from pg_class

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 05:48:50PM +0200, hubert depesz lubaczewski wrote: > On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: > > On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: > > > Working with depesz, I have found the cause. The code I added to fix > >

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: > > On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: > > > Working with depesz, I have found the cause. The code I added to fix > > > pg_upgrade in 9.0.4 and earlier releases

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: > On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: > > Working with depesz, I have found the cause. The code I added to fix > > pg_upgrade in 9.0.4 and earlier releases didn't handle old 8.3 servers > > proper

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: > Working with depesz, I have found the cause. The code I added to fix > pg_upgrade in 9.0.4 and earlier releases didn't handle old 8.3 servers > properly. I mistakenly processed toast table with the same pg_dump > query as used for p

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
daveg wrote: > On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: > > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > > vacuumdb: vacuuming of database "etsy_v2" failed: ERROR: could not access > > status of transaction 3429738606 > > DETAIL:

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: > > Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap > > tables involved? > > Sure: > > =# select oid::regclass, relfrozenxid from pg_class where relname in > ('transactions',

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
FYI, I am working with depesz on IM right now and will report back when we have a cause of the bug. FYI, I was without electric power for 53 hours, which is why I am late in replying to this report. --- daveg wrote: > On Mo

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: > Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap > tables involved? Sure: =# select oid::regclass, relfrozenxid from pg_class where relname in ('transactions', 'pg_toast_106668498'); oid

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
Alvaro Herrera wrote: > > > I don't understand the pg_upgrade code here. It is setting the > > > datfrozenxid and relfrozenxid values to the latest checkpoint's NextXID, > > > > > > /* set pg_class.relfrozenxid */ > > > PQclear(executeQueryOrDie(conn, > > >

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of mié ago 31 13:23:07 -0300 2011: > Alvaro Herrera wrote: > > Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 > > -0300 2011: > > > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > > > > On Fri, Aug 26, 201

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Wed, Aug 31, 2011 at 12:16:03PM -0400, Bruce Momjian wrote: > > hubert depesz lubaczewski wrote: > > > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > > > > > > > OK, this was very helpful. I found out that there is a bug in current > > > >

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > INFO: vacuuming "pg_toast.pg_toast_106668498" > vacuumdb: vacuuming of database "etsy_v2" failed: ERROR: could not access > status of transaction 3429738606 > DETAIL: Could not open file "pg_clog/0CC6": No such file or directory. > > Interestingly. > > In ol

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 12:16:03PM -0400, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: > > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > > > > > OK, this was very helpful. I found out that there is a bug in current > > > 9.0.X, 9.1.X, and HEAD that I introduced rec

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Robert Haas
On Wed, Aug 31, 2011 at 12:16 PM, Bruce Momjian wrote: > hubert depesz lubaczewski wrote: >> On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: >> > >> > OK, this was very helpful.  I found out that there is a bug in current >> > 9.0.X, 9.1.X, and HEAD that I introduced recently when I

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
Alvaro Herrera wrote: > Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 > -0300 2011: > > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > > > On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: > > > > On Fri, Aug 26, 201

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > > > OK, this was very helpful. I found out that there is a bug in current > > 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp > > tables. (The bug is not in any released v

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread daveg
On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > vacuumdb: vacuuming of database "etsy_v2" failed: ERROR: could not access > status of transaction 3429738606 > DETAIL: Could not open file "pg

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread Alvaro Herrera
Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 -0300 2011: > On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > > On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: > > > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Mom

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread hubert depesz lubaczewski
On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: > On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: > > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > > > > > OK, this was very helpful. I found out that there is a bug in curr

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread hubert depesz lubaczewski
On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: > On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > > > OK, this was very helpful. I found out that there is a bug in current > > 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp > >

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-26 Thread hubert depesz lubaczewski
On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: > > OK, this was very helpful. I found out that there is a bug in current > 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp > tables. (The bug is not in any released version of pg_upgrade.) The > attached, app

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp tables. (The bug is not in any released version of pg_upgrade.) The attached, applied patches should fix it for you. I assume you are running 9.0.X, and

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread hubert depesz lubaczewski
On Thu, Aug 25, 2011 at 04:43:02PM -0400, Bruce Momjian wrote: > Please check the old cluster. Sure: =# SELECT reltoastrelid FROM pg_class WHERE relname = 'actions';

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > On Thu, Aug 25, 2011 at 04:33:07PM -0400, Bruce Momjian wrote: > > The problem appears to be that the Postgres catalogs think there is a > > toast table for 'actions', while the file system doesn't seem to have > > such a file. I can you look in pg_class and veri

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread hubert depesz lubaczewski
On Thu, Aug 25, 2011 at 04:33:07PM -0400, Bruce Momjian wrote: > The problem appears to be that the Postgres catalogs think there is a > toast table for 'actions', while the file system doesn't seem to have > such a file. I can you look in pg_class and verify that? > > SELECT reltoastrelid

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
hubert depesz lubaczewski wrote: > hi > > I have 8.3.11 database, ~ 600GB in size. > > I want to upgrade it to 9.0. > > First, I tried with 9.0.4, and when I hit problem (the same) I tried > git, head of 9.0 branch. Good. > pg_upgrade_dump_db.sql- > pg_upgrade_dump_db.sql--- For binary upgrade