Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-04-29 Thread Andres Freund
On 2015-04-28 10:26:54 +0530, Abhijit Menon-Sen wrote: > Hi Andres. Do you intend to commit these patches? (I just verified that > they apply without trouble to HEAD.) I intend to come back to them, yes. I'm not fully sure whether we should really apply them to the back branches. Greetings, Andr

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-04-27 Thread Abhijit Menon-Sen
At 2015-01-30 21:36:42 +0100, and...@2ndquadrant.com wrote: > > > Here's an alternative approach. I think it generally is superior and > > going in the right direction, but I'm not sure it's backpatchable. > > > > It basically consists out of: > > 1) Make GetLockConflicts() actually work. > > alr

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-30 Thread Andres Freund
On 2015-01-27 20:16:43 +0100, Andres Freund wrote: > Here's an alternative approach. I think it generally is superior and > going in the right direction, but I'm not sure it's backpatchable. > > It basically consists out of: > 1) Make GetLockConflicts() actually work. already commited as being a

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Andres Freund
On 2015-01-27 19:24:24 -0500, Robert Haas wrote: > On Tue, Jan 27, 2015 at 2:16 PM, Andres Freund wrote: > > That'd end up essentially being a re-emulation of locks. Don't find that > > all that pretty. But maybe we have to go there. > > The advantage of it is that it is simple. The only thing w

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Robert Haas
On Tue, Jan 27, 2015 at 2:16 PM, Andres Freund wrote: > That'd end up essentially being a re-emulation of locks. Don't find that > all that pretty. But maybe we have to go there. The advantage of it is that it is simple. The only thing we're really giving up is the deadlock detector, which I thi

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Andres Freund
On 2015-01-27 07:16:27 -0500, Robert Haas wrote: > On Mon, Jan 26, 2015 at 4:03 PM, Andres Freund wrote: > >> I basically have two ideas to fix this. > >> > >> 1) Make do_pg_start_backup() acquire a SHARE lock on > >>pg_database. That'll prevent it from starting while a movedb() is > >>sti

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Robert Haas
On Mon, Jan 26, 2015 at 4:03 PM, Andres Freund wrote: >> I basically have two ideas to fix this. >> >> 1) Make do_pg_start_backup() acquire a SHARE lock on >>pg_database. That'll prevent it from starting while a movedb() is >>still in progress. Then additionally add pg_backup_in_progress()

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
On 2015-01-26 22:03:03 +0100, Andres Freund wrote: > Attached is a patch trying to this. Doesn't look too bad and lead me to > discover missing recovery conflicts during a AD ST. > > But: It doesn't actually work on standbys, because lock.c prevents any > stronger lock than RowExclusive from being

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
Hi, On 2015-01-22 19:56:07 +0100, Andres Freund wrote: > Hi, > > On 2015-01-20 16:28:19 +0100, Andres Freund wrote: > > I'm analyzing a problem in which a customer had a pg_basebackup (from > > standby) created 9.2 cluster that failed with "WAL contains references to > > invalid pages". The faile

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: > Because the way it currently works is a major crock. It's more luck than > anything that it actually somewhat works. We normally rely on WAL to > bring us into a consistent state. But around CREATE/MOVE/DROP DATABASE > we've ignored that. Hah: T

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 13:01:34 -0600, Jim Nasby wrote: > On 1/23/15 9:10 AM, Andres Freund wrote: > >On 2015-01-22 22:58:17 +0100, Andres Freund wrote: > >>On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: > >>>I'm trying to figure out why you'd do '2' in master when in discussion > >>>of '1' you say "I a

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 12:52:03 -0600, Jim Nasby wrote: > On 1/22/15 3:18 PM, Andres Freund wrote: > >>, but to then put it's entire contents into WAL? Blech. > >Besides actually having a chance of being correct, doing so will save > >having to do two checkpoints inside movedb(). I think it's pretty likely

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say "I also don't think ALTER DATABASE is even intentionally run at th

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that that actually saves overall IO, even including the

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: > On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: > > I'm trying to figure out why you'd do '2' in master when in discussion > > of '1' you say "I also don't think ALTER DATABASE is even intentionally > > run at the time of a base backup." I ag

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@2ndquadrant.com) wrote: > > 1) Make do_pg_start_backup() acquire a SHARE lock on > >pg_database. That'll prevent it from starting while a movedb() is > >still in progress. Then additionally add pg_back

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Stephen Frost
Andres, * Andres Freund (and...@2ndquadrant.com) wrote: > 1) Make do_pg_start_backup() acquire a SHARE lock on >pg_database. That'll prevent it from starting while a movedb() is >still in progress. Then additionally add pg_backup_in_progress() >function to xlog.c that checks (XLogCtl->

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
On 2015-01-22 14:42:18 -0600, Jim Nasby wrote: > On 1/22/15 1:43 PM, Alvaro Herrera wrote: > >Andres Freund wrote: > > > >>2) Make movedb() (and possibly created(), not sure yet) use proper WAL > >>logging and log the whole copied data. I think this is the right long > >>term fix and would

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Jim Nasby
On 1/22/15 1:43 PM, Alvaro Herrera wrote: Andres Freund wrote: 2) Make movedb() (and possibly created(), not sure yet) use proper WAL logging and log the whole copied data. I think this is the right long term fix and would end up being much more reliable. But it either requires some

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Alvaro Herrera
Andres Freund wrote: > 2) Make movedb() (and possibly created(), not sure yet) use proper WAL >logging and log the whole copied data. I think this is the right long >term fix and would end up being much more reliable. But it either >requires some uglyness during redo (creating nonexist

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
Hi, On 2015-01-20 16:28:19 +0100, Andres Freund wrote: > I'm analyzing a problem in which a customer had a pg_basebackup (from > standby) created 9.2 cluster that failed with "WAL contains references to > invalid pages". The failed record was a "xlog redo visible" > i.e. XLOG_HEAP2_VISIBLE. > > Fi