Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-04-29 Thread Andres Freund
On 2015-04-28 10:26:54 +0530, Abhijit Menon-Sen wrote: Hi Andres. Do you intend to commit these patches? (I just verified that they apply without trouble to HEAD.) I intend to come back to them, yes. I'm not fully sure whether we should really apply them to the back branches. Greetings,

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-04-27 Thread Abhijit Menon-Sen
At 2015-01-30 21:36:42 +0100, and...@2ndquadrant.com wrote: Here's an alternative approach. I think it generally is superior and going in the right direction, but I'm not sure it's backpatchable. It basically consists out of: 1) Make GetLockConflicts() actually work. already

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-30 Thread Andres Freund
On 2015-01-27 20:16:43 +0100, Andres Freund wrote: Here's an alternative approach. I think it generally is superior and going in the right direction, but I'm not sure it's backpatchable. It basically consists out of: 1) Make GetLockConflicts() actually work. already commited as being a

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Andres Freund
On 2015-01-27 07:16:27 -0500, Robert Haas wrote: On Mon, Jan 26, 2015 at 4:03 PM, Andres Freund and...@2ndquadrant.com wrote: I basically have two ideas to fix this. 1) Make do_pg_start_backup() acquire a SHARE lock on pg_database. That'll prevent it from starting while a movedb() is

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Robert Haas
On Tue, Jan 27, 2015 at 2:16 PM, Andres Freund and...@2ndquadrant.com wrote: That'd end up essentially being a re-emulation of locks. Don't find that all that pretty. But maybe we have to go there. The advantage of it is that it is simple. The only thing we're really giving up is the deadlock

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Andres Freund
On 2015-01-27 19:24:24 -0500, Robert Haas wrote: On Tue, Jan 27, 2015 at 2:16 PM, Andres Freund and...@2ndquadrant.com wrote: That'd end up essentially being a re-emulation of locks. Don't find that all that pretty. But maybe we have to go there. The advantage of it is that it is simple.

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-27 Thread Robert Haas
On Mon, Jan 26, 2015 at 4:03 PM, Andres Freund and...@2ndquadrant.com wrote: I basically have two ideas to fix this. 1) Make do_pg_start_backup() acquire a SHARE lock on pg_database. That'll prevent it from starting while a movedb() is still in progress. Then additionally add

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
Hi, On 2015-01-22 19:56:07 +0100, Andres Freund wrote: Hi, On 2015-01-20 16:28:19 +0100, Andres Freund wrote: I'm analyzing a problem in which a customer had a pg_basebackup (from standby) created 9.2 cluster that failed with WAL contains references to invalid pages. The failed record

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
On 2015-01-26 22:03:03 +0100, Andres Freund wrote: Attached is a patch trying to this. Doesn't look too bad and lead me to discover missing recovery conflicts during a AD ST. But: It doesn't actually work on standbys, because lock.c prevents any stronger lock than RowExclusive from being

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-26 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: Because the way it currently works is a major crock. It's more luck than anything that it actually somewhat works. We normally rely on WAL to bring us into a consistent state. But around CREATE/MOVE/DROP DATABASE we've ignored that. Hah:

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 12:52:03 -0600, Jim Nasby wrote: On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't think ALTER DATABASE is even intentionally run at

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that that actually saves overall IO, even including the

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 13:01:34 -0600, Jim Nasby wrote: On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't think ALTER DATABASE is even intentionally run at the time of a base backup. I agree with

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
Hi, On 2015-01-20 16:28:19 +0100, Andres Freund wrote: I'm analyzing a problem in which a customer had a pg_basebackup (from standby) created 9.2 cluster that failed with WAL contains references to invalid pages. The failed record was a xlog redo visible i.e. XLOG_HEAP2_VISIBLE. First I

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Stephen Frost
Andres, * Andres Freund (and...@2ndquadrant.com) wrote: 1) Make do_pg_start_backup() acquire a SHARE lock on pg_database. That'll prevent it from starting while a movedb() is still in progress. Then additionally add pg_backup_in_progress() function to xlog.c that checks

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: Andres, * Andres Freund (and...@2ndquadrant.com) wrote: 1) Make do_pg_start_backup() acquire a SHARE lock on pg_database. That'll prevent it from starting while a movedb() is still in progress. Then additionally add

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Alvaro Herrera
Andres Freund wrote: 2) Make movedb() (and possibly created(), not sure yet) use proper WAL logging and log the whole copied data. I think this is the right long term fix and would end up being much more reliable. But it either requires some uglyness during redo (creating nonexistant

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Jim Nasby
On 1/22/15 1:43 PM, Alvaro Herrera wrote: Andres Freund wrote: 2) Make movedb() (and possibly created(), not sure yet) use proper WAL logging and log the whole copied data. I think this is the right long term fix and would end up being much more reliable. But it either requires

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-22 Thread Andres Freund
On 2015-01-22 14:42:18 -0600, Jim Nasby wrote: On 1/22/15 1:43 PM, Alvaro Herrera wrote: Andres Freund wrote: 2) Make movedb() (and possibly created(), not sure yet) use proper WAL logging and log the whole copied data. I think this is the right long term fix and would end up being

[HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-20 Thread Andres Freund
Hi, I'm analyzing a problem in which a customer had a pg_basebackup (from standby) created 9.2 cluster that failed with WAL contains references to invalid pages. The failed record was a xlog redo visible i.e. XLOG_HEAP2_VISIBLE. First I thought there might be another bug along the line of