Re: [HACKERS] foreign key locks, 2nd attempt

2012-04-05 Thread Peter Geoghegan
On 25 March 2012 09:17, Simon Riggs wrote: > The main thing we're waiting on are the performance tests to confirm > the lack of regression. I have extensively benchmarked the latest revision of the patch (tpc-b.sql), which I pulled from Alvaro's github account. The benchmark was of the feature br

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-25 Thread Simon Riggs
On Sat, Mar 17, 2012 at 10:45 PM, Alvaro Herrera wrote: > Here is v11.  This version is mainly updated to add pg_upgrade support, > as discussed.  It also contains the README file that was posted earlier > (plus wording fixes per Bruce), a couple of bug fixes, and some comment > updates. The mai

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of mar mar 06 17:28:12 -0300 2012: > On Tue, Mar 6, 2012 at 7:39 PM, Alvaro Herrera > wrote: > > > We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is > > super-exclusive locking (used to delete tuples and more generally to update > > tu

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of mar mar 06 18:33:13 -0300 2012: > The lock modes are correct, appropriate and IMHO have meaningful > names. No redesign required here. > > Not sure about the naming of some of the flag bits however. Feel free to suggest improvements ... I've probably seen

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: > > On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas wrote: > >> Regarding performance, the good thing about this patch is that if you > >> have an operation that used to block, it might now not block.  So maybe > >> multixact-r

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of vie mar 16 15:22:05 -0300 2012: > > On Fri, Mar 16, 2012 at 10:08:07AM -0400, Robert Haas wrote: > > On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian wrote: > > > On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: > > >> On Mon, Mar 12, 2012 at 3

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Bruce Momjian
On Fri, Mar 16, 2012 at 10:08:07AM -0400, Robert Haas wrote: > On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian wrote: > > On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: > >> On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs wrote: > >> > I agree with you that some worst case performance

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Bruce Momjian
On Fri, Mar 16, 2012 at 10:40:01AM -0300, Alvaro Herrera wrote: > > Excerpts from Alvaro Herrera's message of vie mar 16 10:36:11 -0300 2012: > > > > Now I am confused. Where do you see the word "hint" used by > > > HEAP_XMAX_EXCL_LOCK and HEAP_XMAX_SHARED_LOCK. These are tuple infomask > > > b

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Robert Haas
On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian wrote: > On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: >> On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs wrote: >> > I agree with you that some worst case performance tests should be >> > done. Could you please say what you think the wo

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of vie mar 16 10:36:11 -0300 2012: > > Now I am confused. Where do you see the word "hint" used by > > HEAP_XMAX_EXCL_LOCK and HEAP_XMAX_SHARED_LOCK. These are tuple infomask > > bits, not hints, meaning they are not optional or there just for > > performa

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of vie mar 16 00:04:06 -0300 2012: > > On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: > > > > Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: > > > > > > On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrot

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: > On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs wrote: > > I agree with you that some worst case performance tests should be > > done. Could you please say what you think the worst cases would be, so > > those can be tested? That would av

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Thu, Mar 15, 2012 at 11:04:06PM -0400, Bruce Momjian wrote: > On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: > > > > Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: > > > > > > On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: > > > > >

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: > > Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: > > > > On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: > > > > When there is a single locker in a tuple, we can just store the locking >

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Noah Misch
On Thu, Mar 15, 2012 at 08:37:36PM -0400, Robert Haas wrote: > On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs wrote: > > On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas wrote: > >>> You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on > >>> mxid > >>> lookups, so I think something m

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Robert Haas's message of jue mar 15 21:37:36 -0300 2012: > > On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs wrote: > > On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas wrote: > >>> You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on > >>> mxid > >>> lookups, so I thi

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Robert Haas
On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs wrote: > On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas wrote: >>> You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on mxid >>> lookups, so I think something more sophisticated is needed to exercise that >>> cost.  Not sure what. >> >>

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 10:13 PM, Alvaro Herrera wrote: > > Excerpts from Simon Riggs's message of jue mar 15 19:04:41 -0300 2012: >> >> On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera >> wrote: >> > >> > Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: >> >> On Thu, Mar 15,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 19:04:41 -0300 2012: > > On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera > wrote: > > > > Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: > >> On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas wrote: > > > >> > But that would o

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera wrote: > > Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: >> On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas wrote: > >> > But that would only make sense if >> > we thought that getting rid of the fsyncs would be more valuable t

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 18:46:44 -0300 2012: > > On Thu, Mar 15, 2012 at 1:17 AM, Alvaro Herrera > wrote: > > > As things stand today > > Can I confirm where we are now? Is there another version of the patch > coming out soon? Yes, another version is coming soon. --

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: > On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas wrote: > > But that would only make sense if > > we thought that getting rid of the fsyncs would be more valuable than > > avoiding the blocking here, and I don't. > > You're ri

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 1:17 AM, Alvaro Herrera wrote: > As things stand today Can I confirm where we are now? Is there another version of the patch coming out soon? --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support, Training & Services -- Se

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas wrote: > On Wed, Mar 14, 2012 at 9:17 PM, Alvaro Herrera > wrote: >>> > Agreed.  But speaking of that, why exactly do we fsync the multixact SLRU >>> > today? >>> >>> Good question.  So far, I can't think of a reason.  "nextMulti" is critical, >>> but

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 2:15 AM, Robert Haas wrote: > On Wed, Mar 14, 2012 at 6:10 PM, Noah Misch wrote: >>> Well, post-release, the cat is out of the bag: we'll be stuck with >>> this whether the performance characteristics are acceptable or not. >>> That's why we'd better be as sure as possible

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas wrote: >> You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on mxid >> lookups, so I think something more sophisticated is needed to exercise that >> cost.  Not sure what. > > I don't think HEAP_XMAX_COMMITTED is much help, because co

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 9:17 PM, Alvaro Herrera wrote: >> > Agreed.  But speaking of that, why exactly do we fsync the multixact SLRU >> > today? >> >> Good question.  So far, I can't think of a reason.  "nextMulti" is critical, >> but we already fsync it with pg_control.  We could delete the oth

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 6:10 PM, Noah Misch wrote: >> Well, post-release, the cat is out of the bag: we'll be stuck with >> this whether the performance characteristics are acceptable or not. >> That's why we'd better be as sure as possible before committing to >> this implementation that there's

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mié mar 14 19:10:00 -0300 2012: > > On Wed, Mar 14, 2012 at 01:23:14PM -0400, Robert Haas wrote: > > On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch wrote: > > > More often than that; each 2-member mxid takes 4 bytes in an offsets file > > > and > > > 10 bytes

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Noah Misch
On Wed, Mar 14, 2012 at 01:23:14PM -0400, Robert Haas wrote: > On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch wrote: > > More often than that; each 2-member mxid takes 4 bytes in an offsets file > > and > > 10 bytes in a members file. ?So, more like one fsync per ~580 mxids. ?Note > > that we alrea

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch wrote: > More often than that; each 2-member mxid takes 4 bytes in an offsets file and > 10 bytes in a members file.  So, more like one fsync per ~580 mxids.  Note > that we already fsync the multixact SLRUs today, so any increase will arise > from the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Noah Misch
On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: > On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs wrote: > > I agree with you that some worst case performance tests should be > > done. Could you please say what you think the worst cases would be, so > > those can be tested? That would av

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs wrote: > I agree with you that some worst case performance tests should be > done. Could you please say what you think the worst cases would be, so > those can be tested? That would avoid wasting time or getting anything > backwards. I've thought about

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: > > On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: > > When there is a single locker in a tuple, we can just store the locking info > > in the tuple itself. We do this by storing the locker's Xid in XMAX,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 9:24 PM, Noah Misch wrote: > When we lock an update-in-progress row, we walk the t_ctid chain and lock all > descendant tuples.  They may all have uncommitted xmins.  This is essential to > ensure that the final outcome of the updating transaction does not affect > whether

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: > Here's a first attempt at a README illustrating this. I intend this to > be placed in src/backend/access/heap/README.tuplock; the first three > paragraphs are stolen from the comment in heap_lock_tuple, so I'd remove > those from th

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Noah Misch
On Mon, Mar 12, 2012 at 01:28:11PM -0400, Robert Haas wrote: > I spent some time thinking about this over the weekend, and I have an > observation, and an idea. Here's the observation: I believe that > locking a tuple whose xmin is uncommitted is always a noop, because if > it's ever possible for

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Simon Riggs
On Mon, Mar 12, 2012 at 6:14 PM, Robert Haas wrote: > Considering that nobody's done any work to resolve the uncertainty > about whether the worst-case performance characteristics of this patch > are acceptable, and considering further that it was undergoing massive > code churn for more than a m

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 1:50 PM, Simon Riggs wrote: > On Mon, Mar 12, 2012 at 5:28 PM, Robert Haas wrote: >> In other words, we'd entirely avoid needing to make mxacts crash-safe, >> and we'd avoid most of the extra SLRU lookups that the current >> implementation requires; they'd only be needed w

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Simon Riggs
On Mon, Mar 12, 2012 at 5:28 PM, Robert Haas wrote: > In other words, we'd entirely avoid needing to make mxacts crash-safe, > and we'd avoid most of the extra SLRU lookups that the current > implementation requires; they'd only be needed when (and for as long > as) the locked tuple was not yet a

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Robert Haas
On Sun, Feb 26, 2012 at 9:47 PM, Robert Haas wrote: >> Regarding performance, the good thing about this patch is that if you >> have an operation that used to block, it might now not block.  So maybe >> multixact-related operation is a bit slower than before, but if it >> allows you to continue op

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
n Wed, Mar 7, 2012 at 11:37 AM, Gokulakannan Somasundaram wrote: >> >> Please explain in detail your idea of how it will work. > So we will take some kind of lock, which will stop such a happening. ... > May be someone can come up with better ideas than this. With respect, I don't call this a de

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
> > > Please explain in detail your idea of how it will work. > > OK. I will try to explain the abstract idea, i have. a) Referential integrity gets violated, when there are referencing key values, not present in the referenced key values. We are maintaining the integrity by taking a Select for Sha

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 10:18 AM, Gokulakannan Somasundaram wrote: >> >> Insert, Update and Delete don't take locks they simply mark the tuples >> they change with an xid. Anybody else wanting to "wait on the lock" >> just waits on the xid. We do insert a lock row for each xid, but not >> one per r

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
> > > Insert, Update and Delete don't take locks they simply mark the tuples > they change with an xid. Anybody else wanting to "wait on the lock" > just waits on the xid. We do insert a lock row for each xid, but not > one per row changed. > I mean the foreign key checks here. They take a Select f

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 9:24 AM, Gokulakannan Somasundaram wrote: > I feel sad, that i followed this topic very late. But i still want to put > forward my views. > Have we thought on the lines of how Robert has implemented relation level > locks. In short it should go like this > > a) The locks for

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
I feel sad, that i followed this topic very late. But i still want to put forward my views. Have we thought on the lines of how Robert has implemented relation level locks. In short it should go like this a) The locks for enforcing Referential integrity should be taken only when the rarest of the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 4:27 PM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: >> >> Preliminary comment: >> >> This README is very helpful. > > Thanks.  I feel silly that I didn't write it earlier. > >> On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 9:10 PM, Robert Haas wrote: > Preliminary comment: > > This README is very helpful. > > On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera > wrote: >> We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is >> super-exclusive locking (used to delete tuples an

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: > > Preliminary comment: > > This README is very helpful. Thanks. I feel silly that I didn't write it earlier. > On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera > wrote: > > We provide four levels of tuple locking strength

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
Preliminary comment: This README is very helpful. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera wrote: > We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is > super-exclusive locking (used to delete tuples and more generally to update > tuples modifying the values of the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 7:39 PM, Alvaro Herrera wrote: > We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is > super-exclusive locking (used to delete tuples and more generally to update > tuples modifying the values of the columns that make up the key of the tuple); > SELEC

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Mon, Mar 5, 2012 at 8:35 PM, Simon Riggs wrote: >>> * Why do we need multixact to be persistent? Do we need every page of >>> multixact to be persistent, or just particular pages in certain >>> circumstances? >> >> Any page that contains at least one multi with an update as a member >> must pe

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: > It does however, illustrate my next review comment which is that the > comments and README items are sorely lacking here. It's quite hard to > see how it works, let along comment on major design decisions. It > would help my

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 7:53 PM, Alvaro Herrera wrote: >> My other comments so far are >> >> * some permutations commented out - no comments as to why >> Something of a fault with the isolation tester that it just shows >> output, there's no way to record expected output in the spec > > The reason

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: > On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera > wrote: > It does however, illustrate my next review comment which is that the > comments and README items are sorely lacking here. It's quite hard to > see how it works, let

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera wrote: > > Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: >> >> On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas wrote: > >> > From a performance standpoint, we really need to think not only about >> > the cases where the patch wi

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: > > On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas wrote: > > From a performance standpoint, we really need to think not only about > > the cases where the patch wins, but also, and maybe more importantly, > > the cases where

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas wrote: > On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera > wrote: >>> This >>> seems like a horrid mess that's going to be unsustainable both from a >>> complexity and a performance standpoint.  The only reason multixacts >>> were tolerable at all was

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Simon Riggs
On Tue, Feb 28, 2012 at 12:28 AM, Noah Misch wrote: > On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: >> On 23.02.2012 18:01, Alvaro Herrera wrote: >>> As far as complexity, yeah, it's a lot more complex now -- no question >>> about that. >> >> How about assigning a new, real,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Noah Misch
On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: > On 23.02.2012 18:01, Alvaro Herrera wrote: >> As far as complexity, yeah, it's a lot more complex now -- no question >> about that. > > How about assigning a new, real, transaction id, to represent the group > of transaction id

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Heikki Linnakangas
On 23.02.2012 18:01, Alvaro Herrera wrote: Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: Alvaro Herrera writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to store extended Xmax informat

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-26 Thread Robert Haas
On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera wrote: >> This >> seems like a horrid mess that's going to be unsustainable both from a >> complexity and a performance standpoint.  The only reason multixacts >> were tolerable at all was that they had only one semantics.  Changing >> it so that ma

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-25 Thread Kevin Grittner
Vik Reykja wrote: > Kevin Grittner wrote: > >> One of the problems that Florian was trying to address is that >> people often have a need to enforce something with a lot of >> similarity to a foreign key, but with more subtle logic than >> declarative foreign keys support. One example would be t

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Vik Reykja
On Thu, Feb 23, 2012 at 19:44, Kevin Grittner wrote: > One of the problems that Florian was trying to address is that > people often have a need to enforce something with a lot of > similarity to a foreign key, but with more subtle logic than > declarative foreign keys support. One example would

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Jeroen Vermeulen
On 2012-02-23 22:12, Noah Misch wrote: That alone would not simplify the patch much. INSERT/UPDATE/DELETE on the foreign side would still need to take some kind of tuple lock on the primary side to prevent primary-side DELETE. You then still face the complicated case of a tuple that's both loc

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 12:43:11PM -0300, Alvaro Herrera wrote: > Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: > > On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch wrote: > > > > >> All in all, I think this is in pretty much final shape. ??Only pg_upgrade > > >> bits are stil

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 06:36:42PM -0300, Alvaro Herrera wrote: > > Excerpts from Noah Misch's message of mi?? feb 22 14:00:07 -0300 2012: > > > > On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: > > > On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: > > > On Tue, Jan 2

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mié feb 22 14:00:07 -0300 2012: > > On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: > On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: > > On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: > > > * Columns that are par

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 02:08:28PM +0100, Jeroen Vermeulen wrote: > On 2012-02-23 10:18, Simon Riggs wrote: > >> However, review of such a large patch should not be simply pass or >> fail. We should be looking back at the original problem and ask >> ourselves whether some subset of the patch could

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera wrote: > Excerpts from Kevin Grittner's message: >> Since the limitation on what can be stored in xmax was the killer >> for Florian's attempt to support SELECT FOR UPDATE in a form >> which was arguably more useful (and certainly more convenient for >> those converting from other

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 4:01 PM, Alvaro Herrera wrote: > As far as complexity, yeah, it's a lot more complex now -- no question > about that. As far as complexity goes, would it be easier if we treated the UPDATE of a primary key column as a DELETE plus an INSERT? There's not really a logical r

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 01:04 PM, Alvaro Herrera wrote: "manual vacuum is teh sux0r" I think you've just named my next conference talk submission. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Greg Smith's message of jue feb 23 14:48:13 -0300 2012: > On 02/23/2012 10:43 AM, Alvaro Herrera wrote: > > I completely understand that you don't want to review this latest > > version of the patch; it's a lot of effort and I wouldn't inflict it on > > anybody who hasn't not volunte

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 10:43 AM, Alvaro Herrera wrote: I completely understand that you don't want to review this latest version of the patch; it's a lot of effort and I wouldn't inflict it on anybody who hasn't not volunteered. However, it doesn't seem to me that this is reason to boot the patch from th

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Kevin Grittner's message of jue feb 23 13:31:36 -0300 2012: > > Alvaro Herrera wrote: > > > As for sanity -- I regard multixacts as a way to store extended > > Xmax information. The original idea was obviously much more > > limited in that the extended info was just locking info

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera wrote: > As for sanity -- I regard multixacts as a way to store extended > Xmax information. The original idea was obviously much more > limited in that the extended info was just locking info. We've > extended it but I don't think it's such a stretch. Since the limitation on

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: > > Alvaro Herrera writes: > > Sure. The problem is that we are allowing updated rows to be locked (and > > locked rows to be updated). This means that we need to store extended > > Xmax information in tuples that goes beyond

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 12:12:13 -0300 2012: > On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera > wrote: > > Sure.  The problem is that we are allowing updated rows to be locked (and > > locked rows to be updated).  This means that we need to store extended > > Xmax info

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: > > On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch wrote: > > >> All in all, I think this is in pretty much final shape.  Only pg_upgrade > >> bits are still missing.  If sharp eyes could give this a critical look > >> and knuc

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Tom Lane
Alvaro Herrera writes: > Sure. The problem is that we are allowing updated rows to be locked (and > locked rows to be updated). This means that we need to store extended > Xmax information in tuples that goes beyond mere locks, which is what we > were doing previously -- they may now have locks

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera wrote: > > Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: >> On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch wrote: >> >> > Making pg_multixact persistent across clean shutdowns is no bridge to cross >> > lightly, since it means

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: > On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch wrote: > > > Making pg_multixact persistent across clean shutdowns is no bridge to cross > > lightly, since it means committing to an on-disk format for an indefinite > > period.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 1:08 PM, Jeroen Vermeulen wrote: > Simon, I think you had a reason why it couldn't work, but I didn't quite get > your meaning and didn't want to distract things further at that stage.  You > wrote that it "doesn't do what KEY LOCKS are designed to do"...  any chance > you

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch wrote: > Making pg_multixact persistent across clean shutdowns is no bridge to cross > lightly, since it means committing to an on-disk format for an indefinite > period.  We should do it; the benefits of this patch justify it, and I haven't > identifie

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Jeroen Vermeulen
On 2012-02-23 10:18, Simon Riggs wrote: However, review of such a large patch should not be simply pass or fail. We should be looking back at the original problem and ask ourselves whether some subset of the patch could solve a useful subset of the problem. For me, that seems quite likely and th

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch wrote: >> All in all, I think this is in pretty much final shape.  Only pg_upgrade >> bits are still missing.  If sharp eyes could give this a critical look >> and knuckle-cracking testers could give it a spin, that would be >> helpful. > > Lack of pg_u

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-22 Thread Noah Misch
On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: > Okay, so this patch fixes the truncation and wraparound issues through a > mechanism much like pg_clog's: it keeps track of the oldest possibly > existing multis on each and every table, and then during tuple freezing > those are rem

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Alvaro Herrera
Excerpts from Jim Nasby's message of mié feb 01 21:33:47 -0300 2012: > > On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: > >> I think it's butt-ugly, but it's only slightly uglier than > >> relfrozenxid which we're already stuck with. The slight amount of > >> additional ugliness is that you

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Jim Nasby
On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: >> I think it's butt-ugly, but it's only slightly uglier than >> relfrozenxid which we're already stuck with. The slight amount of >> additional ugliness is that you're going to use an XID column to store >> a uint4 that is not an XID - but I don

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 11:58 AM, Alvaro Herrera wrote: > Well, we're already storing a multixact in Xmax, so it's not like we > don't assume that we can store multis in space normally reserved for > Xids.  What I've been wondering is not how ugly it is, but rather of the > fact that we're bloatin

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 13:18:30 -0300 2012: > > On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera > wrote: > > Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: > >> I suspect you are right that it is unlikely, but OTOH that sounds like > >> an extre

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: >> I suspect you are right that it is unlikely, but OTOH that sounds like >> an extremely painful recovery procedure.  We probably don't need to >> put a ton of thought i

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: > I suspect you are right that it is unlikely, but OTOH that sounds like > an extremely painful recovery procedure. We probably don't need to > put a ton of thought into handling this case as efficiently as > possible, but I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Mon, Jan 30, 2012 at 6:48 PM, Noah Misch wrote: > On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: >> The biggest item remaining is the point you raised about multixactid >> wraparound.  This is closely related to multixact truncation and the way >> checkpoints are to be handled.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-30 Thread Noah Misch
On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: > The biggest item remaining is the point you raised about multixactid > wraparound. This is closely related to multixact truncation and the way > checkpoints are to be handled. If we think that MultiXactId wraparound > is possible,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-26 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar ene 24 15:47:16 -0300 2012: > Need more code changes for the following: > * export FOR KEY UPDATE lock mode in SQL While doing this, I realized that there's an open item here regarding a transaction that locks a tuple, and then in an aborted savepoi

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Noah Misch
On Wed, Jan 18, 2012 at 05:18:31PM -0300, Alvaro Herrera wrote: > Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: > > On 16.01.2012 21:52, Alvaro Herrera wrote: > > > I was initially thinking that pg_multixact should return the > > > empty set if requested members of

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Alvaro Herrera
Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: > > On 16.01.2012 21:52, Alvaro Herrera wrote: > > > > Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 > > 2012: > >> > >> On 15.01.2012 06:49, Alvaro Herrera wrote: > >>> - pg_upgrade bits are

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-17 Thread Noah Misch
On Mon, Jan 16, 2012 at 04:52:36PM -0300, Alvaro Herrera wrote: > Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: > > On 15.01.2012 06:49, Alvaro Herrera wrote: > > > --- 164,178 > > > #define HEAP_HASVARWIDTH0x0002/* has variable-width > > > attrib

  1   2   >