Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
"Kevin Grittner" wrote: > This is the milestone of having full serializable behavior, albeit > with horrible performance, using the simplest implementation > possible. A tad too simple, as it turns out. It did it's main job of providing a simple milestone to identify code organization and loc

Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
"Kevin Grittner" wrote: > Here you go! :-) Hmmm... I just got a write skew example to show a snapshot isolation anomaly, so I've got something wrong still. :-( Will continue to work on it. Sorry. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
Robert Haas wrote: > Nope, you're on target. Although - if I were you - I would post > the ACCESS EXCLUSIVE lock version of the patch for feedback. I > can't speak for anyone else, but I'll read it. Here you go! :-) This is the milestone of having full serializable behavior, albeit with h

Re: [HACKERS] Serializable Isolation without blocking

2010-01-09 Thread Nicolas Barbier
2010/1/8 Kevin Grittner : > Nicolas Barbier wrote: > >> I assume here that PG's non-SI-compatible behavior of not always >> rollbacking any transaction that writes to a non-last version will >> be disabled in serializable mode. > > Can that currently happen in PostgreSQL's snapshot isolation?!?  

Re: [HACKERS] Serializable Isolation without blocking

2010-01-09 Thread Markus Wanner
Hi, Kevin Grittner wrote: > Nicolas Barbier wrote: >> AFAICS, detecting a "rw" dependency where the read executes after >> the write is rather easy: the writer has created a row version >> that is not visible to the reader's snapshot. Therefore, any time >> a reader reads a non-last version of a

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Again, a somewhat tardy post from a question found in review. Markus Wanner wrote: >> I suppose these more persistent write locks should >> be kept out of the DEFAULT lock method, too > > I fail to understand that part. What's the DEFAULT lock method? With some adjustment of line wrappin

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Nicolas Barbier wrote: > AFAICS, detecting a "rw" dependency where the read executes after > the write is rather easy: the writer has created a row version > that is not visible to the reader's snapshot. Therefore, any time > a reader reads a non-last version of a row, there is a rw > dependency

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Nicolas Barbier
2010/1/8 Markus Wanner : > SIREAD atop predicate locking serves detecting vulnerable edges (I hope > I'm using that term correctly here) between newly inserted tuples and > reads, right? I was trying to figure if it would make sense to use > predicate locking (instead of table or row level locking

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner wrote: > I don't remember reading about predicate locking in the paper I > read. Either he didn't cover that in his first implementation > (based on page level locking), or I've simply re-used that part of > my in-brain-memory. If you read the first paper but not the second, all

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: > I had this flagged as needing a response, but it fell through the > cracks yesterday. Apologies for the delayed response. No problem. > Markus Wanner wrote: > When the Cahill paper talks about predicate locking, it *is* talking > about what to lock with SIREAD locks

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: > SIREAD locks need to be acquired according to the exact same rules > as "normal" read locks in predicate locking schemes. Understood. I didn't take that into account at first. Thanks for pointing it out. (Whatever "normal" read locks are...) > We're just > using a loc

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Greg Stark wrote: > well the one place you *cannot* attach them is on the tuples. because > you need to new able to lock hypothetical new tuples which don't exist > yet. Well, maybe "attaching" here is meant in a more general or theoretical sense. I think we all agree that adding them to the

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
I had this flagged as needing a response, but it fell through the cracks yesterday. Apologies for the delayed response. Markus Wanner wrote: > I'm not clear if Kevin plans to go down to tuple level locking > with granularity of the SIREAD thing. Eventually, where possible, subject to escala

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Greg Stark wrote: > well the one place you *cannot* attach them is on the tuples. The predicate locking schemes I've been reading about do attach locks to tuples, as *part* of a complete strategy. > you need to new able to lock hypothetical new tuples which don't > exist yet. That, too. W

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner wrote: > Kevin Grittner wrote: >> As I understand it, Greg's line of thinking is that we should use >> a technique which has never proven practical on a large scale: >> matching database changes against a list of predicate lock >> expressions. > > I find that approach to predicate l

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner wrote: > Greg Stark wrote: > That's about predicate locks. I've been talking about SIREAD, > which is a different thing (and which I don't consider to be a > lock). The SIREAD thingie certainly doesn't help implement > predicate locks. And predicate locking isn't necessarily > suff

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Robert Haas
On Fri, Jan 8, 2010 at 7:34 AM, Greg Stark wrote: > On Thursday, January 7, 2010, Robert Haas wrote: >> On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark wrote: >>> On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner wrote: Row level locks are very fine grained, but those are spilled to disk in i

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Greg Stark
On Thursday, January 7, 2010, Robert Haas wrote: > On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark wrote: >> On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner wrote: >>> Row level locks are very fine grained, but those are spilled to disk in >>> its current implementation. So those are an even worse fit

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: > As I understand it, Greg's line of thinking is that we should use a > technique which has never proven practical on a large scale: > matching database changes against a list of predicate lock > expressions. I find that approach to predicate locking pretty interesting.

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Greg Stark wrote: > I think we're still talking past the issue. Predicate locks are not > row level, nor page level, nor table level. They're locks on > predicates. Ie, you have to lock against values which aren't actually > currently in the table at all. You need to be able to detect a > conf

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Jeff Davis
On Thu, 2010-01-07 at 15:01 -0600, Kevin Grittner wrote: > As I understand it, Greg's line of thinking is that we should use a > technique which has never proven practical on a large scale: > matching database changes against a list of predicate lock > expressions. It's not that I want to do it an

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Kevin Grittner
Jeff Davis wrote: > Consider an exclusion constraint, which is a kind of predicate > lock. You could say that the lock is in the index (now) -- but my > first implementation used a shared memory structure instead, so > it's clearly not required to exist in the index. You could also > say that th

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Jeff Davis
On Thu, 2010-01-07 at 15:02 -0500, Robert Haas wrote: > > I think we're still talking past the issue. Predicate locks are not > > row level, nor page level, nor table level. > > They're not? They're just floating out in space somewhere? There are > several possible ways to implement predicate lo

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Robert Haas
On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark wrote: > On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner wrote: >> Row level locks are very fine grained, but those are spilled to disk in >> its current implementation. So those are an even worse fit for the needs >> of SIREAD. > > I think we're still ta

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Greg Stark
On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner wrote: > Row level locks are very fine grained, but those are spilled to disk in > its current implementation. So those are an even worse fit for the needs > of SIREAD. > I think we're still talking past the issue. Predicate locks are not row level,

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Magnus Hagander
On Thu, Jan 7, 2010 at 19:08, Kevin Grittner wrote: > "Kevin Grittner" wrote: >> Robert Haas wrote: >> >>> I think you should have users/kgrittner/postgres.git rather than >>> serializable.git.  serializable sounds more like the branch name. >> >> I'll wait a bit for other comments before taking

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Kevin Grittner
"Kevin Grittner" wrote: > Robert Haas wrote: > >> I think you should have users/kgrittner/postgres.git rather than >> serializable.git. serializable sounds more like the branch name. > > I'll wait a bit for other comments before taking any action. Robert's advice being the last (and only)

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Nicolas Barbier wrote: > The specifics of relation databases can be entirely ignored in case > serializability is provided on the "page layer" level. Aha, I now see very vaguely how that could work, yes. Thank you for elaborating on this. I agree that this isn't the best way forward for Postg

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Kevin Grittner wrote: > I'm probably not quite as clueless as you think on this; I realize > that keeping SIREAD locks in memory will require many more slots for > locks, escalation from tuple level to page or coarser when there are > many on a table, or (most likely) both. ..oh, there's the

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Nicolas Barbier
2010/1/7 Markus Wanner : > (It's interesting that with "database page" level granularity, he states > that predicate locking would not be necessary. Instead any page can be > locked at any time. For this to work, according to my reasoning, you'd > have to know in advance on which page potentially

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Kevin Grittner wrote: > We're very much on the same page. My goal was to get predicate > locking that didn't miss anything, even though it was ridiculously > coarse, then implement the simplest possible SSI on top of it, without > worrying about optimizations, then incrementally move toward >

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Markus Wanner
Hi, Greg Stark wrote: > aaah... I think I see where we've gone off track in previous > discussions...you think postgres keeps row level locks in a shared > memory data structure. It doesn't it stores all row level locks *in* > the tuple itself. It only stores the lock in memory briefly while > a

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Robert Haas
On Wed, Jan 6, 2010 at 5:04 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> I think you should have users/kgrittner/postgres.git rather than >> serializable.git.  serializable sounds more like the branch name. > > Hmmm  On a multi-year project it seemed more than remotely > possible that

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Kevin Grittner
Robert Haas wrote: > I think you should have users/kgrittner/postgres.git rather than > serializable.git. serializable sounds more like the branch name. Hmmm On a multi-year project it seemed more than remotely possible that responsibility for the project could shift, either to an extern

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Dimitri Fontaine
"Kevin Grittner" writes: > I've read through some git tutorials, but > there's a lot to digest and I'm not entirely sure this is a good way > to proceed. I found that the following video is really helpful at grasping the concepts of git, that it exposes pretty directly even though it's meant to p

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Robert Haas
On Wed, Jan 6, 2010 at 4:29 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> And use git so you can keep merging up to CVS HEAD easily. > > Regarding this, I was thinking that it would make sense to use a > repository on git.postgresql.org to coordinate my work with any > other people interest

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Kevin Grittner
Robert Haas wrote: > And use git so you can keep merging up to CVS HEAD easily. Regarding this, I was thinking that it would make sense to use a repository on git.postgresql.org to coordinate my work with any other people interested in the project. I would clone from there for the repository

Re: [HACKERS] Serializable Isolation without blocking

2010-01-03 Thread Kevin Grittner
Jeff Davis wrote: > I started a wiki page here: > > http://wiki.postgresql.org/wiki/Serializable I've filled it in with all relevant information which came to mind. If you can spot something I missed, please feel free to correct that or let me know so that I can. -Kevin -- Sent via pgsql

Re: [HACKERS] Serializable Isolation without blocking

2010-01-01 Thread Kevin Grittner
"Albe Laurenz" wrote: > If I remember right, one necessity for the SIREAD lock technique > was that SIREAD locks taken by a transaction have to be kept after > the transaction has ended. Correct. An SIREAD lock must be held until completion of the last serializable transaction holding a snaps

Re: [HACKERS] Serializable Isolation without blocking

2010-01-01 Thread Albe Laurenz
Kevin Grittner wrote: >> It seems to me that the hard part of this problem is to describe >> the general mechanism by which conflicts will be detected, with >> specific references to the types of data structures that will be >> used to hold that information. > > Well, the general approach to tracki

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 7:45 PM, Kevin Grittner wrote: > Robert Haas  wrote: > >> What predicate locking? If you take ACCESS EXCLUSIVE locks on every >> read, that should serialize all access to every table. Predicate >> locking wouldn't do anything, because the table would be completely >> inacce

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: > What predicate locking? If you take ACCESS EXCLUSIVE locks on every > read, that should serialize all access to every table. Predicate > locking wouldn't do anything, because the table would be completely > inaccessible to all competing transactions. Yeah, that's the benef

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 4:44 PM, Kevin Grittner wrote: >> I wonder if you couldn't start with something ridiculously poor, >> like maybe an S2PL implementation with only table-level granularity >> - just make any operation that reads or writes a table grab an >> ACCESS EXCLUSIVE lock until transac

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: > aaah... I think I see where we've gone off track in previous > discussions...you think postgres keeps row level locks in a shared > memory data structure. It doesn't it stores all row level locks > *in* the tuple itself. It only stores the lock in memory briefly > while actua

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
aaah... I think I see where we've gone off track in previous discussions...you think postgres keeps row level locks in a shared memory data structure. It doesn't it stores all row level locks *in* the tuple itself. It only stores the lock in memory briefly while actually acquiring the lock. Once

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Jeff Davis wrote: > I started a wiki page here: > > http://wiki.postgresql.org/wiki/Serializable I'll try to get that filled in with something useful over the weekend. I'm heading to a party soon, and may not be in shape to work on it tomorrow ;-) Happy New Year, all! -Kevin -- S

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: > OK. I think it will become more clear whether the existing lock > data structures are adequate as you move into detailed design. I've gotten far enough in reviewing it to be pretty convinced that they'll cover all the granularities I'm likely to want unless I get to the po

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Jeff Davis
On Thu, 2009-12-31 at 09:11 -0600, Kevin Grittner wrote: > Yeah, that's why this is a two to four year project. I started a wiki page here: http://wiki.postgresql.org/wiki/Serializable I didn't add much content yet, but can participants in this discussion please try to organize the various issu

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 1:43 PM, Kevin Grittner wrote: > Robert Haas  wrote: >> It seems to me that the hard part of this problem is to describe >> the general mechanism by which conflicts will be detected, with >> specific references to the types of data structures that will be >> used to hold th

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: > It seems to me that the hard part of this problem is to describe > the general mechanism by which conflicts will be detected, with > specific references to the types of data structures that will be > used to hold that information. Well, the general approach to tracking SIR

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 12:10 PM, Kevin Grittner wrote: > Greg Stark  wrote: > >> Hm, this raises the issue that you'll have to figure out what >> should happen if two different transactions are using different >> isolation modes. Currently our two isolation modes only control >> behaviour within

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: > Hm, this raises the issue that you'll have to figure out what > should happen if two different transactions are using different > isolation modes. Currently our two isolation modes only control > behaviour within your transaction so they co-exist perfectly fine. > > ISTM you

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
On Thu, Dec 31, 2009 at 3:11 PM, Kevin Grittner wrote: > Yeah, that's why this is a two to four year project.  And I would > point out that if there *wasn't* a performance hit in serializable > mode, none of the other isolation levels would exist.  These less > rigorous modes exist precisely becau

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: > The unsolved problems that have been raised are: > [legion] Yeah, that's why this is a two to four year project. And I would point out that if there *wasn't* a performance hit in serializable mode, none of the other isolation levels would exist. These less rigorous modes

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
On Thu, Dec 31, 2009 at 12:45 PM, Nicolas Barbier wrote: > * To answer your specific question: T0 might not read that specific > row, but the COUNT(..) definitely must read *something* that must be > modified by T1 when it updates the ishighlander field: The problem occurs when the update happens

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Nicolas Barbier
[ Reviving this old thread because a recent one referred to it. ] 2009/5/7 Albe Laurenz : > Kevin Grittner wrote: > >> > maybe I misunderstood something. >> > >> > Consider a function >> > "makehighlander(personid integer) RETURNS void" >> > defined like this: >> > >> >    SELECT ishighlander INT

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Kevin Grittner
Simon Riggs wrote: > Do we need table-level predicate locks at all? What would they give > us? Why not just go straight for fine-grained page-level locks? I don't want to get too far into implementation discussions at this phase (see Tom's slides ;-)), but suffice it to say that a table scan

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Simon Riggs
On Wed, 2009-05-27 at 13:34 -0500, Kevin Grittner wrote: > For the record, it became clear that I did a bad job of communicating > on this thread... You did good work, IMHO. Not everything will reach consensus and that's not your fault. > first implement table level predicate > locks, since tha

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Kevin Grittner
I wrote: > Greg Stark wrote: >> This thread has really been one of those cases where everyone >> thought they were having a different kind of discussion. > > Apparently so. In light of discussions at PGCon, I'm going to abandon this thread in favor of a discussion of what this feature would

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Greg Stark wrote: > Kevin Grittner wrote: >> Greg Stark wrote: >> >>> I thought the big problem with providing true serializability was >>> the predicate locking. If it doesn't address that need then does >>> this get us any closer? >> >> I thought the big problem was the perception that perfor

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
"Albe Laurenz" wrote: > In my first reply I wondered if the presence of concurrent "read > committed" transactions would somehow affect the correctness of the > algorithm, as the authors don't mention that. Yeah, I was concerned about that, too. In thinking it through I've convinced myself th

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Greg Stark
On Mon, May 11, 2009 at 3:11 PM, Kevin Grittner wrote: > Greg Stark wrote: > >> I thought the big problem with providing true serializability was >> the predicate locking. If it doesn't address that need then does >> this get us any closer? > > I thought the big problem was the perception that pe

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Greg Stark wrote: > I thought the big problem with providing true serializability was > the predicate locking. If it doesn't address that need then does > this get us any closer? I thought the big problem was the perception that performance would suffer and that the level of blocking required

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Albe Laurenz
Kevin Grittner wrote: > > All the authors show with regard to predicate handling is > > handwaving, > > That is because predicate locking is a mature technology with many > known implementations. The best technique for any database product > will depend on that product, and their technique doesn

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Greg Stark
On Mon, May 11, 2009 at 2:49 PM, Kevin Grittner wrote: > "Albe Laurenz" wrote: > >> All the authors show with regard to predicate handling is >> handwaving, > > That is because predicate locking is a mature technology with many > known implementations.  The best technique for any database product

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
"Albe Laurenz" wrote: > All the authors show with regard to predicate handling is > handwaving, That is because predicate locking is a mature technology with many known implementations. The best technique for any database product will depend on that product, and their technique doesn't depend

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Albe Laurenz
Kevin Grittner wrote: > > I still haven't actually read the paper so I should probably bow out > > from the conversation until I do. I was apparently already under > > one misapprehension as Laurenz just claimed the paper does not show > > how to prevent "phantoms" (phantom reads I assume?). Perha

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:49 PM, Kevin Grittner wrote: > Greg Stark wrote: > >> Well I don't understand what storing locks in an index can >> accomplish if other queries might use other indexes or sequential >> scans to access the records and never see those locks. >> >> Or does this method only r

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Tom Lane wrote: > Greg Stark writes: >> ... Argh, sorry, as soon as I hit send I realized this is wrong. >> Writers already need to insert into every index, so that's not a >> problem. > > What about HOT? I think that a read would need to lock both the row or tuple (not sure exactly how that

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Tom Lane
Greg Stark writes: > ... Argh, sorry, as soon as I hit send I realized this is wrong. Writers > already need to insert into every index, so that's not a problem. What about HOT? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 4:12 PM, Greg Stark wrote: > On Fri, May 8, 2009 at 3:49 PM, Kevin Grittner > wrote: >> Greg Stark wrote: >> >>> Well I don't understand what storing locks in an index can >>> accomplish if other queries might use other indexes or sequential >>> scans to access the records

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Greg Stark wrote: > Well I don't understand what storing locks in an index can > accomplish if other queries might use other indexes or sequential > scans to access the records and never see those locks. > > Or does this method only require that writers discover the locks and > therefore only

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
"Albe Laurenz" wrote: > As far as I know, only the table rows that are found in the index > scan are examined for visibility. Which would be only one in my > example. S2PL locking schemes routinely use index range locks. > But in your attempt to sketch a way how true serializability could >

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:00 PM, Kevin Grittner wrote: > Greg Stark wrote: >> On Thu, May 7, 2009 at 11:08 PM, Kevin Grittner >> wrote: >>> I would assume that SELECT shown above would either resolve to a >>> table scan, in which case you would have to have an SIREAD lock at >>> the table level >

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:00 PM, Kevin Grittner wrote: > >> I've removed the broken email address for now -- please re-add the >> correct email address. > > I'll see what I can find.  When I last corresponded with him, he was > still at the University of Sidney, working on his PhD by applying > the

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Greg Stark wrote: > On Thu, May 7, 2009 at 11:08 PM, Kevin Grittner > wrote: >> I would assume that SELECT shown above would either resolve to a >> table scan, in which case you would have to have an SIREAD lock at >> the table level > > That sounds like we're back to the MSSQL/Sybase way of do

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: > > What if there is an index on the "ishighlander" row? > > Then an index scan would find only one candidate to examine, > > and the other rows would not even be touched by the execution plan. > > I assume you're talking about this line of your function: > > SELECT coun

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 22:47 +0100, Greg Stark wrote: > On Thu, May 7, 2009 at 6:13 PM, Simon Riggs wrote: > > Apologies Michael, I see that my mail did remove you. That was a > > unconscious error; I was particularly interested in your comments > > regarding my assessment of the algorithmic compl

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Greg Stark
On Thu, May 7, 2009 at 11:08 PM, Kevin Grittner wrote: > I would assume that SELECT shown above would either resolve to a > table scan, in which case you would have to have an SIREAD lock at the > table level That sounds like we're back to the MSSQL/Sybase way of doing things where you have to un

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Greg Stark wrote: > If I do something like "SELECT count(*) FROM tab WHERE > complex_function(a,b) = 5" > > And then you "INSERT INTO tab (a,b) VALUES (1,2)". How would you > store any record of the fact that there's a serialization failure > iff complex_function(1,2)=5 in any way that lets yo

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Greg Stark wrote: > However on subsequent messages you attempted to re-add him but got > his email address wrong. I assume everyone else got a bounce like I > got? Some of my emails are getting through; some not. I haven't figured out why. I'm calling it "best effort" for now, and will send

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Greg Stark
On Thu, May 7, 2009 at 6:13 PM, Simon Riggs wrote: > Apologies Michael, I see that my mail did remove you. That was a > unconscious error; I was particularly interested in your comments > regarding my assessment of the algorithmic complexity of the new theory > and existing serialization technique

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs wrote: > Do we need full locking of everything we might > touch, or tracking of what we have touched? > If you need the "might touch" then you either need to implement > locking that will effect everybody (which ain't ever gonna fly round > here), or you implement a scheme that is

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Greg Stark
On Thu, May 7, 2009 at 6:31 PM, Simon Riggs wrote: > Each user must compare against work performed by all other users. O(N). > > There are N users, so O(N^2). i think this logic only works if you must scan every item for every other user every time. If you have data structures like binary trees o

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 15:10 -0500, Kevin Grittner wrote: > The assertion that > there is some need for each session to wade through something for > every other session seems baseless to me. I'm wondering what I might > be missing. That's Greg's point. Do we need full locking of everything we mig

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs wrote: > On Thu, 2009-05-07 at 12:39 -0500, Kevin Grittner wrote: >> Simon Riggs wrote: >> >> > Each user must compare against work performed by all other users. >> > O(N). >> > >> > There are N users, so O(N^2). >> >> Why does that apply here and not in the update conflict detec

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 12:39 -0500, Kevin Grittner wrote: > Simon Riggs wrote: > > > Each user must compare against work performed by all other users. > O(N). > > > > There are N users, so O(N^2). > > Why does that apply here and not in the update conflict detection? I think the shoe is on t

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs wrote: > Each user must compare against work performed by all other users. O(N). > > There are N users, so O(N^2). Why does that apply here and not in the update conflict detection? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 10:56 -0500, Kevin Grittner wrote: > > It's clear that any new-theory solution will cost significantly more > > as the number of users increases, at least O(N^2), whereas simply > > waiting is only O(N), AFAICS. > > I'm not following your reasoning on the O(N^2). Could you

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 11:15 -0500, Kevin Grittner wrote: > Please keep Michael Cahill copied on this thread, per his request. > > I just noticed the omission on a few messages and will forward them to > him. Apologies Michael, I see that my mail did remove you. That was a unconscious error; I

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Please keep Michael Cahill copied on this thread, per his request. I just noticed the omission on a few messages and will forward them to him. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs wrote: > It wouldn't be 692 lines of code Agreed. The original implementation was in an MVCC database which already supported full serializability using strict 2 phase locking and used page level locks. Both of these made the implementation simpler than it would be in PostgreSQL.

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
"Albe Laurenz" wrote: > What if there is an index on the "ishighlander" row? > Then an index scan would find only one candidate to examine, > and the other rows would not even be touched by the execution plan. I assume you're talking about this line of your function: SELECT count(*) INTO

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: > > Where does T1 select rows that were modified by T0? It selects only > > one row, the one it modified itself, right? > > You have to select it to know whether to count it, right? We are getting closer. So an SIREAD lock is taken for every row that is examined during the

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Heikki Linnakangas wrote: > Simon Riggs wrote: >> It wouldn't be 692 lines of code and even if it were the impact of >> that code would be such that it would need to be optional, since it >> would differ in definition from an existing SQL Standard isolation >> mode and it would have additional pe

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
"Albe Laurenz" wrote: > Kevin Grittner wrote: >> I do think you misunderstood. If there are two concurrent >> executions and each reads one row, there will be an SIREAD lock for >> each of those rows. As an example, let's say that one of them (T0) >> updates its row and does its count, finds

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: > > maybe I misunderstood something. > > > > Consider a function > > "makehighlander(personid integer) RETURNS void" > > defined like this: > > > >SELECT ishighlander INTO b FROM scots WHERE id=personid; > >IF b THEN > > RETURN; /* no need to do anything */ > >

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Gregory Stark wrote: > "Albe Laurenz" writes: > >> So I think one would have to add intention locks for rows >> considered in the WHERE clause to guarantee true serializability. > > Does the paper explain how to deal with rows "considered" in the > WHERE clause which don't yet exist? Ie, "SELE

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
"Albe Laurenz" wrote: > maybe I misunderstood something. > > Consider a function > "makehighlander(personid integer) RETURNS void" > defined like this: > >SELECT ishighlander INTO b FROM scots WHERE id=personid; >IF b THEN > RETURN; /* no need to do anything */ >END IF; >

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Greg Stark wrote: > > So I think one would have to add intention locks for rows considered > > in the WHERE clause to guarantee true serializability. > > Does the paper explain how to deal with rows "considered" in the WHERE clause > which don't yet exist? Ie, "SELECT count(*) WHERE foo" needs to

  1   2   >