Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-06 Thread Kevin Grittner
Kevin Grittner wrote: > For an example of how things can look OK in terms of enforced > constraints as of different moments in time, yet those moments in > time could be inconsistent, see this section: > > http://wiki.postgresql.org/wiki/SSI#Enforcing_Business_Rules_in_Triggers Copy/paste erro

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-06 Thread Kevin Grittner
Andres Freund wrote: > Peter Eisentraut schrieb: >> On 12/18/13, 2:22 PM, Andres Freund wrote: >>> It would only force serialization for transactions that modify >>> tables covered by the assert, that doesn't seem to bad. >>> Anything covered by an assert shoulnd't be modified frequently, >>> oth

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Andres Freund
Peter Eisentraut schrieb: >On 12/18/13, 2:22 PM, Andres Freund wrote: >> It would only force serialization for transactions that modify tables >> covered by the assert, that doesn't seem to bad. Anything covered by >an >> assert shoulnd't be modified frequently, otherwise you'll run into >major

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 4:04 PM, Andrew Dunstan wrote: > Umm, that's really a major limitation in utility. We need to come up > with a better answer than this, which would essentially hobble the > facility. We don't have any facility to run a trigger on just any command, it needs to be triggered by a table.

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 2:45 PM, Andres Freund wrote: > Well, the patch's syntax seems to only allow to directly specify a SQL > query to check - we could iterate over the querytree to gather all > related tables and reject any function we do not understand. Creating an assertion scans the query tree and att

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 2:22 PM, Andres Freund wrote: > It would only force serialization for transactions that modify tables > covered by the assert, that doesn't seem to bad. Anything covered by an > assert shoulnd't be modified frequently, otherwise you'll run into major > performance problems. I think th

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-27 Thread David Fetter
On Mon, Nov 25, 2013 at 12:10:22PM -0800, Kevin Grittner wrote: > David Fetter wrote: > > On Mon, Nov 25, 2013 at 11:04:23AM -0800, Kevin Grittner wrote: > > >> As soon as we are out of this CF, I am planning to write code to > >> capture deltas and fire functions to process them "eagerly" > >> (

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-19 Thread Florian Pflug
On Dec18, 2013, at 20:39 , Alvaro Herrera wrote: > Andres Freund wrote: >> On 2013-12-18 13:44:15 -0300, Alvaro Herrera wrote: >>> Heikki Linnakangas wrote: >>> Ah, I see. You don't need to block anyone else from modifying the table, you just need to block anyone else from committing a

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Kevin Grittner
Josh Berkus wrote: > On 12/18/2013 11:26 AM, Jim Nasby wrote: >> Another possibility is to allow for two different types of >> assertions, one based on SSI and one based on locking. > > The locking version would have to pretty much lock on a table > basis (or even a whole-database basis) every ti

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Josh Berkus
On 12/18/2013 11:26 AM, Jim Nasby wrote: > The flip-side is that now you can get serialization failures, and I > think there's a ton of software that has no clue how to deal with that. > So now you don't get to use assertions at all unless you re-engineer > your application (but see below). Well,

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andrew Dunstan
On 12/18/2013 04:09 PM, Heikki Linnakangas wrote: On 12/18/2013 11:04 PM, Andrew Dunstan wrote: On 12/18/2013 02:45 PM, Andres Freund wrote: On 2013-12-18 16:39:58 -0300, Alvaro Herrera wrote: Andres Freund wrote: It would only force serialization for transactions that modify tables covered

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 11:04 PM, Andrew Dunstan wrote: On 12/18/2013 02:45 PM, Andres Freund wrote: On 2013-12-18 16:39:58 -0300, Alvaro Herrera wrote: Andres Freund wrote: It would only force serialization for transactions that modify tables covered by the assert, that doesn't seem to bad. Anything c

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andrew Dunstan
On 12/18/2013 02:45 PM, Andres Freund wrote: On 2013-12-18 16:39:58 -0300, Alvaro Herrera wrote: Andres Freund wrote: It would only force serialization for transactions that modify tables covered by the assert, that doesn't seem to bad. Anything covered by an assert shoulnd't be modified frequ

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Kevin Grittner
Jim Nasby wrote: > On 12/18/13, 1:42 PM, Kevin Grittner wrote: >> Jim Nasby wrote: >> >>> This is another case where it would be very useful to restrict >>> what relations a transaction (or in this case, a substransaction) >>> can access. If we had the ability to make that restriction then >>> we

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Jim Nasby
On 12/18/13, 1:42 PM, Kevin Grittner wrote: Jim Nasby wrote: This is another case where it would be very useful to restrict what relations a transaction (or in this case, a substransaction) can access. If we had the ability to make that restriction then we could force assertions that aren't pl

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andres Freund
On 2013-12-18 16:39:58 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > It would only force serialization for transactions that modify tables > > covered by the assert, that doesn't seem to bad. Anything covered by an > > assert shoulnd't be modified frequently, otherwise you'll run into maj

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Kevin Grittner
Jim Nasby wrote: > This is another case where it would be very useful to restrict > what relations a transaction (or in this case, a substransaction) > can access. If we had the ability to make that restriction then > we could force assertions that aren't plain SQL to explicitly > specify what ta

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Alvaro Herrera
Andres Freund wrote: > On 2013-12-18 13:44:15 -0300, Alvaro Herrera wrote: > > Heikki Linnakangas wrote: > > > > > Ah, I see. You don't need to block anyone else from modifying the > > > table, you just need to block anyone else from committing a > > > transaction that had modified the table. So t

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Jim Nasby
On 12/18/13, 10:44 AM, Alvaro Herrera wrote: It might prove useful to note that any given assertion involves tables A, B, C. If a transaction doesn't modify any of those tables then there's no need to run the assertion when the transaction commits, skipping acquisition of the assertion lock. Pr

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Jim Nasby
On 12/18/13, 11:57 AM, Josh Berkus wrote: On 12/18/2013 08:44 AM, Alvaro Herrera wrote: Another thought: at the initial run of the assertion, note which tables it locked, and record this as an OID array in the catalog row for the assertion; consider running the assertion only when those tables a

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andres Freund
On 2013-12-18 13:44:15 -0300, Alvaro Herrera wrote: > Heikki Linnakangas wrote: > > > Ah, I see. You don't need to block anyone else from modifying the > > table, you just need to block anyone else from committing a > > transaction that had modified the table. So the locks shouldn't > > interfere

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Josh Berkus
On 12/18/2013 08:44 AM, Alvaro Herrera wrote: > Another thought: at the initial run of the assertion, note which tables > it locked, and record this as an OID array in the catalog row for the > assertion; consider running the assertion only when those tables are > touched. This doesn't work if the

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Alvaro Herrera
Heikki Linnakangas wrote: > Ah, I see. You don't need to block anyone else from modifying the > table, you just need to block anyone else from committing a > transaction that had modified the table. So the locks shouldn't > interfere with regular table locks. A ShareUpdateExclusiveLock on > the as

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Kevin Grittner
Josh Berkus wrote: > Serializable or not, *what* do we lock for assertions?  It's not > rows.  Tables?  Which tables?  What if the assertion is an > interpreted language function?  Does the SSI reference counter > really take care of all of this? The simple answer is that, without adding any add

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Kevin Grittner
Heikki Linnakangas wrote: >On 12/18/2013 01:39 PM, Andres Freund wrote: >> On 2013-12-18 13:07:51 +0200, Heikki Linnakangas wrote: >>> Here's another idea that doesn't involve SSI: >>> >>> At COMMIT, take a new snapshot and check that the assertion still passes >>> with that snapshot. Now, there's

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andrew Dunstan
On 12/18/2013 06:00 AM, Heikki Linnakangas wrote: If you don't force everything to run in SSI mode, then you have to somehow figure out what parts do need to run in SSI mode to enforce the assertion. For example, if the assertion refers tables A and B, perhaps you can get away without predi

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 01:50 PM, Andres Freund wrote: On 2013-12-18 13:46:59 +0200, Heikki Linnakangas wrote: On 12/18/2013 01:39 PM, Andres Freund wrote: On 2013-12-18 13:07:51 +0200, Heikki Linnakangas wrote: Here's another idea that doesn't involve SSI: At COMMIT, take a new snapshot and check that

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andres Freund
On 2013-12-18 13:46:59 +0200, Heikki Linnakangas wrote: > On 12/18/2013 01:39 PM, Andres Freund wrote: > >On 2013-12-18 13:07:51 +0200, Heikki Linnakangas wrote: > >>Here's another idea that doesn't involve SSI: > >> > >>At COMMIT, take a new snapshot and check that the assertion still passes > >>w

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 01:39 PM, Andres Freund wrote: On 2013-12-18 13:07:51 +0200, Heikki Linnakangas wrote: Here's another idea that doesn't involve SSI: At COMMIT, take a new snapshot and check that the assertion still passes with that snapshot. Now, there's a race condition, if another transaction i

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Andres Freund
On 2013-12-18 13:07:51 +0200, Heikki Linnakangas wrote: > Here's another idea that doesn't involve SSI: > > At COMMIT, take a new snapshot and check that the assertion still passes > with that snapshot. Now, there's a race condition, if another transaction is > committing at the same time, and per

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 02:59 AM, Josh Berkus wrote: On 12/17/2013 01:42 PM, Kevin Grittner wrote: It works fine as long as you set default_transaction_isolation = 'serializable' and never override that. :-) Of course, it sure would be nice to have a way to prohibit overrides, but that's another issue.

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 02:59 AM, Josh Berkus wrote: On 12/17/2013 01:42 PM, Kevin Grittner wrote: Josh Berkus wrote: Going back over this patch, I haven't seen any further discussion of the point Heikki raises above, which seems like a bit of a showstopper. Heikki, did you have specific ideas on how t

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-17 Thread Josh Berkus
On 12/17/2013 01:42 PM, Kevin Grittner wrote: > Josh Berkus wrote: >> Going back over this patch, I haven't seen any further discussion of the >> point Heikki raises above, which seems like a bit of a showstopper. >> >> Heikki, did you have specific ideas on how to solve this? Right now my >> min

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-17 Thread Gavin Flower
On 18/12/13 10:48, Andrew Dunstan wrote: On 12/17/2013 04:42 PM, Kevin Grittner wrote: Josh Berkus wrote: On 11/15/2013 05:41 AM, Heikki Linnakangas wrote: A fundamental problem with this is that it needs to handle isolation reliable, so that the assertion cannot be violated when two concurr

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-17 Thread Andrew Dunstan
On 12/17/2013 04:42 PM, Kevin Grittner wrote: Josh Berkus wrote: On 11/15/2013 05:41 AM, Heikki Linnakangas wrote: A fundamental problem with this is that it needs to handle isolation reliable, so that the assertion cannot be violated when two concurrent backends do things. Consider the examp

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-17 Thread Kevin Grittner
Josh Berkus wrote: > On 11/15/2013 05:41 AM, Heikki Linnakangas wrote: >> A fundamental problem with this is that it needs to handle isolation >> reliable, so that the assertion cannot be violated when two concurrent >> backends do things. Consider the example from the manual, which checks >> that

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-12-17 Thread Josh Berkus
On 11/15/2013 05:41 AM, Heikki Linnakangas wrote: > A fundamental problem with this is that it needs to handle isolation > reliable, so that the assertion cannot be violated when two concurrent > backends do things. Consider the example from the manual, which checks > that a table has at least one

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-26 Thread Peter Eisentraut
On 11/24/13, 10:03 AM, Simon Riggs wrote: > So we'd need to get access to the changed rows, rather than > re-executing a huge SQL command that re-checks every row of the table. > That last point will make it unusable for sensible amounts of data. SQL assertions work with arbitrary expressions. So

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-25 Thread Kevin Grittner
David Fetter wrote: > On Mon, Nov 25, 2013 at 11:04:23AM -0800, Kevin Grittner wrote: >> As soon as we are out of this CF, I am planning to write code to >> capture deltas and fire functions to process them "eagerly" >> (within the creating transaction).  There has been suggestions >> that the ch

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-25 Thread David Fetter
On Mon, Nov 25, 2013 at 11:04:23AM -0800, Kevin Grittner wrote: > Andrew Tipton wrote: > > Simon Riggs wrote: > >> So we'd need to get access to the changed rows, rather than > >> re-executing a huge SQL command that re-checks every row of the > >> table.  That last point will make it unusable fo

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-25 Thread Kevin Grittner
Andrew Tipton wrote: > Simon Riggs wrote: >> So we'd need to get access to the changed rows, rather than >> re-executing a huge SQL command that re-checks every row of the >> table.  That last point will make it unusable for sensible >> amounts of data. > > That sounds very similar to handling in

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-24 Thread Andrew Tipton
On Sun, Nov 24, 2013 at 11:03 PM, Simon Riggs wrote: > So we'd need to get access to the changed rows, rather than > re-executing a huge SQL command that re-checks every row of the table. > That last point will make it unusable for sensible amounts of data. That sounds very similar to handling in

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-24 Thread Simon Riggs
On 15 November 2013 03:30, Peter Eisentraut wrote: > A constraint trigger performs the actual checking. Good, that is consistent with other constraints. > This is not a performance feature. It's for things like, this table > should have at most 10 rows, or all the values in this table must be

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-15 Thread Heikki Linnakangas
On 15.11.2013 05:30, Peter Eisentraut wrote: Various places in the constraint checking code say something like, if we ever implement assertions, here is where it should go. I've been fiddling with filling in those gaps for some time now, and the other day I noticed, hey, this actually kind of wo

Re: [HACKERS] [PATCH] SQL assertions prototype

2013-11-14 Thread Pavel Stehule
+1 interesting feature Pavel 2013/11/15 Peter Eisentraut > Various places in the constraint checking code say something like, if we > ever implement assertions, here is where it should go. I've been > fiddling with filling in those gaps for some time now, and the other day > I noticed, hey,