Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-05 Thread Robert Haas
On Wed, Sep 4, 2013 at 8:08 PM, Andres Freund and...@2ndquadrant.com wrote: You've proposed an framework and algorithm for something I'd really, really like to see. I don't think that it can work explicitly as you proposed, so I roughly sketched out a solution I can see. I don't want my idea

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Robert Haas
On Sat, Aug 31, 2013 at 2:34 PM, Andres Freund and...@2ndquadrant.com wrote: After some thinking I don't think any solution primarily based on holding page level locks across other index operations is going to scale ok. I'd like to chime in with a large +1 for this sentiment and pretty much

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: Concurrent readers will block in a non-interruptible wait if they try to access a buffer, and that's a situation that will be intolerable if, for example, it can persist across a disk I/O. And I don't see any way to

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Andres Freund
On 2013-09-04 15:01:57 -0700, Peter Geoghegan wrote: On Wed, Sep 4, 2013 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: Concurrent readers will block in a non-interruptible wait if they try to access a buffer, and that's a situation that will be intolerable if, for example, it can

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 3:39 PM, Andres Freund and...@2ndquadrant.com wrote: Sorry to be harsh here, but I don't think I need to do that. I've explained most of the reasons I see that that approach won't work out and so far I don't see those refuted. And to me those issues seem to be fatal for

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Andres Freund
Hi, We seem to be miscommunication a bit. You've proposed an framework and algorithm for something I'd really, really like to see. I don't think that it can work explicitly as you proposed, so I roughly sketched out a solution I can see. I don't want my idea to win, I want a idea to win. I

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Peter Geoghegan
On Wed, Sep 4, 2013 at 5:08 PM, Andres Freund and...@2ndquadrant.com wrote: I don't want my idea to win, I want a idea to win. I know. I want the same thing. You're the patch author here whose plans are laid open to be scrutinized ;). If you think my idea has merit, use and adapt it to

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-04 Thread Gavin Flower
On 05/09/13 08:26, Robert Haas wrote: On Sat, Aug 31, 2013 at 2:34 PM, Andres Freund and...@2ndquadrant.com wrote: After some thinking I don't think any solution primarily based on holding page level locks across other index operations is going to scale ok. I'd like to chime in with a large +1

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-03 Thread Craig Ringer
On 09/03/2013 06:18 AM, Peter Geoghegan wrote: On Mon, Sep 2, 2013 at 6:25 AM, Craig Ringer cr...@2ndquadrant.com wrote: It'll be yet another way for people to get upsert wrong, of course. They'll use a wCTE with RETURNING REJECTS to do an UPDATE of the rejects w/o locking the table against

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-03 Thread Peter Geoghegan
On Tue, Sep 3, 2013 at 12:52 AM, Craig Ringer cr...@2ndquadrant.com wrote: That'd certainly be the ideal, but as you say is far from simple, and IIRC prior discussions here have suggested the SSI / predicate locking stuff won't help. I think that by far the strongest argument for what Andres

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-03 Thread Peter Geoghegan
On Tue, Sep 3, 2013 at 2:11 AM, Peter Geoghegan p...@heroku.com wrote: and it would be totally unacceptable if that meant that lots of people blocked on the page lock that an upserter held while it tried to acquire locks on tuples By which I mean: it seems shaky that I'd then be assuming that

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-03 Thread Andres Freund
On 2013-09-02 21:59:42 -0700, Peter Geoghegan wrote: I can't really agree with this part. First off, a heap_insert() certainly isn't lightweight operation. There's several things that can take extended time: * toasting of rows (writing a gigabyte worth of data...) * extension of the

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-02 Thread Peter Eisentraut
On Fri, 2013-08-30 at 15:09 -0700, Peter Geoghegan wrote: The attached WIP patch implements this for Postgres, with a few notable differences: Your patch breaks the ECPG regression tests: ../../preproc/ecpg --regression -I./../../include -o insupd.c -I. insupd.pgc insupd.pgc:22: ERROR: syntax

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-02 Thread Peter Geoghegan
On Sun, Sep 1, 2013 at 11:38 PM, Peter Eisentraut pete...@gmx.net wrote: ../../preproc/ecpg --regression -I./../../include -o insupd.c -I. insupd.pgc insupd.pgc:22: ERROR: syntax error at or near into make[3]: *** [insupd.c] Error 3 I'll work on fixing it then. Thanks. -- Peter Geoghegan

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-02 Thread Craig Ringer
On 08/31/2013 06:40 AM, Josh Berkus wrote: 3) RETURNING is expanded - RETURNING REJECTS * is now possible where that makes sense. Oh, nifty! OK, now I can *really* use this feature. Absolutely; especially combined with COPY to a staging TEMPORARY or UNLOGGED table. It'll be yet another way

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2013 at 6:25 AM, Craig Ringer cr...@2ndquadrant.com wrote: It'll be yet another way for people to get upsert wrong, of course. They'll use a wCTE with RETURNING REJECTS to do an UPDATE of the rejects w/o locking the table against writes first. Documenting this pitfall should be

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-02 Thread Peter Geoghegan
On Sun, Sep 1, 2013 at 4:06 AM, Andres Freund and...@2ndquadrant.com wrote: We're looking for the first duplicate. So it would probably be okay for the IGNORE case to not bother retrying and re-locking if the other transaction committed (which, at least very broadly speaking, is the likely

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-01 Thread Peter Geoghegan
On Sat, Aug 31, 2013 at 7:53 PM, Peter Geoghegan p...@heroku.com wrote: With a table with many unique indexes and many reasons to decide to reject a tuple proposed for insertion by INSERT...ON DUPLICATE KEY IGNORE, it isn't hard to imagine them all becoming heavily bloated very quickly. There

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-01 Thread Andres Freund
On 2013-08-31 19:38:49 -0700, Peter Geoghegan wrote: On Sat, Aug 31, 2013 at 11:34 AM, Andres Freund and...@2ndquadrant.com wrote: Won't S2 in this case exclusively lock a page in foo_a_key (the one where 2 will reside on) for 3600s, while it's waiting for S1 to finish while getting

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-09-01 Thread Andres Freund
On 2013-08-31 23:02:46 -0700, Peter Geoghegan wrote: On Sat, Aug 31, 2013 at 7:53 PM, Peter Geoghegan p...@heroku.com wrote: With a table with many unique indexes and many reasons to decide to reject a tuple proposed for insertion by INSERT...ON DUPLICATE KEY IGNORE, it isn't hard to

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-31 Thread Andres Freund
Hi, On 2013-08-30 19:38:24 -0700, Peter Geoghegan wrote: On Fri, Aug 30, 2013 at 5:47 PM, Andres Freund and...@2ndquadrant.com wrote: While I ponder on it some more, could you explain whether I understood something correctly? Consider the following scenario: CREATE TABLE foo(a int

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-31 Thread Peter Geoghegan
On Sat, Aug 31, 2013 at 11:34 AM, Andres Freund and...@2ndquadrant.com wrote: Won't S2 in this case exclusively lock a page in foo_a_key (the one where 2 will reside on) for 3600s, while it's waiting for S1 to finish while getting the speculative insertion into foo_b_key? Yes. The way the

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-31 Thread Peter Geoghegan
On Sat, Aug 31, 2013 at 7:38 PM, Peter Geoghegan p...@heroku.com wrote: Imo that solution is fairly elegant because it doesn't cause any heap bloat, and it causes the same amount of index bloat insert-into-heap-first type of solutions cause. I don't think that's a reasonable comparison.

[HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-30 Thread Peter Geoghegan
For many years now, MySQL has a feature called INSERT IGNORE [1]; SQLite has INSERT ON CONFLICT IGNORE [2]; SQL Server has an option called IGNORE_DUP_KEY and Oracle has a hint called IGNORE_ROW_ON_DUPKEY_INDEX (they acknowledge that it's a bit odd that a hint changes the semantics of a DML

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-30 Thread Josh Berkus
On 08/30/2013 03:09 PM, Peter Geoghegan wrote: The attached WIP patch implements this for Postgres, with a few notable differences: Thank you for addressing this. If nobody is going to hack out MERGE, we could really use at least this feature. 3) RETURNING is expanded - RETURNING REJECTS *

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-30 Thread Peter Geoghegan
On Fri, Aug 30, 2013 at 3:40 PM, Josh Berkus j...@agliodbs.com wrote: Does this work with multiple VALUES rows? Yes. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-30 Thread Andres Freund
Hi, This is awesome. On 2013-08-30 15:09:59 -0700, Peter Geoghegan wrote: 1) The patch is only interested in unique index violations (specifically, violations of amcanunique access method unique indexes); it will not do anything with NULL constraint violations, as the MySQL feature does, for

Re: [HACKERS] INSERT...ON DUPLICATE KEY IGNORE

2013-08-30 Thread Peter Geoghegan
On Fri, Aug 30, 2013 at 5:47 PM, Andres Freund and...@2ndquadrant.com wrote: This is awesome. Thanks. All that seems sane to me. I very, very much do not want it to deal with NOT NULL violations. Sure. But there's nothing stopping us from doing that as a totally orthogonal thing. Not that I