Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Currently, the planner and executor are mostly independent of each other: the planner doesn't really know when the plan will be executed, and the executor doesn't know how recently the

Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 27.07.2011 17:43, Alexander Korotkov wrote: OK, thanks. I also found behaviour of GiST(without patch) with streaming replication that seems strange for me. On master there are only few rightlinks are InvalidBlockNumber while on slave there are a lot of them. I hack gevel for getting index

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 10:38 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.07.2011 17:43, Alexander Korotkov wrote: OK, thanks. I also found behaviour of GiST(without patch) with streaming replication that seems strange for me. On master there are only few rightlinks

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 13:13, Simon Riggs wrote: On Mon, Aug 1, 2011 at 10:38 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Attached is a patch for that for 9.1/master. The 9.0 GiST replay code was quite different, it will require a separate patch. Hmm, I was assured no changes

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Thom Brown
On 1 August 2011 11:44, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 13:13, Simon Riggs wrote: And what does NSN stand for? :-) Hmm, I don't know actually know what NSN is an acronym for :-). Node Sequence Number. -- Thom Brown Twitter: @darkixion IRC

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Achim Domma
Am 26.07.2011 um 00:40 schrieb Florian Pflug: On Jul25, 2011, at 22:31 , Achim Domma wrote: Am 25.07.2011 um 14:48 schrieb Florian Pflug: A more low-level API is provided by {heap,index}_{beginscan,endscan}, heap_{insert,update,delete} and index_insert. However, correct handling of

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown t...@linux.com wrote: On 1 August 2011 11:44, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 13:13, Simon Riggs wrote: And what does NSN stand for? :-) Hmm, I don't know actually know what NSN is an acronym for :-).

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Thom Brown
On 1 August 2011 12:25, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown t...@linux.com wrote: On 1 August 2011 11:44, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 13:13, Simon Riggs wrote: And what does NSN stand for? :-)

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Alexander Korotkov
On Mon, Aug 1, 2011 at 3:25 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown t...@linux.com wrote: On 1 August 2011 11:44, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 13:13, Simon Riggs wrote: And what does NSN stand

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Does the order of locking of the buffers matter? I'm sure it does. Yep. Do you mean that the BlockNumbers are already in correct sequence, or that you will be adding this code to redo? --  Simon

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Florian Pflug
On Aug1, 2011, at 13:23 , Achim Domma wrote: I have tables which store two integer IDs and a floating point rank. So the table MyTable might have these columns: EntityID - int PropertyID - int Rank - float My algorithm needs to retrieve EntityID-Rank-Pairs for some given PropertyIDs.

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Robert Haas
On Sun, Jul 31, 2011 at 8:32 PM, Jeff Davis pg...@j-davis.com wrote: fpLocalTransactionId is redundant with the lxid, and the explanation is that one that they have different locking semantics. That looks reasonable, and it avoided the need for the careful ordering while starting/ending a

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 14:35, Simon Riggs wrote: On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Does the order of locking of the buffers matter? I'm sure it does. Yep. Do you mean that the BlockNumbers are already in correct sequence, or that you

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 14:35, Simon Riggs wrote: On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: Does the order of locking of the buffers matter? I'm sure it

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 17:26, Simon Riggs wrote: On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I believe we code acquire the locks in right order already, and the patch I posted fixes the premature release of locks at page split. Your patch is good, but

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 01:23:26PM +0200, Achim Domma wrote: Am 26.07.2011 um 00:40 schrieb Florian Pflug: On Jul25, 2011, at 22:31 , Achim Domma wrote: Am 25.07.2011 um 14:48 schrieb Florian Pflug: A more low-level API is provided by {heap,index}_{beginscan,endscan},

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:04 AM, David Fetter da...@fetter.org wrote: SQL is Turing-complete, so the chances are excellent that it's possible to express that algorithm in it. ;) Look into common table expressions for iteration/recursion, and windowing functions, commonly used for time series.

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Jeff Davis
On Mon, 2011-08-01 at 08:12 -0400, Robert Haas wrote: Is the LocalTransactionIdIsValid(lxid) a guard against calling VirtualXactLockTableCleanup twice? Can that happen? Or is it just defensive coding to avoid making an additional assumption? lxid there is just a local variable storing

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: If I'm reading the code right, it only modifies __libc_drand48_data on first call, so as long as we called erand48() at least once before spawning the child threads, it would probably work. That seems pretty fragile in the face of the fact that they

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 3:34 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 01.08.2011 17:26, Simon Riggs wrote: On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: I believe we code acquire the locks in right order already, and

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Jul 30, 2011 at 9:08 AM, Robert Haas robertmh...@gmail.com wrote: If I'm reading the code right, it only modifies __libc_drand48_data on first call, so as long as we called erand48() at least once before spawning the child threads, it would

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: I have already got plans for a significantly more sophisticated approach to this. I'd like to move forwards on this capability in this release cycle. I want to be able to tell

[HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
I've been thinking some more about the long-standing problem of the AFTER TRIGGER queue using too much memory, and I think that the situation can be improved by using some basic compression. Currently each event added to the AFTER TRIGGER queue uses 10 bytes per trigger per row for INSERTs and

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:21 AM, Jeff Davis pg...@j-davis.com wrote: On Mon, 2011-08-01 at 08:12 -0400, Robert Haas wrote: Is the LocalTransactionIdIsValid(lxid) a guard against calling VirtualXactLockTableCleanup twice? Can that happen? Or is it just defensive coding to avoid making an

[HACKERS] PgWest CFP extended for 12 days

2011-08-01 Thread Joshua D. Drake
Hello Hackers! The CFP for West has been extended until the 12th of August. Let's get those talks in! https://www.postgresqlconference.org/talk_types JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: I've been thinking some more about the long-standing problem of the AFTER TRIGGER queue using too much memory, and I think that the situation can be improved by using some basic compression. Currently each event added to the AFTER TRIGGER queue

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 17:49, Tom Lane t...@sss.pgh.pa.us wrote: Dean Rasheed dean.a.rash...@gmail.com writes: I've been thinking some more about the long-standing problem of the AFTER TRIGGER queue using too much memory, and I think that the situation can be improved by using some basic

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 4:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: I have already got plans for a significantly more sophisticated approach to this. I'd like to move forwards on this

Re: [HACKERS] error: could not find pg_class tuple for index 2662

2011-08-01 Thread Tom Lane
daveg da...@sonic.net writes: On Sun, Jul 31, 2011 at 11:44:39AM -0400, Tom Lane wrote: I think we need to start adding some instrumentation so we can get a better handle on what's going on in your database. If I were to send you a source-code patch for the server that adds some more logging

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: On 1 August 2011 17:49, Tom Lane t...@sss.pgh.pa.us wrote: Ummm ... I only read the data structure comments, not the code, but I don't see where you store the second CTID for an update event? Ah yes, I forgot to mention that bit. I'm using

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 1:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dean Rasheed dean.a.rash...@gmail.com writes: On 1 August 2011 17:49, Tom Lane t...@sss.pgh.pa.us wrote: Ummm ... I only read the data structure comments, not the code, but I don't see where you store the second CTID for an

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: One of the things I was looking at doing was allowing the operator estimation functions mark the plan as one-shot if they used non-uniform data to predict the estimate. That would require most functions to observe the rule that if a plan is marked

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 18:36, Robert Haas robertmh...@gmail.com wrote: On Mon, Aug 1, 2011 at 1:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dean Rasheed dean.a.rash...@gmail.com writes: On 1 August 2011 17:49, Tom Lane t...@sss.pgh.pa.us wrote: Ummm ... I only read the data structure comments, not the

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think you had better plan on incorporating GNU readline into installer builds for Lion. Unfortunately the licence makes that a non-starter. What is it about a

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: H ... not sure.  It seems a bit scary, but on the other hand we should be able to assume that the updating subtransaction hasn't been rolled back (else surely we shouldn't be firing the trigger).  So in principle

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: Don't we already do that when pruning HOT chains? I thought that only happens after the transaction is committed, and old enough, whereas the trigger code only needs to

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 6:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: One of the things I was looking at doing was allowing the operator estimation functions mark the plan as one-shot if they used non-uniform data to predict the estimate. That would

Re: [HACKERS] WIP fix proposal for bug #6123

2011-08-01 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: By the way, my current patch does break two existing UPDATE statements in the regression test misc.sql file: Fixed in the attached version of the patch. I consider the trigger behavior addressed by this patch to be a bug, and serious enough

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Dave Page
On Mon, Aug 1, 2011 at 6:46 PM, David Fetter da...@fetter.org wrote: On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think you had better plan on incorporating GNU readline into installer builds for Lion.

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Andrew Dunstan
On 08/01/2011 02:17 PM, Dave Page wrote: On Mon, Aug 1, 2011 at 6:46 PM, David Fetterda...@fetter.org wrote: On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lanet...@sss.pgh.pa.us wrote: I think you had better plan on incorporating GNU

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 18:55, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: Don't we already do that when pruning HOT chains? I thought that only happens after the transaction is committed,

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: On 1 August 2011 18:55, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: Don't we already do that when pruning HOT chains? I thought that

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Merlin Moncure
On Mon, Aug 1, 2011 at 10:18 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Aug 1, 2011 at 11:04 AM, David Fetter da...@fetter.org wrote: SQL is Turing-complete, so the chances are excellent that it's possible to express that algorithm in it. ;) Look into common table expressions for

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 19:56, Tom Lane t...@sss.pgh.pa.us wrote: Dean Rasheed dean.a.rash...@gmail.com writes: On 1 August 2011 18:55, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: Don't we

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: OK, so I should split this into 2 patches? Even without the compression, it's probably worth the 16 - 10 byte reduction that would result from removing the 2nd CTID in the UPDATE case, and that part would be a pretty small patch. Yeah, my point

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Alvaro Herrera
Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: 2011/7/29 Tom Lane t...@sss.pgh.pa.us: It would likely be better to not expose the struct type, just individual lookup functions. If so, individual functions to expose a certain property of the supplied object type

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 7:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, this means that Dean is not simply adding some self-contained compression logic; he's eliminating information from the data structure on the grounds that he can get it from elsewhere.  I think that that ought to be

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: 2011/7/29 Tom Lane t...@sss.pgh.pa.us: It would likely be better to not expose the struct type, just individual lookup functions. If

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: On Mon, Aug 1, 2011 at 6:46 PM, David Fetter da...@fetter.org wrote: On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think you had better plan on

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Alvaro Herrera
Excerpts from Robert Haas's message of lun ago 01 16:12:56 -0400 2011: On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: 2011/7/29 Tom Lane t...@sss.pgh.pa.us: It would likely be

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Dave Page
On Monday, August 1, 2011, David Fetter da...@fetter.org wrote: On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: On Mon, Aug 1, 2011 at 6:46 PM, David Fetter da...@fetter.org wrote: On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Andrew Dunstan
On 08/01/2011 04:23 PM, David Fetter wrote: The scope or number of files is irrelevant. Having just a single GPL'd file in those installers makes it impossible for ISVs to bundle them with their products, unless they open source them under a GPL compatible licence. I know of no one, not even

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Tom Lane
David Fetter da...@fetter.org writes: On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: The scope or number of files is irrelevant. Having just a single GPL'd file in those installers makes it impossible for ISVs to bundle them with their products, unless they open source them under

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 09:38:30PM +0100, Dave Page wrote: On Monday, August 1, 2011, David Fetter da...@fetter.org wrote: On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: On Mon, Aug 1, 2011 at 6:46 PM, David Fetter da...@fetter.org wrote: On Sat, Jul 30, 2011 at 08:32:03PM

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Greg Smith
On 07/30/2011 09:08 AM, Robert Haas wrote: If I'm reading the code right, it only modifies __libc_drand48_data on first call, so as long as we called erand48() at least once before spawning the child threads, it would probably work. That seems pretty fragile in the face of the fact that they

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 4:27 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of lun ago 01 16:12:56 -0400 2011: On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Attached is a try at that approach. I don't find this to be a particularly safe idea:  #ifndef HAVE_ERAND48 -/* we assume all of these are present or missing together */ -extern double erand48(unsigned short xseed[3]);