Re: BRIN indexes (was Re: [HACKERS] Minmax indexes)

2014-09-14 Thread Emanuel Calvo
El 08/09/14 13:02, Alvaro Herrera escribió: > Here's version 18. I have renamed it: These are now BRIN indexes. > > I have fixed numerous race conditions and deadlocks. In particular I > fixed this problem you noted: > > Heikki Linnakangas wrote: >> Another race condition: >> >> If a new tuple i

Re: BRIN indexes (was Re: [HACKERS] Minmax indexes)

2014-09-09 Thread Heikki Linnakangas
On 09/08/2014 07:02 PM, Alvaro Herrera wrote: Here's version 18. I have renamed it: These are now BRIN indexes. I have fixed numerous race conditions and deadlocks. In particular I fixed this problem you noted: Heikki Linnakangas wrote: Another race condition: If a new tuple is inserted to

Re: [HACKERS] Minmax indexes

2014-08-20 Thread Alvaro Herrera
Alvaro Herrera wrote: > So here's v16, rebased on top of 9bac66020. As far as I am concerned, > this is the last version before I start renaming everything to BRIN and > then commit. FWIW in case you or others have interest, here's the diff between your patch and v16. Also, for illustrative pur

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Alvaro Herrera
Fujii Masao wrote: > I've not read the patch yet. But while testing the feature, I found that > > * Brin index cannot be created on CHAR(n) column. >Maybe other data types have the same problem. Yeah, it's just a matter of adding an opclass for it -- pretty simple stuff really, because you d

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much better. The insert and update code is somewhat mor

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Fujii Masao
On Fri, Aug 15, 2014 at 8:02 AM, Alvaro Herrera wrote: > Alvaro Herrera wrote: >> Heikki Linnakangas wrote: > >> > I'm sure this still needs some cleanup, but here's the patch, based >> > on your v14. Now that I know what this approach looks like, I still >> > like it much better. The insert and u

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 10:26 AM, Heikki Linnakangas wrote: On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much better. The insert and update code is somewhat mor

Re: [HACKERS] Minmax indexes

2014-08-12 Thread Alvaro Herrera
Heikki Linnakangas wrote: > I couldn't resist starting to hack on this, and implemented the > scheme I've been having in mind: > > 1. MMTuple contains the block number of the heap page (range) that > the tuple represents. Vacuum is no longer needed to clean up old > tuples; when an index tuples is

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Claudio Freire
On Fri, Aug 8, 2014 at 6:01 AM, Heikki Linnakangas wrote: > It's possible that two backends arrive at phase 3 at the same time, with > different values. For example, backend A wants to update the minimum to > contain 10, and and backend B wants to update it to 5. Now, if backend B > gets to update

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Heikki Linnakangas
On 08/10/2014 12:42 PM, Simon Riggs wrote: On 8 August 2014 16:03, Heikki Linnakangas wrote: I couldn't resist starting to hack on this, and implemented the scheme I've been having in mind: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no l

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Heikki Linnakangas
On 08/10/2014 12:22 PM, Simon Riggs wrote: On 8 August 2014 16:03, Heikki Linnakangas wrote: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no longer needed to clean up old tuples; when an index tuples is updated, the old tuple is deleted ato

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 16:03, Heikki Linnakangas wrote: > I couldn't resist starting to hack on this, and implemented the scheme I've > been having in mind: > > 1. MMTuple contains the block number of the heap page (range) that the tuple > represents. Vacuum is no longer needed to clean up old tuples;

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 10:01, Heikki Linnakangas wrote: > It's possible that two backends arrive at phase 3 at the same time, with > different values. For example, backend A wants to update the minimum to > contain 10, and and backend B wants to update it to 5. Now, if backend B > gets to update the tu

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 16:03, Heikki Linnakangas wrote: > 1. MMTuple contains the block number of the heap page (range) that the tuple > represents. Vacuum is no longer needed to clean up old tuples; when an index > tuples is updated, the old tuple is deleted atomically with the insertion of > a new tu

Re: [HACKERS] Minmax indexes

2014-08-08 Thread Heikki Linnakangas
Another race condition: If a new tuple is inserted to the range while summarization runs, it's possible that the new tuple isn't included in the tuple that the summarization calculated, nor does the insertion itself udpate it. 1. There is no index tuple for page range 1-10 2. Summarization be

Re: [HACKERS] Minmax indexes

2014-08-08 Thread Heikki Linnakangas
I think there's a race condition in mminsert, if two backends insert a tuple to the same heap page range concurrently. mminsert does this: 1. Fetch the MMtuple for the page range 2. Check if any of the stored datums need updating 3. Unlock the page. 4. Lock the page again in exclusive mode. 5. U

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Josh Berkus
On 08/07/2014 05:52 PM, Michael Paquier wrote: > On Fri, Aug 8, 2014 at 9:47 AM, Josh Berkus wrote: >> On 08/07/2014 08:38 AM, Oleg Bartunov wrote: >>> +1 for BRIN ! >>> >>> On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs wrote: On 7 August 2014 14:53, Robert Haas wrote: A better descripti

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Peter Geoghegan
On Thu, Aug 7, 2014 at 7:58 AM, Robert Haas wrote: > range index might get confused with range types; block range index > seems better. I like summary, but I'm fine with block range index or > block filter index, too. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Michael Paquier
On Fri, Aug 8, 2014 at 9:47 AM, Josh Berkus wrote: > On 08/07/2014 08:38 AM, Oleg Bartunov wrote: >> +1 for BRIN ! >> >> On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs wrote: >>> On 7 August 2014 14:53, Robert Haas wrote: >>> A better description would be "block range index" since we are >>> indexi

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Josh Berkus
On 08/07/2014 08:38 AM, Oleg Bartunov wrote: > +1 for BRIN ! > > On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs wrote: >> On 7 August 2014 14:53, Robert Haas wrote: >> A better description would be "block range index" since we are >> indexing a range of blocks (not just one block). Perhaps a better

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Petr Jelinek
On 07/08/14 16:16, Simon Riggs wrote: A better description would be "block range index" since we are indexing a range of blocks (not just one block). Perhaps a better one would be simply "range index", which we could abbreviate to RIN or BRIN. +1 for block range index (BRIN) -- Petr Jelinek

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Nicolas Barbier
2014-08-07 Oleg Bartunov : > +1 for BRIN ! +1, rolls off the tongue smoothly and captures the essence :-). Nicolas -- A. Because it breaks the logical sequence of discussion. Q. Why is top posting bad? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Oleg Bartunov
+1 for BRIN ! On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs wrote: > On 7 August 2014 14:53, Robert Haas wrote: >> On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier >> wrote: >>> 2014-08-06 Claudio Freire : >>> So, I like blockfilter a lot. I change my vote to blockfilter ;) >>> >>> +1 for blo

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Robert Haas
On Thu, Aug 7, 2014 at 10:16 AM, Simon Riggs wrote: > On 7 August 2014 14:53, Robert Haas wrote: >> On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier >> wrote: >>> 2014-08-06 Claudio Freire : >>> So, I like blockfilter a lot. I change my vote to blockfilter ;) >>> >>> +1 for blockfilter, beca

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Alvaro Herrera
Simon Riggs wrote: > On 7 August 2014 14:53, Robert Haas wrote: > > On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier > > wrote: > >> 2014-08-06 Claudio Freire : > >> > >>> So, I like blockfilter a lot. I change my vote to blockfilter ;) > >> > >> +1 for blockfilter, because it stresses the fact th

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Claudio Freire
On Thu, Aug 7, 2014 at 11:16 AM, Simon Riggs wrote: > On 7 August 2014 14:53, Robert Haas wrote: >> On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier >> wrote: >>> 2014-08-06 Claudio Freire : >>> So, I like blockfilter a lot. I change my vote to blockfilter ;) >>> >>> +1 for blockfilter, beca

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Simon Riggs
On 7 August 2014 14:53, Robert Haas wrote: > On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier > wrote: >> 2014-08-06 Claudio Freire : >> >>> So, I like blockfilter a lot. I change my vote to blockfilter ;) >> >> +1 for blockfilter, because it stresses the fact that the "physical" >> arrangement of

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Robert Haas
On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier wrote: > 2014-08-06 Claudio Freire : > >> So, I like blockfilter a lot. I change my vote to blockfilter ;) > > +1 for blockfilter, because it stresses the fact that the "physical" > arrangement of rows in blocks matters for this index. I don't like

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Nicolas Barbier
2014-08-06 Claudio Freire : > So, I like blockfilter a lot. I change my vote to blockfilter ;) +1 for blockfilter, because it stresses the fact that the "physical" arrangement of rows in blocks matters for this index. Nicolas -- A. Because it breaks the logical sequence of discussion. Q. Why i

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:55 PM, Alvaro Herrera wrote: > Claudio Freire wrote: >> On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera >> wrote: >> > CREATE INDEX foo ON t USING crange (cols) -- misspelling of "cringe"? >> > CREATE INDEX foo ON t USING comprange (cols) >> > CREATE INDEX foo ON t USIN

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:35 PM, Bruce Momjian wrote: > On Wed, Aug 6, 2014 at 01:31:14PM -0300, Claudio Freire wrote: >> On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera >> wrote: >> > CREATE INDEX foo ON t USING crange (cols) -- misspelling of "cringe"? >> > CREATE INDEX foo ON t USING compran

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera wrote: > CREATE INDEX foo ON t USING crange (cols) -- misspelling of "cringe"? > CREATE INDEX foo ON t USING comprange (cols) > CREATE INDEX foo ON t USING compressedrng (cols) -- ugh > -- or use an identifier with whitespace: > CREATE INDEX foo O

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Bruce Momjian
On Wed, Aug 6, 2014 at 01:31:14PM -0300, Claudio Freire wrote: > On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera > wrote: > > CREATE INDEX foo ON t USING crange (cols) -- misspelling of "cringe"? > > CREATE INDEX foo ON t USING comprange (cols) > > CREATE INDEX foo ON t USING compressedrng (col

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera wrote: > "Summary" seems good. If I get enough votes I can change it to that. > > CREATE INDEX foo ON t USING summary (cols) > > "Summarizing" seems weird on that command. Not sure about "compressed > range", as you would have to use an abbreviation

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Robert Haas
On Tue, Aug 5, 2014 at 7:55 PM, Josh Berkus wrote: > On 08/05/2014 04:41 PM, Alvaro Herrera wrote: >> I have chosen to keep the name "minmax", even if the opclasses now let >> one implement completely different things on top of it such as geometry >> bounding boxes and bloom filters (aka bitmap in

Re: [HACKERS] Minmax indexes

2014-08-05 Thread Alvaro Herrera
FWIW I think I haven't responded appropriately to the points raised by Heikki. Basically, as I see it there are three main items: 1. the revmap physical-to-logical mapping is too complex; let's use something else. We had revmap originally in a separate fork. The current approach grew out of the

Re: [HACKERS] Minmax indexes

2014-08-05 Thread Josh Berkus
On 08/05/2014 04:41 PM, Alvaro Herrera wrote: > I have chosen to keep the name "minmax", even if the opclasses now let > one implement completely different things on top of it such as geometry > bounding boxes and bloom filters (aka bitmap indexes). I don't see a > need for a rename: essentially,

Re: [HACKERS] Minmax indexes

2014-07-29 Thread Heikki Linnakangas
On 07/10/2014 12:41 AM, Alvaro Herrera wrote: Heikki Linnakangas wrote: On 06/23/2014 08:07 PM, Alvaro Herrera wrote: I feel that the below would nevertheless be simpler: I wonder if it would be simpler to just always store the revmap pages in the beginning of the index, before any other pa

Re: [HACKERS] Minmax indexes

2014-07-14 Thread Robert Haas
On Wed, Jul 9, 2014 at 5:16 PM, Alvaro Herrera wrote: > The way it works now, each opclass needs to have three support > procedures; I've called them getOpers, maybeUpdateValues, and compare. > (I realize these names are pretty bad, and will be changing them.) > getOpers is used to obtain informat

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Fri, Jul 11, 2014 at 3:47 PM, Greg Stark wrote: > On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire > wrote: >> Marking as read-only is ok, or emitting a NOTICE so that if anyone >> changes those parameters that change the shape of the index, they know >> it needs a rebuild would be OK too. Bot

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Greg Stark
On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire wrote: > Marking as read-only is ok, or emitting a NOTICE so that if anyone > changes those parameters that change the shape of the index, they know > it needs a rebuild would be OK too. Both mechanisms work for me. We don't actually have any of the

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Thu, Jul 10, 2014 at 4:20 PM, Alvaro Herrera wrote: > Claudio Freire wrote: >> On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera >> wrote: >> > Another thing I noticed is that version 8 of the patch blindly believed >> > the "pages_per_range" declared in catalogs. This meant that if somebody >>

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Alvaro Herrera
Fujii Masao wrote: > On Thu, Jul 10, 2014 at 6:16 AM, Alvaro Herrera > wrote: > > Here's a new version of this patch, which is more generic the original > > versions, and similar to what you describe. > > I've not read the discussion so far at all, but I found the problem > when I played with th

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 10 July 2014 00:13, Alvaro Herrera wrote: > Josh Berkus wrote: >> On 07/09/2014 02:16 PM, Alvaro Herrera wrote: >> > The way it works now, each opclass needs to have three support >> > procedures; I've called them getOpers, maybeUpdateValues, and compare. >> > (I realize these names are pretty

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 9 July 2014 23:54, Peter Geoghegan wrote: > On Wed, Jul 9, 2014 at 2:16 PM, Alvaro Herrera > wrote: >> All this being said, I'm sticking to the name "Minmax indexes". There >> was a poll in pgsql-advocacy >> http://www.postgresql.org/message-id/53a0b4f8.8080...@agliodbs.com >> about a new na

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Fujii Masao
On Thu, Jul 10, 2014 at 6:16 AM, Alvaro Herrera wrote: > Claudio Freire wrote: > >> An aggregate to generate a "compressed set" from several values >> A function which adds a new value to the "compressed set" and returns >> the new "compressed set" >> A function which tests if a value is in a "com

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Greg Stark
On Thu, Jul 10, 2014 at 10:29 PM, Alvaro Herrera wrote: > > What I think should happen is that if the value is changed, the index > sholud be rebuilt right there. I disagree. It would be a non-orthogonal interface if ALTER TABLE sometimes causes the index to be rebuilt and sometimes just makes a

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Jeff Janes
On Thu, Jul 10, 2014 at 2:30 PM, Jaime Casanova wrote: > On Thu, Jul 10, 2014 at 3:50 PM, Josh Berkus wrote: > > On 07/10/2014 12:20 PM, Alvaro Herrera wrote: > >>> So I guess the only thing left is to issue a NOTICE when said alter > >>> > takes place (I don't see that on the patch, but maybe i

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Josh Berkus
On 07/10/2014 02:30 PM, Jaime Casanova wrote: > How is this different from "ALTER TABLE foo SET (FILLFACTOR=80); " or > from "ALTER TABLE foo ALTER bar SET STORAGE EXTERNAL; " ? > > we don't get a notice for these cases either Good idea. We should also emit notices for those. Well, maybe not fo

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Alvaro Herrera
Josh Berkus wrote: > On 07/10/2014 12:20 PM, Alvaro Herrera wrote: > >> So I guess the only thing left is to issue a NOTICE when said alter > >> > takes place (I don't see that on the patch, but maybe it's there?) > > That's not in the patch. I don't think we have an appropriate place to > > emit

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Jaime Casanova
On Thu, Jul 10, 2014 at 3:50 PM, Josh Berkus wrote: > On 07/10/2014 12:20 PM, Alvaro Herrera wrote: >>> So I guess the only thing left is to issue a NOTICE when said alter >>> > takes place (I don't see that on the patch, but maybe it's there?) >> That's not in the patch. I don't think we have an

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Josh Berkus
On 07/10/2014 12:20 PM, Alvaro Herrera wrote: >> So I guess the only thing left is to issue a NOTICE when said alter >> > takes place (I don't see that on the patch, but maybe it's there?) > That's not in the patch. I don't think we have an appropriate place to > emit such a notice. What do you m

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Alvaro Herrera
Claudio Freire wrote: > On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera > wrote: > > Another thing I noticed is that version 8 of the patch blindly believed > > the "pages_per_range" declared in catalogs. This meant that if somebody > > did "alter index foo set pages_per_range=123" the index woul

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Claudio Freire
On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera wrote: > Another thing I noticed is that version 8 of the patch blindly believed > the "pages_per_range" declared in catalogs. This meant that if somebody > did "alter index foo set pages_per_range=123" the index would > immediately break (i.e. retur

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Greg Stark
On Wed, Jul 9, 2014 at 10:16 PM, Alvaro Herrera wrote: > there is no hardcoded assumption on the number of index > values stored per heap column, so it is possible to build an opclass > that stores a bounding box column for a geometry heap column, for > instance. I think the more Postgresy thing

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Alvaro Herrera
Josh Berkus wrote: > On 07/09/2014 02:16 PM, Alvaro Herrera wrote: > > The way it works now, each opclass needs to have three support > > procedures; I've called them getOpers, maybeUpdateValues, and compare. > > (I realize these names are pretty bad, and will be changing them.) > > I kind of like

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Peter Geoghegan
On Wed, Jul 9, 2014 at 2:16 PM, Alvaro Herrera wrote: > All this being said, I'm sticking to the name "Minmax indexes". There > was a poll in pgsql-advocacy > http://www.postgresql.org/message-id/53a0b4f8.8080...@agliodbs.com > about a new name, but there were no suggestions supported by more tha

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Josh Berkus
On 07/09/2014 02:16 PM, Alvaro Herrera wrote: > The way it works now, each opclass needs to have three support > procedures; I've called them getOpers, maybeUpdateValues, and compare. > (I realize these names are pretty bad, and will be changing them.) I kind of like "maybeUpdateValues". Very ...

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 06/23/2014 08:07 PM, Alvaro Herrera wrote: > I feel that the below would nevertheless be simpler: > > >>I wonder if it would be simpler to just always store the revmap > >>pages in the beginning of the index, before any other pages. Finding > >>the revmap page would

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Robert Haas
On Thu, Jun 19, 2014 at 12:32 PM, Greg Stark wrote: > On Wed, Jun 18, 2014 at 4:51 AM, Heikki Linnakangas > wrote: >> Implementing something is a good way to demonstrate how it would look like. >> But no, I don't insist on implementing every possible design whenever a new >> feature is proposed.

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Heikki Linnakangas
On 06/23/2014 08:07 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: With 8k block size, that's just enough to cover the full range of 2^32-1 blocks that you'll need if you set mm_pages_per_range=1. Each regular revmap page can store about 8192/6 = 1365 item pointers, each array revmap page c

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Alvaro Herrera
Heikki Linnakangas wrote: > Some comments, aside from the design wrt. bounding boxes etc. : Thanks. I haven't commented on that sub-thread because I think it's possible to come up with a reasonable design that solves the issue by adding a couple of amprocs. I need to do some more thinking to ens

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Heikki Linnakangas
Some comments, aside from the design wrt. bounding boxes etc. : On 06/15/2014 05:34 AM, Alvaro Herrera wrote: Robert Haas wrote: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far. Thanks to Thom Brown, J

Re: [HACKERS] Minmax indexes

2014-06-21 Thread Martijn van Oosterhout
I'm sorry if I missed something, but ISTM this is beginning to look a lot like GiST. This was pointed out by Robert Haas last year. On Wed, Jun 18, 2014 at 12:09:42PM -0300, Claudio Freire wrote: > So, you have: > > An aggregate to generate a "compressed set" from several values Which GiST does

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Claudio Freire
On Wed, Jun 18, 2014 at 8:51 AM, Heikki Linnakangas wrote: > > I liked Greg's sketch of what the opclass support functions would be. It > doesn't seem significantly more complicated than what's in the patch now. Which was On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark wrote: > An aggregate to gen

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Claudio Freire
On Thu, Jun 19, 2014 at 10:06 AM, Heikki Linnakangas wrote: > On 06/18/2014 06:09 PM, Claudio Freire wrote: >> >> On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark wrote: >>> >>> An aggregate to generate a min/max "bounding box" from several values >>> A function which takes an "bounding box" and a new

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Greg Stark
On Wed, Jun 18, 2014 at 4:51 AM, Heikki Linnakangas wrote: > Implementing something is a good way to demonstrate how it would look like. > But no, I don't insist on implementing every possible design whenever a new > feature is proposed. > > I liked Greg's sketch of what the opclass support functi

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Tom Lane
Vik Fearing writes: > On 06/18/2014 12:46 PM, Andres Freund wrote: >> So to implement a feature one now has to implement the most generic >> variant as a prototype first? Really? > Well, there is the inventor's paradox to consider. I have not seen anyone demanding a different implementation in t

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Heikki Linnakangas
On 06/18/2014 06:09 PM, Claudio Freire wrote: On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark wrote: An aggregate to generate a min/max "bounding box" from several values A function which takes an "bounding box" and a new value and returns the new "bounding box" A function which tests if a value is

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Vik Fearing
On 06/18/2014 12:46 PM, Andres Freund wrote: >>> Isn't 'simpler implementation' a valid reason that's already been >>> > >discussed onlist? Obviously simpler implementation doesn't trump >>> > >everything, but it's one valid reason... >>> > >Note that I have zap to do with the design of this featur

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Robert Haas
On Tue, Jun 17, 2014 at 2:16 PM, Andres Freund wrote: >> But I'm not trying to say that we absolutely have to support that kind >> of thing; what I am trying to say is that there should be a README or >> a mailing list post or some such that says: "We thought about how >> generic to make this. We

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Heikki Linnakangas
On 06/18/2014 01:46 PM, Andres Freund wrote: On 2014-06-18 12:18:26 +0300, Heikki Linnakangas wrote: The main problem with using it for geometric types is that you can't easily CLUSTER the table to make the minmax index effective again. But there are ways around that. Which are? Sure you can t

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Andres Freund
On 2014-06-17 16:48:07 -0700, Greg Stark wrote: > On Tue, Jun 17, 2014 at 11:16 AM, Andres Freund > wrote: > > Well, it might be doable to correlate them along one axis, but along > > both? That's more complicated... And even alongside one axis you > > already get into problems if your geometrie

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Andres Freund
On 2014-06-18 12:18:26 +0300, Heikki Linnakangas wrote: > On 06/17/2014 09:16 PM, Andres Freund wrote: > >Well, it might be doable to correlate them along one axis, but along > >both? That's more complicated... And even alongside one axis you > >already get into problems if your geometries are irr

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Heikki Linnakangas
On 06/17/2014 09:16 PM, Andres Freund wrote: On 2014-06-17 12:14:00 -0400, Robert Haas wrote: On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund wrote: Well, I'm not the guy who does things with geometric data, but I don't want to ignore the significant percentage of our users who are. As you mu

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Greg Stark
On Tue, Jun 17, 2014 at 11:16 AM, Andres Freund wrote: > Well, it might be doable to correlate them along one axis, but along > both? That's more complicated... And even alongside one axis you > already get into problems if your geometries are irregularly sized. > Asingle large polygon will compl

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Josh Berkus
On 06/17/2014 09:14 AM, Robert Haas wrote: > Well, I don't know: suppose you're loading geospatial data showing the > location of every building in some country. It might easily be the > case that the data is or can be loaded in an order that provides > pretty good spatial locality, leading to tig

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Claudio Freire
On Tue, Jun 17, 2014 at 1:04 PM, Andres Freund wrote: > For me minmax indexes are helpful because they allow to generate *small* > 'coarse' indexes over large volumes of data. From my pov that's possible > possible because they don't contain item pointers for every contained > row. But minmax is

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 12:14:00 -0400, Robert Haas wrote: > On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund > wrote: > >> Well, I'm not the guy who does things with geometric data, but I don't > >> want to ignore the significant percentage of our users who are. As > >> you must surely know, the GIST impl

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund wrote: >> Well, I'm not the guy who does things with geometric data, but I don't >> want to ignore the significant percentage of our users who are. As >> you must surely know, the GIST implementations for geometric data >> types store bounding boxes

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 11:48:10 -0400, Robert Haas wrote: > On Tue, Jun 17, 2014 at 10:31 AM, Andres Freund > wrote: > > On 2014-06-17 10:26:11 -0400, Robert Haas wrote: > >> On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera > >> wrote: > >> > Robert Haas wrote: > >> >> On Wed, Sep 25, 2013 at 4:34 PM, Al

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Tue, Jun 17, 2014 at 10:31 AM, Andres Freund wrote: > On 2014-06-17 10:26:11 -0400, Robert Haas wrote: >> On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera >> wrote: >> > Robert Haas wrote: >> >> On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera >> >> wrote: >> >> > Here's an updated version of

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Greg Stark
On Tue, Jun 17, 2014 at 3:31 PM, Andres Freund wrote: > Is there actually a significant usecase behind that wish or just a > general demand for being generic? To me it seems fairly unlikely you'd > end up with something useful by doing a minmax index over bounding > boxes. Isn't min/max just a 2d

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 10:26:11 -0400, Robert Haas wrote: > On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera > wrote: > > Robert Haas wrote: > >> On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera > >> wrote: > >> > Here's an updated version of this patch, with fixes to all the bugs > >> > reported so far.

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera >> wrote: >> > Here's an updated version of this patch, with fixes to all the bugs >> > reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and >> > Ami

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Greg Stark
On Fri, Jan 24, 2014 at 12:58 PM, Claudio Freire wrote: > > What's the status? > > I believe I have more than a use for minmax indexes, and wouldn't mind > lending a hand if it's within my grasp. I'm also interested in looking at this. Mostly because I have ideas for other "summary" functions th

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Claudio Freire
On Fri, Jan 24, 2014 at 2:54 PM, Thom Brown wrote: > On 24 January 2014 17:53, Alvaro Herrera wrote: >> Thom Brown wrote: >>> On 8 November 2013 20:11, Alvaro Herrera wrote: >>> > Erik Rijkers wrote: >>> >> On Thu, September 26, 2013 00:34, Erik Rijkers wrote: >>> >> > On Wed, September 25, 2013

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Thom Brown
On 24 January 2014 17:53, Alvaro Herrera wrote: > Thom Brown wrote: >> On 8 November 2013 20:11, Alvaro Herrera wrote: >> > Erik Rijkers wrote: >> >> On Thu, September 26, 2013 00:34, Erik Rijkers wrote: >> >> > On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: >> >> > >> >> >> [minmax-5.pa

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Alvaro Herrera
Thom Brown wrote: > On 8 November 2013 20:11, Alvaro Herrera wrote: > > Erik Rijkers wrote: > >> On Thu, September 26, 2013 00:34, Erik Rijkers wrote: > >> > On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: > >> > > >> >> [minmax-5.patch] > >> > > >> > I have the impression it's not quite w

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Thom Brown
On 8 November 2013 20:11, Alvaro Herrera wrote: > Erik Rijkers wrote: >> On Thu, September 26, 2013 00:34, Erik Rijkers wrote: >> > On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: >> > >> >> [minmax-5.patch] >> > >> > I have the impression it's not quite working correctly. > > Here's a ver

Re: [HACKERS] Minmax indexes

2013-11-15 Thread Jeff Janes
On Fri, Nov 8, 2013 at 12:11 PM, Alvaro Herrera wrote: > Erik Rijkers wrote: > > On Thu, September 26, 2013 00:34, Erik Rijkers wrote: > > > On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: > > > > > >> [minmax-5.patch] > > > > > > I have the impression it's not quite working correctly. > >

Re: [HACKERS] Minmax indexes (timings)

2013-11-15 Thread Kevin Grittner
Erik Rijkers wrote: > Perhaps someone finds these timings useful. > '--enable-cassert' Assertions can really distort the timings, and not always equally for all code paths.  Any chance of re-running those tests without that? -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise Pos

Re: [HACKERS] Minmax indexes (timings)

2013-11-15 Thread Andres Freund
On 2013-11-15 17:11:46 +0100, Erik Rijkers wrote: > I've been messing with minmax indexes some more so here are some results of > that. > > Perhaps someone finds these timings useful. > > > Centos 5.7, 32 GB memory, 2 quadcores. > > '--prefix=/var/data1/pg_stuff/pg_installations/pgsql.minmax'

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Jeff Janes
On Mon, Nov 11, 2013 at 12:53 AM, Erik Rijkers wrote: > On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: > > > > Here's a version 7 of the patch, which fixes these bugs and adds > > opclasses for a bunch more types (timestamp, timestamptz, date, time, > > timetz), courtesy of Martín Marqués.

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Erik Rijkers
On Mon, November 11, 2013 09:53, Erik Rijkers wrote: > On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: >> >> Here's a version 7 of the patch, which fixes these bugs and adds >> opclasses for a bunch more types (timestamp, timestamptz, date, time, >> timetz), courtesy of Martín Marqués. It's

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Erik Rijkers
On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: > > Here's a version 7 of the patch, which fixes these bugs and adds > opclasses for a bunch more types (timestamp, timestamptz, date, time, > timetz), courtesy of Martín Marqués. It's also been rebased to apply > cleanly on top of today's mast

Re: [HACKERS] Minmax indexes

2013-11-08 Thread Alvaro Herrera
Alvaro Herrera escribió: > I have been playing with having the revmap in the main fork of the index > rather than a separate one. ... > This is not complete yet; although I have a proof-of-concept working, I > still need to write XLog support code and update the pageinspect code to > match. Just

Re: [HACKERS] Minmax indexes

2013-11-08 Thread Alvaro Herrera
Robert Haas escribió: > On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera > wrote: > > Here's an updated version of this patch, with fixes to all the bugs > > reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and > > Amit Kapila for the reports. > > I'm not very happy with the us

Re: [HACKERS] Minmax indexes

2013-10-01 Thread Robert Haas
On Mon, Sep 30, 2013 at 1:20 PM, Heikki Linnakangas wrote: > You can almost create a bounding box opclass in the current implementation, > by mapping < operator to "contains" and > to "not contains". But there's no > support for creating a new, larger, bounding box on insert. It will just > replac

  1   2   >