Re: [HACKERS] Bitmap index thoughts (another segfault)

2007-04-08 Thread Gavin Sherry
On Sat, 7 Apr 2007, Mark Kirkwood wrote: > Mark Kirkwood wrote: > bitmap=# SELECT count(*) FROM bitmaptest > WHERE val1 in (1,7) > AND val0 IN (4,3) > ; > > ERROR: XX000: unknown stream type 2 > LOCATION: stream_add_node, tidbitmap.c:1033 Thanks. Turned out t

Re: [HACKERS] Bitmap index thoughts (another segfault)

2007-04-08 Thread Gavin Sherry
> I'm seeing a segfault on a size TPC-H size 10 database. The patch and > code are: > - bitmap patch from 12 Mar > - 8.3 dev from 27 Mar Thanks Mark. I tracked this down. I'll post a new patch soon. Gavin ---(end of broadcast)--- TIP 6: explain an

Re: [HACKERS] Bitmap index thoughts (another segfault)

2007-04-06 Thread Mark Kirkwood
Mark Kirkwood wrote: I'm seeing a segfault on a size TPC-H size 10 database. The patch and code are: - bitmap patch from 12 Mar - 8.3 dev from 27 Mar SELECT count(distinct(o_orderkey)) FROM orders orders_alias WHERE o_orderpriority IN ('1-URGENT', '3-MEDIUM') AND o_orderstatus='P'; (gdb) bt #

Re: [HACKERS] Bitmap index thoughts (another segfault)

2007-04-05 Thread Mark Kirkwood
I'm seeing a segfault on a size TPC-H size 10 database. The patch and code are: - bitmap patch from 12 Mar - 8.3 dev from 27 Mar SELECT count(distinct(o_orderkey)) FROM orders orders_alias WHERE o_orderpriority IN ('1-URGENT', '3-MEDIUM') AND o_orderstatus='P'; (gdb) bt #0 0x in ?? ()

Re: [HACKERS] Bitmap index thoughts

2007-02-08 Thread Gavin Sherry
On Thu, 8 Feb 2007, Heikki Linnakangas wrote: > Gavin Sherry wrote: > > I will update the code tomorrow. The focus will be cleaning up the > > executor modifications. Please look else where for now. > > I'm getting a segfault with this test script: > > > CREATE TABLE bmtest (i int); > > I

Re: [HACKERS] Bitmap index thoughts

2007-02-08 Thread Heikki Linnakangas
Gavin Sherry wrote: I will update the code tomorrow. The focus will be cleaning up the executor modifications. Please look else where for now. I'm getting a segfault with this test script: CREATE TABLE bmtest (i int); INSERT INTO bmtest SELECT 1 FROM generate_series(1,10) a; INSE

Re: [HACKERS] Bitmap index thoughts

2007-02-06 Thread Heikki Linnakangas
Gavin Sherry wrote: On Thu, 1 Feb 2007, Bruce Momjian wrote: Where are we on this patch? Does it have performance tests to show where it is beneificial? Is it ready to be reviewed? Here's an updated patch: http://www.alcove.com.au/~swm/bitmap-2007-02-02.patch In this patch, I rewrote the

Re: [HACKERS] Bitmap index thoughts

2007-02-02 Thread Gavin Sherry
On Thu, 1 Feb 2007, Bruce Momjian wrote: > > Where are we on this patch? Does it have performance tests to show > where it is beneificial? Is it ready to be reviewed? Here's an updated patch: http://www.alcove.com.au/~swm/bitmap-2007-02-02.patch In this patch, I rewrote the index build system

Re: [HACKERS] Bitmap index thoughts

2007-02-01 Thread Gavin Sherry
On Thu, 1 Feb 2007, Bruce Momjian wrote: > > Where are we on this patch? Does it have performance tests to show > where it is beneificial? Is it ready to be reviewed? I've got an updated patch which adds significant performance improvements for worse case data distributions. It also contains a

Re: [HACKERS] Bitmap index thoughts

2007-02-01 Thread Bruce Momjian
Where are we on this patch? Does it have performance tests to show where it is beneificial? Is it ready to be reviewed? --- Heikki Linnakangas wrote: > I've been skimming through the bitmap index patch... > > A scan needs

Re: [HACKERS] Bitmap index thoughts

2006-12-28 Thread Jie Zhang
On 12/27/06 3:16 AM, "Gavin Sherry" <[EMAIL PROTECTED]> wrote: > On Wed, 27 Dec 2006, Heikki Linnakangas wrote: > >> Jie Zhang wrote: >>> The "bitmap data segment" sounds good in terms of space. The problem is that >>> one bitmap is likely to occupy more pages than before, which may hurt the >>>

Re: [HACKERS] Bitmap index thoughts

2006-12-28 Thread Jie Zhang
On 12/27/06 3:10 AM, "Gavin Sherry" <[EMAIL PROTECTED]> wrote: > On Wed, 27 Dec 2006, Heikki Linnakangas wrote: > >> Gavin Sherry wrote: >>> On Tue, 26 Dec 2006, Heikki Linnakangas wrote: for typical bitmap index use cases and most of the needed pages should stay in memory, but could w

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Simon Riggs
On Wed, 2006-12-27 at 22:16 +1100, Gavin Sherry wrote: > > But actually I'm not convinced we need to worry about efficient storage > > of small bitmaps at all. The typical use case for bitmap indexes is > > large tables with small number of distinct values, and the problem > > doesn't really arise

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread mark
On Wed, Dec 27, 2006 at 03:42:36PM +, Heikki Linnakangas wrote: > >I wonder what would happen if somebody implemented automatic index > >exclusion conditions after use of an INDEX proved to be in the realm > >of the worst case scenario? :-) > I'm sorry, I don't understand that sentence... I wa

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Heikki Linnakangas
[EMAIL PROTECTED] wrote: On Wed, Dec 27, 2006 at 10:16:54PM +1100, Gavin Sherry wrote: On Wed, 27 Dec 2006, Heikki Linnakangas wrote: But actually I'm not convinced we need to worry about efficient storage of small bitmaps at all. The typical use case for bitmap indexes is large tables with sma

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread mark
On Wed, Dec 27, 2006 at 10:16:54PM +1100, Gavin Sherry wrote: > On Wed, 27 Dec 2006, Heikki Linnakangas wrote: > > But actually I'm not convinced we need to worry about efficient storage > > of small bitmaps at all. The typical use case for bitmap indexes is > > large tables with small number of di

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Gavin Sherry
On Wed, 27 Dec 2006, Heikki Linnakangas wrote: > Jie Zhang wrote: > > The "bitmap data segment" sounds good in terms of space. The problem is that > > one bitmap is likely to occupy more pages than before, which may hurt the > > query performance. > > We could have segments of say 1/5 of page. Whe

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Gavin Sherry
On Wed, 27 Dec 2006, Heikki Linnakangas wrote: > Gavin Sherry wrote: > > On Tue, 26 Dec 2006, Heikki Linnakangas wrote: > >> for typical bitmap index use cases and most of the needed pages should > >> stay in memory, but could we simplify this? Why do we need the auxiliary > >> heap, couldn't we j

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Heikki Linnakangas
Jie Zhang wrote: The "bitmap data segment" sounds good in terms of space. The problem is that one bitmap is likely to occupy more pages than before, which may hurt the query performance. We could have segments of say 1/5 of page. When a bitmap grows larger than that, the bitmap would be moved

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Heikki Linnakangas
Gavin Sherry wrote: On Tue, 26 Dec 2006, Heikki Linnakangas wrote: for typical bitmap index use cases and most of the needed pages should stay in memory, but could we simplify this? Why do we need the auxiliary heap, couldn't we just store the blk+offset of the LOV item directly in the b-tree in

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Jie Zhang
>> And instead of having separate LOV pages that store a number of LOV >> items, how about storing each LOV item on a page of it's own, and using >> the rest of the page to store the last chunk of the bitmap. That would >> eliminate one page access, but more importantly, maybe we could then get >>

Re: [HACKERS] Bitmap index thoughts

2006-12-26 Thread Gavin Sherry
Hey Heikki, On Tue, 26 Dec 2006, Heikki Linnakangas wrote: > I've been skimming through the bitmap index patch... > > A scan needs to access at least five pages: > > 1. B-tree index (root+others, depending on depth) > 2. The auxiliary heap page > 3. bitmap index meta page > 4. LOV page > 5. bitma

[HACKERS] Bitmap index thoughts

2006-12-26 Thread Heikki Linnakangas
I've been skimming through the bitmap index patch... A scan needs to access at least five pages: 1. B-tree index (root+others, depending on depth) 2. The auxiliary heap page 3. bitmap index meta page 4. LOV page 5. bitmap page That seems like a lot of indirection. A high startup cost is probabl