Re: [HACKERS] bitmap AM design

2005-03-05 Thread Pailloncy Jean-Gerard
Pailloncy Jean-Gerard wrote: You should have a look to this thread http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php Take a look at this paper about lock-free parallel hash table http://www.cs.rug.nl/~wim/mechver/hashtable/ Is this relevant? Hash indexes are on-disk data

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Pailloncy Jean-Gerard
Le 2 mars 05, à 21:17, Hannu Krosing a écrit : Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas [EMAIL PROTECTED]: Now, it occurs to me that if my document reference table can refer to something other than an indexed primary key, I can save a lot of index processing time in

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Neil Conway
Pailloncy Jean-Gerard wrote: You should have a look to this thread http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php Take a look at this paper about lock-free parallel hash table http://www.cs.rug.nl/~wim/mechver/hashtable/ Is this relevant? Hash indexes are on-disk data

Re: [HACKERS] bitmap AM design

2005-03-04 Thread pgsql
Pailloncy Jean-Gerard wrote: You should have a look to this thread http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php Take a look at this paper about lock-free parallel hash table http://www.cs.rug.nl/~wim/mechver/hashtable/ Is this relevant? Hash indexes are on-disk data

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Tom Lane
Neil Conway [EMAIL PROTECTED] writes: (BTW, is poor concurrency really the biggest issue with hash indexes? If so, there is some low-hanging fruit that I noticed a few years ago, but never got around to fixing: _hash_doinsert() write-locks the hash metapage on every insertion merely to

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Tom Lane
[EMAIL PROTECTED] writes: Anyway, IMHO, hash indexes would be dramatically improved if you could specify your own hashing function That's called a custom operator class. and declare initial table size. It would be interesting to see if setting up the hashtable with about the right number of

Re: [HACKERS] bitmap AM design

2005-03-04 Thread pgsql
[EMAIL PROTECTED] writes: Anyway, IMHO, hash indexes would be dramatically improved if you could specify your own hashing function That's called a custom operator class. Would I also be able to query the bucket size and all that? and declare initial table size. It would be interesting

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Victor Y. Yegorov
Some more thoughts and questions. The main idea above bitmaps is narrowing some data sets' possible values to yes and no (i.e. 1 and 0) and then organize the data in the series of bits, where bit's position determines values to consider. In the cases, where several indexed attributes are used in

Re: [HACKERS] bitmap AM design

2005-03-03 Thread Hannu Krosing
Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas [EMAIL PROTECTED]: Now, it occurs to me that if my document reference table can refer to something other than an indexed primary key, I can save a lot of index processing time in PostgreSQL if I can have a safe analogy to CTID.

Re: [HACKERS] bitmap AM design

2005-03-03 Thread pgsql
Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas [EMAIL PROTECTED]: Now, it occurs to me that if my document reference table can refer to something other than an indexed primary key, I can save a lot of index processing time in PostgreSQL if I can have a safe analogy to

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
I don't think we really need any more fundamentally nonconcurrent index types :-( Tom, I posted a message about a week ago (I forget the name) about a persistent reference index, sort of like CTID, but basically a table lookup. The idea is to simulate a structure that ISAM sort of techniques

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Victor Y. Yegorov
* Tom Lane [EMAIL PROTECTED] [01.03.2005 09:37]: The other stuff you mentioned implies that an insertion therefore requires exclusive lock on the index (because you may have to relocate everything in sight to add one more CTID slot). No, exclusive lock on index is worst thing to do. All lists

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
[EMAIL PROTECTED] writes: Tom, I posted a message about a week ago (I forget the name) about a persistent reference index, sort of like CTID, but basically a table lookup. The idea is to simulate a structure that ISAM sort of techniques can work in PostgreSQL. Eliminating the bitmap index

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
[EMAIL PROTECTED] writes: Tom, I posted a message about a week ago (I forget the name) about a persistent reference index, sort of like CTID, but basically a table lookup. The idea is to simulate a structure that ISAM sort of techniques can work in PostgreSQL. Eliminating the bitmap index

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
[EMAIL PROTECTED] writes: A persistent reference index would be like almost any other index except that as new items are added to a table a new entry is added to the end of the index. When a row is updated, its CTID is updated in the table. This *does not work* under MVCC: it can't cope with

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
Victor Y. Yegorov [EMAIL PROTECTED] writes: All lists (list of ctids, bitmaps) will only grow, no data will be deleted, as deletes will require relocation and possibly exclusive lock on the index. Extending lists will need only a short-term exclusive locks on the pages in the tails of each

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Victor Y. Yegorov
* Tom Lane [EMAIL PROTECTED] [01.03.2005 21:07]: Hmm, you seem to be envisioning that these are actually lists, not arrays --- that is, to find the N'th page in a list requires traversing list links starting at the first page. That doesn't sound workable. If you can't access all the entries

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
OK, lets step back a bit and see if there is a solution that fits what we think we need and PostgreSQL. Lets talk about FTSS, its something I can discuss easily. It is a two stage system with an indexer and a server. Only the data to be indexed is in the database, all the FTSS data structures are

[HACKERS] bitmap AM design

2005-02-28 Thread Victor Y. Yegorov
Here's the design of bitmap AM I'm planning to implement. I've discussed it with Neil, but I'm willing to get more feedback on it. There are going to be 1 metapage, 1 list of CTIDs (LOC), one list of attribute values (LOV, including attributes for multi-column indexes) and a bitmap for each

Re: [HACKERS] bitmap AM design

2005-02-28 Thread Tom Lane
Victor Y. Yegorov [EMAIL PROTECTED] writes: Neil suggested a very good way how to handle updates. Of course, it's not necessary to strictly follow tuple layout in the table's heap, as I wanted to do initially. All that's needed, is that bit positions in bitmaps would be tied with CTID