Re: [PERFORM] COUNT(*) again (was Re: [HACKERS] Index/Function organized table layout)

2003-10-05 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: I think that's not happening, conditionally or otherwise. The atomicity problems alone are sufficient reason why not, even before you look at the performance issues. What are the atomicity problems of adding a create/expire xid to the

Re: [HACKERS] Index/Function organized table layout

2003-10-05 Thread Hannu Krosing
Greg Stark kirjutas P, 05.10.2003 kell 00:17: I've never seen anyone use this feature, and I never seriously considered it myself. It sort of has the feel of an antiquated feature that traded too much flexibility and abstraction for raw performance on very slow disk hardware. Read A

Re: [HACKERS] Index/Function organized table layout

2003-10-05 Thread Mike Mascari
Hannu Krosing wrote: Greg Stark kirjutas P, 05.10.2003 kell 00:17: I've never seen anyone use this feature, and I never seriously considered it myself. It sort of has the feel of an antiquated feature that traded too much flexibility and abstraction for raw performance on very slow disk

Re: [HACKERS] Index/Function organized table layout (from Re:

2003-10-04 Thread Hannu Krosing
Christopher Browne kirjutas R, 03.10.2003 kell 00:57: [EMAIL PROTECTED] (Jean-Luc Lachance) writes: That's one of the draw back of MVCC. I once suggested that the transaction number and other house keeping info be included in the index, but was told to forget it... It would solve once

Re: [HACKERS] Index/Function organized table layout

2003-10-04 Thread James Rogers
On 10/2/03 11:34 PM, Hannu Krosing [EMAIL PROTECTED] wrote: James Rogers kirjutas N, 02.10.2003 kell 23:44: Not exactly. What you are describing is more akin to partitioning or hash-organized tables i.e. sorting insert/update tuples to various pages according to some hash function. What I

COUNT(*) again (was Re: [HACKERS] Index/Function organized table layout)

2003-10-04 Thread Tom Lane
Hannu Krosing [EMAIL PROTECTED] writes: Christopher Browne kirjutas R, 03.10.2003 kell 00:57: A while back I outlined how this would have to be done, and for it to be done efficiently, it would be anything BUT simple. Could this be made a TODO item, perhaps with your attack plan. If I

Re: [HACKERS] Index/Function organized table layout

2003-10-04 Thread Greg Stark
James Rogers [EMAIL PROTECTED] writes: On 10/2/03 11:34 PM, Hannu Krosing [EMAIL PROTECTED] wrote: What I actually thought I was describing is how CLUSTER should work in a postgres flavour of MVCC storage ;). Not the CLUSTER command, but the whole feature. Yeah, I can see this.

Re: [HACKERS] Index/Function organized table layout

2003-10-04 Thread Tom Lane
Greg Stark [EMAIL PROTECTED] writes: So I'm a bit confused about the term Clustering. It seems Postgres uses it to mean simply ordering the tuple storage within an otherwise normal table. However in other databases it seems to mean something more complex. My take is that clustering means not

Re: [PERFORM] COUNT(*) again (was Re: [HACKERS] Index/Function organized table layout)

2003-10-04 Thread Tom Lane
Hannu Krosing [EMAIL PROTECTED] writes: The point I was trying to make was that faster count(*)'s is just a side effect. If we could (conditionally) keep visibility info in indexes, I think that's not happening, conditionally or otherwise. The atomicity problems alone are sufficient reason why

Re: [HACKERS] Index/Function organized table layout

2003-10-03 Thread Hannu Krosing
James Rogers kirjutas N, 02.10.2003 kell 23:44: On Thu, 2003-10-02 at 12:09, Hannu Krosing wrote: So what you really need is the CLUSTER command to leave pages half-empty and the tuple placement logic on inserts/updates to place new tuples near the place where they would be placed by

Re: [HACKERS] Index/Function organized table layout

2003-10-02 Thread James Rogers
On Wed, 2003-10-01 at 09:29, Alvaro Herrera wrote: On Wed, Oct 01, 2003 at 11:37:38AM -0400, Tom Lane wrote: Hm, are you sure that smarter buffer management wouldn't serve the purpose? It doesn't help when there a lot of access locality in searching. In my case I want to select some

Re: [HACKERS] Index/Function organized table layout

2003-10-02 Thread Hannu Krosing
James Rogers kirjutas N, 02.10.2003 kell 20:50: To give a real world example, a standard query on one of our tables that has not been CLUSTER-ed recently (i.e. within the last several days) generates an average of ~2,000 cache misses. Recently CLUSTER-ed, it generates ~0 cache misses on

Re: [HACKERS] Index/Function organized table layout

2003-10-02 Thread James Rogers
On Thu, 2003-10-02 at 12:09, Hannu Krosing wrote: So what you really need is the CLUSTER command to leave pages half-empty and the tuple placement logic on inserts/updates to place new tuples near the place where they would be placed by CLUSTER. I.e. the code that does actual inserting should

Re: [HACKERS] Index/Function organized table layout

2003-10-02 Thread Alvaro Herrera Munoz
On Thu, Oct 02, 2003 at 10:09:12PM +0300, Hannu Krosing wrote: So what you really need is the CLUSTER command to leave pages half-empty and the tuple placement logic on inserts/updates to place new tuples near the place where they would be placed by CLUSTER. I.e. the code that does actual

[HACKERS] Index/Function organized table layout

2003-10-01 Thread James Rogers
Hi, I am running a large and mature mission-critical PostgreSQL implementation with multiple tables that are growing at a rate of several million rows per month (the rate of addition is growing as well). It is a 24x7 operation, so downtime is a party foul and apparent performance has to be

Re: [HACKERS] Index/Function organized table layout

2003-10-01 Thread Tom Lane
James Rogers [EMAIL PROTECTED] writes: Now, I've actually hacked commercial MVCC engines back in the day, and am comfortable playing around in database internals. I have an itch to scratch for improving the scalability of Really Large Tables by explicitly allowing control of table layouts as

Re: [HACKERS] Index/Function organized table layout

2003-10-01 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: As for other indexes, I'm not sure why you say this precludes the use of other indexes. The only thing they have to do is keep pointers to index elements, instead of heap elements. Doesn't sound impossible to me. However, btree feels free to move

Re: [HACKERS] Index/Function organized table layout

2003-10-01 Thread Alvaro Herrera
On Wed, Oct 01, 2003 at 11:37:38AM -0400, Tom Lane wrote: James Rogers [EMAIL PROTECTED] writes: Both of these things really are attempts to address the same basic problem, which is optimizing the number of buffers a given query uses by making the tables layout reflect typical queries.