Re: [HACKERS] Dead Space Map for vacuum

2007-01-16 Thread ITAGAKI Takahiro
I can see that there are two issues in the design of Dead Space Map in the recent discussions: 1. information accuracy of dead spaces 2. memory management I'll write up the discussion about the 1st for now. We need to increase page-tracking status for effective vacuum. 1 bit per block is

Re: [HACKERS] Dead Space Map for vacuum

2007-01-05 Thread Jim Nasby
On Jan 3, 2007, at 11:42 PM, ITAGAKI Takahiro wrote: BTW, if we want to achieve the index-only scan, we might have to do more aggressive VACUUM FREEZE. There were many comments that we should avoid vacuuming pages that contain only unfrozen tuples or a few dead tuples. I think it it true for

Re: [HACKERS] Dead Space Map for vacuum

2007-01-03 Thread ITAGAKI Takahiro
"Jochem van Dieten" <[EMAIL PROTECTED]> wrote: > On 12/28/06, ITAGAKI Takahiro wrote: > > > > | [TODO item] Allow data to be pulled directly from indexes > > | Another idea is to maintain a bitmap of heap pages where all rows are > > | visible to all backends, and allow index lookups to reference

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Simon Riggs
On Sat, 2006-12-30 at 09:22 +1100, Russell Smith wrote: > Simon Riggs wrote: > > FSM code ignores any block with less space than 1 average tuple, which > > is a pretty reasonable rule. > > > FSM serves a different purpose than DSM and therefore has an entirely > different set of rules governing

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Russell Smith
Simon Riggs wrote: On Fri, 2006-12-29 at 10:49 -0500, Tom Lane wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: I would suggest that we tracked whether a block has had 0, 1 or 1+ updates/deletes against it. When a block has 1+ it can then be worthwhile to VACUUM it and to place it onto

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Simon Riggs
On Fri, 2006-12-29 at 16:41 -0500, Tom Lane wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > On Fri, 2006-12-29 at 10:49 -0500, Tom Lane wrote: > >> Counterexample: table in which all tuples exceed half a page. > > > Current FSM code will ignore those too, if they are less than the > > avera

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > On Fri, 2006-12-29 at 10:49 -0500, Tom Lane wrote: >> Counterexample: table in which all tuples exceed half a page. > Current FSM code will ignore those too, if they are less than the > average size of the tuple so far requested. Thats a pretty wierd > c

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Simon Riggs
On Fri, 2006-12-29 at 10:49 -0500, Tom Lane wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > I would suggest that we tracked whether a block has had 0, 1 or 1+ > > updates/deletes against it. When a block has 1+ it can then be > > worthwhile to VACUUM it and to place it onto the FSM. Two dead

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > I would suggest that we tracked whether a block has had 0, 1 or 1+ > updates/deletes against it. When a block has 1+ it can then be > worthwhile to VACUUM it and to place it onto the FSM. Two dead tuples is > really the minimum space worth reclaiming on a

Re: [HACKERS] Dead Space Map for vacuum

2006-12-29 Thread Simon Riggs
On Thu, 2006-12-28 at 21:35 +, Heikki Linnakangas wrote: > I only used 1 bit, just like in Itagaki's approach. 1 bit may not be enough. In many cases, a block will receive only 1 UPDATE or DELETE. If we then mark this in the DSM, when we VACUUM that block it will not have sufficient space f

Re: [HACKERS] Dead Space Map for vacuum

2006-12-28 Thread Heikki Linnakangas
Gavin Sherry wrote: On Thu, 28 Dec 2006, Heikki Linnakangas wrote: ITAGAKI Takahiro wrote: I experimented with a different DSM design last winter. I got busy with other things and never posted it AFAIR, but the idea was to store a bitmap in the special area on every 32k heap page. That had some

Re: [HACKERS] Dead Space Map for vacuum

2006-12-28 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > I experimented with a different DSM design last winter. I got busy with > other things and never posted it AFAIR, but the idea was to store a > bitmap in the special area on every 32k heap page. That had some advantages: > * doesn't require a new

Re: [HACKERS] Dead Space Map for vacuum

2006-12-28 Thread ITAGAKI Takahiro
Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > We use 1 bit per block, so we cannot separate pages that need either > > vacuum or freeze only. The reason is that we cannot determine where to > > record before/after updated tuples. If the transaction is commited, > > before-version should be re

Re: [HACKERS] Dead Space Map for vacuum

2006-12-28 Thread Jochem van Dieten
On 12/28/06, ITAGAKI Takahiro wrote: | [TODO item] Allow data to be pulled directly from indexes | Another idea is to maintain a bitmap of heap pages where all rows are | visible to all backends, and allow index lookups to reference that bitmap | to avoid heap lookups It is not done yet, but we

Re: [HACKERS] Dead Space Map for vacuum

2006-12-28 Thread Gavin Sherry
On Thu, 28 Dec 2006, Heikki Linnakangas wrote: > ITAGAKI Takahiro wrote: > > Hello, > > > > NTT staffs are working on TODO item: > > | Create a bitmap of pages that need vacuuming > > > > We call the bitmap "Dead Space Map" (DSM), that allows VACUUM to scan > > only pages that need vacuuming or fr

Re: [HACKERS] Dead Space Map for vacuum

2006-12-27 Thread Heikki Linnakangas
ITAGAKI Takahiro wrote: Hello, NTT staffs are working on TODO item: | Create a bitmap of pages that need vacuuming We call the bitmap "Dead Space Map" (DSM), that allows VACUUM to scan only pages that need vacuuming or freezing. We'd like to discuss the design on hackers and make agreements wit

[HACKERS] Dead Space Map for vacuum

2006-12-27 Thread ITAGAKI Takahiro
Hello, NTT staffs are working on TODO item: | Create a bitmap of pages that need vacuuming We call the bitmap "Dead Space Map" (DSM), that allows VACUUM to scan only pages that need vacuuming or freezing. We'd like to discuss the design on hackers and make agreements with community. We implement