Re: [HACKERS] visibility maps and heap_prune

2010-02-27 Thread Bruce Momjian
Heikki Linnakangas wrote: > Bruce Momjian wrote: > > Pavan Deolasee wrote: > >> On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian wrote: > >> > >>> Whatever happened to this? It was in the first 9.0 commitfest but was > >>> returned with feedback but never updated: > >>> > >>> > >> Though Alex did s

Re: [HACKERS] visibility maps and heap_prune

2010-02-27 Thread Heikki Linnakangas
Bruce Momjian wrote: > Pavan Deolasee wrote: >> On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian wrote: >> >>> Whatever happened to this? It was in the first 9.0 commitfest but was >>> returned with feedback but never updated: >>> >>> >> Though Alex did some useful tests and review, and in fact con

Re: [HACKERS] visibility maps and heap_prune

2010-02-26 Thread Bruce Momjian
Pavan Deolasee wrote: > On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian wrote: > > > > > Whatever happened to this? It was in the first 9.0 commitfest but was > > returned with feedback but never updated: > > > > > Though Alex did some useful tests and review, and in fact confirmed that the > VAC

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Pavan Deolasee
On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian wrote: > > Whatever happened to this? It was in the first 9.0 commitfest but was > returned with feedback but never updated: > > Though Alex did some useful tests and review, and in fact confirmed that the VACUUM time dropped from 16494 msec to 366

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 10:32 PM, Bruce Momjian wrote: > Robert Haas wrote: >> On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian wrote: >> > Whatever happened to this? ?It was in the first 9.0 commitfest but was >> > returned with feedback but never updated: >> > >> > ? ? ? ?https://commitfest.postg

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Bruce Momjian
Robert Haas wrote: > On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian wrote: > > Whatever happened to this? ?It was in the first 9.0 commitfest but was > > returned with feedback but never updated: > > > > ? ? ? ?https://commitfest.postgresql.org/action/patch_view?id=75 > > Well, the patch author c

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian wrote: > Whatever happened to this?  It was in the first 9.0 commitfest but was > returned with feedback but never updated: > >        https://commitfest.postgresql.org/action/patch_view?id=75 Well, the patch author chose not to pursue it. It's clea

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Bruce Momjian
Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: https://commitfest.postgresql.org/action/patch_view?id=75 --- Pavan Deolasee wrote: > ISTM that the PD

Re: [HACKERS] visibility maps and heap_prune

2009-07-25 Thread Robert Haas
On Tue, Jul 21, 2009 at 2:37 AM, Pavan Deolasee wrote: > On Tue, Jul 21, 2009 at 10:38 AM, Robert Haas wrote: >> Pavan, are you planning to respond to Alex's comments and/or update this >> patch? > > Yes, I will. Hopefully  by end of this week. Since it has now been 10 days since this patch was r

Re: [HACKERS] visibility maps and heap_prune

2009-07-20 Thread Pavan Deolasee
On Tue, Jul 21, 2009 at 10:38 AM, Robert Haas wrote: > > Pavan, are you planning to respond to Alex's comments and/or update this > patch? > Yes, I will. Hopefully by end of this week. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers ma

Re: [HACKERS] visibility maps and heap_prune

2009-07-20 Thread Robert Haas
On Wed, Jul 15, 2009 at 11:44 PM, Alex Hunsaker wrote: > On Mon, Dec 8, 2008 at 06:56, Pavan Deolasee wrote: >> Here is a patch which implements this. The PD_ALL_VISIBLE flag is set if all >> tuples in the page are visible to all transactions and there are no DEAD >> line pointers in the page. The

Re: [HACKERS] visibility maps and heap_prune

2009-07-15 Thread Alex Hunsaker
On Mon, Dec 8, 2008 at 06:56, Pavan Deolasee wrote: > Here is a patch which implements this. The PD_ALL_VISIBLE flag is set if all > tuples in the page are visible to all transactions and there are no DEAD > line pointers in the page. The second check is required so that VACUUM takes > up the page.

Re: [HACKERS] visibility maps

2009-01-20 Thread Heikki Linnakangas
Bruce Momjian wrote: Tom Lane wrote: "Pavan Deolasee" writes: OTOH I think we can still set PD_ALL_VISIBLE page header flag even when there are DEAD line pointers. That would mean the header flag means something different than the map bit does, which would mean extra tests need to be made bef

Re: [HACKERS] visibility maps and heap_prune

2009-01-20 Thread Heikki Linnakangas
Bruce Momjian wrote: Pavan Deolasee wrote: On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee wrote: On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas < heikki.linnakan...@enterprisedb.com> wrote: If you see a straightforward way, please submit a patch! Will do that. Here is a patch whi

Re: [HACKERS] visibility maps

2009-01-20 Thread Bruce Momjian
Tom Lane wrote: > "Pavan Deolasee" writes: > > OTOH I think we can still set PD_ALL_VISIBLE page header flag even > > when there are DEAD line pointers. > > That would mean the header flag means something different than the > map bit does, which would mean extra tests need to be made before > pro

Re: [HACKERS] visibility maps and heap_prune

2009-01-20 Thread Bruce Momjian
Pavan Deolasee wrote: > On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee > wrote: > > > > > > > On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas < > > heikki.linnakan...@enterprisedb.com> wrote: > > > >> > >> If you see a straightforward way, please submit a patch! > >> > >> > > Will do that. >

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Robert Haas
> Yeah, I dropped the ball on that one. It's been knocking in the back of my > head since, but I've never gotten around. I'm feeling reluctant to review it > since it's not really a high priority thing, and I'm not sure whether we > want it or not. In that case perhaps we should add it to http://w

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Jan 15, 2009 at 7:10 AM, Bruce Momjian wrote: Is this something for 8.4 CVS? I worked out the patch as per Heikki's suggestion. So I think he needs to review and decide it's fate. Yeah, I dropped the ball on that one. It's been knocking in the back of my head

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Pavan Deolasee
On Thu, Jan 15, 2009 at 7:10 AM, Bruce Momjian wrote: > > Is this something for 8.4 CVS? > I worked out the patch as per Heikki's suggestion. So I think he needs to review and decide it's fate. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hac

Re: [HACKERS] visibility maps

2009-01-14 Thread Bruce Momjian
Is there anything to do with the below issue? --- Pavan Deolasee wrote: > On Wed, Dec 17, 2008 at 7:11 PM, Tom Lane wrote: > > > > > > I think what you are suggesting is that we should set the visibility map > > bit while d

Re: [HACKERS] visibility maps and heap_prune

2009-01-14 Thread Bruce Momjian
Is this something for 8.4 CVS? --- Pavan Deolasee wrote: > On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee > wrote: > > > > > > > On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas < > > heikki.linnakan...@enterprisedb.com

Re: [HACKERS] visibility maps

2008-12-17 Thread Tom Lane
"Pavan Deolasee" writes: > OTOH I think we can still set PD_ALL_VISIBLE page header flag even > when there are DEAD line pointers. That would mean the header flag means something different than the map bit does, which would mean extra tests need to be made before propagating the flag bit to the m

Re: [HACKERS] visibility maps

2008-12-17 Thread Pavan Deolasee
On Wed, Dec 17, 2008 at 7:11 PM, Tom Lane wrote: > > > I think what you are suggesting is that we should set the visibility map > bit while dead line pointers (tombstones) still remain. If that's what > you meant it's a bad idea. No, I'm not suggesting that. I understand the problem there. I was

Re: [HACKERS] visibility maps

2008-12-17 Thread Tom Lane
"Pavan Deolasee" writes: > On Wed, Dec 17, 2008 at 3:29 PM, Heikki Linnakangas > wrote: >> I don't quite understand this paragraph. If there's any DEAD tuples or >> line-pointers, the all-visible flag can't be set. > No, I am saying, HOT-prune removes all DEAD tuples from the page (not > the DEA

Re: [HACKERS] visibility maps

2008-12-17 Thread Pavan Deolasee
On Wed, Dec 17, 2008 at 3:29 PM, Heikki Linnakangas wrote: > > > I don't quite understand this paragraph. If there's any DEAD tuples or > line-pointers, the all-visible flag can't be set. After an UPDATE or DELETE, > it indeed takes two vacuums until the bits in the visibility map are set. > > Or

Re: [HACKERS] visibility maps

2008-12-17 Thread Heikki Linnakangas
Pavan Deolasee wrote: Another thing I noticed is the since VACUUM tries to set the bit in the first phase, it's working only because HOT prunes DEAD tuples just before we do another scan on line pointers (which I had earlier talked about getting rid of. May be its time I do that). Otherwise, the

Re: [HACKERS] visibility maps

2008-12-12 Thread Dimitri Fontaine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Le 12 déc. 08 à 13:11, Pavan Deolasee a écrit : I did some tests today with pgbench on a decent SMP machine. The goal was to see if multiple clients (20 in the test) tries to update tuples in different data blocks, if the EX lock on the VM page

Re: [HACKERS] visibility maps

2008-12-12 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 8:09 PM, Heikki Linnakangas wrote: > Pavan Deolasee wrote: > > >> I can do some if we don't have already. > > Oh, yes please! > I did some tests today with pgbench on a decent SMP machine. The goal was to see if multiple clients (20 in the test) tries to update tuples in d

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 8:09 PM, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > > I'm not sure if we should set the bits in very aggressively. If we're more > aggressive about setting the bits, it also means that we have to clear the > bits more often, increasing the likelihood of contention tha

Re: [HACKERS] visibility maps

2008-12-11 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Dec 11, 2008 at 7:15 PM, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: Do we have any tests to prove that the VM page lock does not indeed become a bottleneck ? No. I can do some if we don't have already. Oh, yes please! Only the first update to a page needs

Re: [HACKERS] visibility maps

2008-12-11 Thread Tom Lane
"Pavan Deolasee" <[EMAIL PROTECTED]> writes: > On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala <[EMAIL PROTECTED]> wrote: >> IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) >> can access to the same memory address(es)* in same time*. The question is >> how compiler compile

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 7:15 PM, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > > Yeah, if we accept that bits can be bogusly set. There is scenarios where > that can happen already, but they involve crashing, not during normal > operation and clean shut down. In the future, I'd like to move in

Re: [HACKERS] visibility maps

2008-12-11 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala <[EMAIL PROTECTED]> wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to assembler. But this

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 7:03 PM, Zdenek Kotala <[EMAIL PROTECTED]> wrote: > > Yes, because it is not simple write operation. You need to read byte from > memory to register, set bit and write it back. Write memory itself is atomic > but somebody could change other bits between read and write. > Y

Re: [HACKERS] visibility maps

2008-12-11 Thread Zdenek Kotala
Pavan Deolasee napsal(a): On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala <[EMAIL PROTECTED]> wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to assembler. But t

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala <[EMAIL PROTECTED]> wrote: > >> > > IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) > can access to the same memory address(es)* in same time*. The question is > how compiler compile C code to assembler. But this code seems t

Re: [HACKERS] visibility maps

2008-12-11 Thread Zdenek Kotala
Heikki Linnakangas napsal(a): Pavan Deolasee wrote: /* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] & (1 << mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read

Re: [HACKERS] visibility maps and heap_prune

2008-12-08 Thread Pavan Deolasee
On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee <[EMAIL PROTECTED]>wrote: > > > On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas < > [EMAIL PROTECTED]> wrote: > >> >> If you see a straightforward way, please submit a patch! >> >> > Will do that. > > Here is a patch which implements this. The PD

Re: [HACKERS] visibility maps and heap_prune

2008-12-07 Thread Pavan Deolasee
On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas < [EMAIL PROTECTED]> wrote: > > If you see a straightforward way, please submit a patch! > > Will do that. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: /* * Size of the bitmap on each visibility map page, in bytes. There's no * extra headers, so the whole page minus except for the standard page header * is used for the bitmap. */ #define MAPSIZE (BLCKSZ - SizeOfPageHeaderData) ISTM that we should MAXALIGN the SizeOfPa

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Sat, Dec 6, 2008 at 7:57 PM, Heikki Linnakangas < [EMAIL PROTECTED]> wrote: Umm, what non-atomic state could the bit be in? Half-set, half-cleared? Or do you think that if some other bit in proximity is changed, the other bit would temporarily flip 0->1->0, or something

Re: [HACKERS] visibility maps

2008-12-06 Thread Pavan Deolasee
On Sat, Dec 6, 2008 at 7:57 PM, Heikki Linnakangas < [EMAIL PROTECTED]> wrote: > > Umm, what non-atomic state could the bit be in? Half-set, half-cleared? Or > do you think that if some other bit in proximity is changed, the other bit > would temporarily flip 0->1->0, or something like that? I don

Re: [HACKERS] visibility maps and heap_prune

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: ISTM that the PD_ALL_VISIBLE flag and the visibility map bit can be set at the end of pruning operation if we know that there are only tuples visible to all transactions left in the page. Right. The way pruning is done, I think it would be straight forward to get this in

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: /* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] & (1 << mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read atomically on all platforms.

Re: [HACKERS] visibility maps

2008-12-04 Thread Pavan Deolasee
/* * Size of the bitmap on each visibility map page, in bytes. There's no * extra headers, so the whole page minus except for the standard page header * is used for the bitmap. */ #define MAPSIZE (BLCKSZ - SizeOfPageHeaderData) ISTM that we should MAXALIGN the SizeOfPageHeaderData to compute

[HACKERS] visibility maps

2008-12-04 Thread Pavan Deolasee
/* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] & (1 << mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read atomically on all platforms. I think the only caller of

[HACKERS] visibility maps and heap_prune

2008-12-04 Thread Pavan Deolasee
ISTM that the PD_ALL_VISIBLE flag and the visibility map bit can be set at the end of pruning operation if we know that there are only tuples visible to all transactions left in the page. The way pruning is done, I think it would be straight forward to get this information. Thanks, Pavan -- Pava