Re: [HACKERS] extending relations more efficiently

2012-05-31 Thread Jeff Davis
On Tue, 2012-05-01 at 10:08 -0400, Robert Haas wrote: > We've previously discussed the possible desirability of extending > relations in larger increments, rather than one block at a time, for > performance reasons. I attempted to determine how much performance we > could possibly buy this way, an

Re: [HACKERS] extending relations more efficiently

2012-05-03 Thread Stephen Frost
Robert, * Stephen Frost (sfr...@snowman.net) wrote: > > In all seriousness, this is not a great test case unless you can > > provide some scripts to make it easy to run it in a reproducible > > fashion. Can you? > > Yeah, sure, I'll do that. The PostGIS folks have scripts, but they're > kind of

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié may 02 12:55:17 -0400 2012: > > On Wed, May 2, 2012 at 12:46 PM, Alvaro Herrera > wrote: > > Agreed.  Perhaps to solve this issue what we need is a way to migrate > > tuples from later pages into earlier ones.  (This was one of the points, > > never res

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Robert Haas
On Wed, May 2, 2012 at 12:46 PM, Alvaro Herrera wrote: > Agreed.  Perhaps to solve this issue what we need is a way to migrate > tuples from later pages into earlier ones.  (This was one of the points, > never resolved, that we discussed during the VACUUM FULL rework.) Yeah, I agree. And frankly

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié may 02 12:37:35 -0400 2012: > > On Wed, May 2, 2012 at 12:26 PM, Alvaro Herrera > wrote: > > Excerpts from Robert Haas's message of mié may 02 08:14:36 -0400 2012: > >> On Wed, May 2, 2012 at 7:16 AM, Jeroen Vermeulen wrote: > >> > On 2012-05-01 22:06,

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Tom Lane
Robert Haas writes: > On Wed, May 2, 2012 at 12:26 PM, Alvaro Herrera > wrote: >> Hm. I see those two things as different -- to me, bloat is unremoved >> dead tuples, whereas slack space would be free space that can be reused >> by new tuples. Slack space is useful as it avoids relation extensi

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Robert Haas
On Wed, May 2, 2012 at 12:26 PM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of mié may 02 08:14:36 -0400 2012: >> On Wed, May 2, 2012 at 7:16 AM, Jeroen Vermeulen wrote: >> > On 2012-05-01 22:06, Robert Haas wrote: >> >> It might also be interesting to provide a mechanism to pre-

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié may 02 08:14:36 -0400 2012: > On Wed, May 2, 2012 at 7:16 AM, Jeroen Vermeulen wrote: > > On 2012-05-01 22:06, Robert Haas wrote: > >> It might also be interesting to provide a mechanism to pre-extend a > >> relation to a certain number of blocks, though

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Robert Haas
On Wed, May 2, 2012 at 7:16 AM, Jeroen Vermeulen wrote: > On 2012-05-01 22:06, Robert Haas wrote: >> It might also be interesting to provide a mechanism to pre-extend a >> relation to a certain number of blocks, though if we did that we'd >> have to make sure that autovac got the memo not to trunc

Re: [HACKERS] extending relations more efficiently

2012-05-02 Thread Jeroen Vermeulen
On 2012-05-01 22:06, Robert Haas wrote: It might also be interesting to provide a mechanism to pre-extend a relation to a certain number of blocks, though if we did that we'd have to make sure that autovac got the memo not to truncate those pages away again. Good point. And just to check befo

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, May 1, 2012 at 3:48 PM, Stephen Frost wrote: > > I'd be happy to help construct that case, as well as test any changes in > > this area which might help address it (on a 10G SSD-backed SAN..). > > Hmm, can you ship me one of those 10

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
On Tue, May 1, 2012 at 3:48 PM, Stephen Frost wrote: > Try loading the TIGER 2011 data into a single table, where you load each > county (or perhaps state) in a separate, parallel, process.  That's what > I was doing when I hit this lock full-force and bitched about it on this > list. > > I'd be h

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > I think the first thing we need here is a good test case, so we're > clear on what we're trying to solve. I was just hoping to make file > extension *faster* and what you and Simon are talking about is making > it scale better in the face of heavy par

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
On Tue, May 1, 2012 at 3:01 PM, Alvaro Herrera wrote: >> The system is supposed to be designed to avoid that; we try to hand out >> pages with free space to different backends.  One of the advantages of >> the current page-at-a-time setup is that when there is *no* free space >> according to the F

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mar may 01 13:19:48 -0400 2012: > Christopher Browne writes: > > Try to make sure that one of the scenarios involves there being > > multiple writers. > > > One of the relevant cases is where there are multiple writers, where > > they may wind up fighting over

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Tom Lane
Christopher Browne writes: > Try to make sure that one of the scenarios involves there being > multiple writers. > One of the relevant cases is where there are multiple writers, where > they may wind up fighting over the last page in the table. If free > space is in that one page, they might ser

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Christopher Browne
On Tue, May 1, 2012 at 10:22 AM, Simon Riggs wrote: > On Tue, May 1, 2012 at 3:08 PM, Robert Haas wrote: > >> We've previously discussed the possible desirability of extending >> relations in larger increments, rather than one block at a time, for >> performance reasons.  I attempted to determine

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
On Tue, May 1, 2012 at 11:42 AM, Andres Freund wrote: >> > There is the question whether this should be done in the background >> > though, so the relation extension lock is never hit in anything >> > time-critical... >> Yeah, although I'm fuzzy on how and whether that can be made to work, >> whic

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Andres Freund
On Tuesday, May 01, 2012 05:06:11 PM Robert Haas wrote: > On Tue, May 1, 2012 at 10:31 AM, Andres Freund wrote: > >> efficient than our current method - I'm guessing that it actually > >> writes the updated metadata back to disk, where write() does not (this > >> makes one wonder how safe it is to

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
On Tue, May 1, 2012 at 10:31 AM, Andres Freund wrote: >> efficient than our current method - I'm guessing that it actually >> writes the updated metadata back to disk, where write() does not (this >> makes one wonder how safe it is to count on write to have the behavior >> we need here in the firs

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
On Tue, May 1, 2012 at 10:22 AM, Simon Riggs wrote: > Fair enough, but my understanding was that tests showed that the > extension lock was a bottleneck, so doing extensions in larger chunks > should reduce the time we spend waiting for a lock and thus improve > performance. So while your results

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Andres Freund
On Tuesday, May 01, 2012 04:08:27 PM Robert Haas wrote: > We've previously discussed the possible desirability of extending > relations in larger increments, rather than one block at a time, for > performance reasons. I attempted to determine how much performance we > could possibly buy this way,

Re: [HACKERS] extending relations more efficiently

2012-05-01 Thread Simon Riggs
On Tue, May 1, 2012 at 3:08 PM, Robert Haas wrote: > We've previously discussed the possible desirability of extending > relations in larger increments, rather than one block at a time, for > performance reasons.  I attempted to determine how much performance we > could possibly buy this way, and

[HACKERS] extending relations more efficiently

2012-05-01 Thread Robert Haas
We've previously discussed the possible desirability of extending relations in larger increments, rather than one block at a time, for performance reasons. I attempted to determine how much performance we could possibly buy this way, and found that, as far as I can see, the answer is, basically, n