Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-06 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: OK, here's an updated version... Applied with some small stylistic revisions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-05 Thread Robert Haas
OK, here's an updated version... 1. Use IncrBufferRefCount() so that we can do unconditional ReleaseBuffers elsewhere. I'm not sure this is really any simpler, and although IncrBufferRefCount() is pretty cheap, it's certainly not as cheap as a NULL pointer test. 2. Consolidate a bunch of logic

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-04 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: Patch resnapped to HEAD, with straightforward adjustments to compensate for Heikki's changes to the ReadBuffer interface. See attached. I looked this over a bit. A couple of suggestions: 1. You could probably simplify life a bit by treating the

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-04 Thread Robert Haas
2. The logic changes in RelationGetBufferForTuple seem bizarre and overcomplicated. ISTM that the buffer saved by the bistate ought to be about equivalent to relation-rd_targblock, ie, it's your first trial location and also a place to save the located buffer on the way out. I'd suggest

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-04 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: 2. The logic changes in RelationGetBufferForTuple seem bizarre and overcomplicated. ISTM that the buffer saved by the bistate ought to be about equivalent to relation-rd_targblock, ie, it's your first trial location and also a place to save the located

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-01 Thread Robert Haas
Can you test whether using the buffer access strategy is a win or a loss? Most of that gain is probably coming from the reduction in pinning. Patch resnapped to HEAD, with straightforward adjustments to compensate for Heikki's changes to the ReadBuffer interface. See attached. New testing

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-11-01 Thread Simon Riggs
On Sat, 2008-11-01 at 13:23 -0400, Robert Haas wrote: Can you test whether using the buffer access strategy is a win or a loss? Most of that gain is probably coming from the reduction in pinning. --PATCHED-- Time: 13869.724 ms (median) --PATCHED with BAS disabled-- Time: 14460.432 ms

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-31 Thread Simon Riggs
On Thu, 2008-10-30 at 23:05 -0400, Robert Haas wrote: Whatever timings you have are worth publishing. Here are the timings for copying the first ten million integers into a one-column table created in the same transaction, with and without the patch. As you can see, now that I've

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-31 Thread Simon Riggs
On Thu, 2008-10-30 at 22:46 -0400, Robert Haas wrote: You should try profiling the patch. You can count the invocations of the buffer access routines to check its all working in the right ratios. *goes and learns how to do profile PostgreSQL* OK, that was a good suggestion. It looks

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-30 Thread Robert Haas
You should try profiling the patch. You can count the invocations of the buffer access routines to check its all working in the right ratios. *goes and learns how to do profile PostgreSQL* OK, that was a good suggestion. It looks like part of my problem here is that I didn't put the CREATE

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-30 Thread Robert Haas
Whatever timings you have are worth publishing. Here are the timings for copying the first ten million integers into a one-column table created in the same transaction, with and without the patch. As you can see, now that I've corrected my previous error of not putting CREATE TABLE and COPY in

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-29 Thread Simon Riggs
On Tue, 2008-10-28 at 23:45 -0400, Robert Haas wrote: One concern that I have about this approach is that the situation in which people are probably most concerned about COPY performance is restoring a dump. In that case, the COPY will be the only thing running, and using a

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-29 Thread Robert Haas
If you say its a loss you should publish timings to support that. Using a BAS for VACUUM was a performance gain, not a loss. Well, I can dig up and publish the timings from my laptop, but I'm not sure where that will get us. Trust me, the numbers were higher with BAS, otherwise I wouldn't be

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-29 Thread Simon Riggs
On Wed, 2008-10-29 at 21:58 -0400, Robert Haas wrote: If you say its a loss you should publish timings to support that. Using a BAS for VACUUM was a performance gain, not a loss. Well, I can dig up and publish the timings from my laptop, but I'm not sure where that will get us. Trust me,

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-28 Thread Robert Haas
And here's the patch, which based on comments thus far does the following: - Replaces the use_wal, use_fsm arguments in various places with a single options argument. - Creates a BAS_BULKWRITE buffer access strategy. - Creates a BulkInsertState object so that COPY and CTAS can use BAS_BULKWRITE

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-26 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: I am kind of inclined to define flags like this: #define HEAP_INSERT_SKIP_WAL 0x0001 #define HEAP_INSERT_SKIP_FSM 0x0002 #define HEAP_INSERT_BULK 0x0004 /* do we even need this one? */ And then: Oid heap_insert(Relation relation, HeapTuple tup,

Re: [HACKERS] BufferAccessStrategy for bulk insert

2008-10-26 Thread Robert Haas
Seems sane to me. I don't see the point of the HEAP_INSERT_BULK flag bit --- providing or not providing bistate would cover that, and if you have a bit as well then you have to define what the inconsistent combinations mean. I concur with making all-zeroes be the typical state of the flag