Re: [HACKERS] Producer/Consumer Issues in the COPY across network

2008-02-28 Thread Martijn van Oosterhout
On Thu, Feb 28, 2008 at 01:57:49AM +, Simon Riggs wrote: AIUI the server merely streams the rows to you, the client doesn't get to say how many :) Right, but presumably we generate a new message per PQgetCopyData() request? So my presumption is we need to wait for that to be

Re: [HACKERS] Producer/Consumer Issues in the COPY across network

2008-02-28 Thread Simon Riggs
On Thu, 2008-02-28 at 15:39 +0100, Martijn van Oosterhout wrote: That's why I was wondering about the rows per packet. Sending bigger packets reduces overall overhead. (The malloc/free per row doesn't seem too efficient.) I guess neither of us know then. Oh well. That's good 'cos it sounds

Re: [HACKERS] Producer/Consumer Issues in the COPY across network

2008-02-27 Thread Simon Riggs
On Tue, 2008-02-26 at 12:29 +0100, Martijn van Oosterhout wrote: When we're running a COPY over a high latency link then network time is going to become dominant, so potentially, running COPY asynchronously might help performance for loads or initial Slony configuration. This is

[HACKERS] Producer/Consumer Issues in the COPY across network

2008-02-26 Thread Simon Riggs
I'm looking at ways to reduce the number of network calls and/or the waiting time while we perform network COPY. The COPY calls in libpq allow asynchronous actions, yet are coded in a synchronous manner in pg_dump, Slony and psql \copy. Does anybody have any experience with running COPY in

Re: [HACKERS] Producer/Consumer Issues in the COPY across network

2008-02-26 Thread Martijn van Oosterhout
On Tue, Feb 26, 2008 at 11:00:33AM +, Simon Riggs wrote: I'm looking at ways to reduce the number of network calls and/or the waiting time while we perform network COPY. The COPY calls in libpq allow asynchronous actions, yet are coded in a synchronous manner in pg_dump, Slony and psql