Improvements are welcome, but to compete in the industry, loading will need to speed up by a factor of 100.
Note that Bizgres loader already does many of these ideas and it sounds like pgloader does too. - Luke Msg is shrt cuz m on ma treo -----Original Message----- From: Dimitri Fontaine [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 06, 2008 12:41 PM Eastern Standard Time To: pgsql-performance@postgresql.org Cc: Greg Smith Subject: Re: [PERFORM] Benchmark Data requested --- pgloader CE design ideas Le mercredi 06 février 2008, Greg Smith a écrit : > If I'm loading a TB file, odds are good I can split that into 4 or more > vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders > at once, and get way more than 1 disk worth of throughput reading. pgloader already supports starting at any input file line number, and limit itself to any number of reads: -C COUNT, --count=COUNT number of input lines to process -F FROMCOUNT, --from=FROMCOUNT number of input lines to skip So you could already launch 4 pgloader processes with the same configuration fine but different command lines arguments. It there's interest/demand, it's easy enough for me to add those parameters as file configuration knobs too. Still you have to pay for client to server communication instead of having the backend read the file locally, but now maybe we begin to compete? Regards, -- dim
