Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-03 Thread Sam Mason
On Fri, Apr 03, 2009 at 11:09:56AM +0200, Ivan Sergio Borgonovo wrote: > On Fri, 3 Apr 2009 02:05:19 +0100 Sam Mason wrote: > > On Thu, Apr 02, 2009 at 09:48:33PM +0200, Ivan Sergio Borgonovo wrote: > > > I didn't find any elegant example of cursor use in PHP... OK PHP > > > is not the most elegan

Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-03 Thread Ivan Sergio Borgonovo
On Fri, 3 Apr 2009 02:05:19 +0100 Sam Mason wrote: > On Thu, Apr 02, 2009 at 09:48:33PM +0200, Ivan Sergio Borgonovo > wrote: > > I didn't find any elegant example of cursor use in PHP... OK PHP > > is not the most elegant language around... but still any good > > exapmle someone could point me a

Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-02 Thread Scott Marlowe
On Thu, Apr 2, 2009 at 7:05 PM, Sam Mason wrote: > On Thu, Apr 02, 2009 at 09:48:33PM +0200, Ivan Sergio Borgonovo wrote: >> I didn't find any elegant example of cursor use in PHP... OK PHP is >> not the most elegant language around... but still any good exapmle >> someone could point me at? > > I

Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-02 Thread Sam Mason
On Thu, Apr 02, 2009 at 09:48:33PM +0200, Ivan Sergio Borgonovo wrote: > I didn't find any elegant example of cursor use in PHP... OK PHP is > not the most elegant language around... but still any good exapmle > someone could point me at? I don't program PHP; but my guess would be something like:

Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-02 Thread Ivan Sergio Borgonovo
On Thu, 2 Apr 2009 17:27:55 +0100 Sam Mason wrote: > On Thu, Apr 02, 2009 at 11:20:02AM +0200, Ivan Sergio Borgonovo > wrote: > > This is the work-flow I've in mind: > > > > 1a) take out *all* data from a table in chunks (M record for each > > file, one big file?) (\copy??, from inside a scriptin

Re: [GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-02 Thread Sam Mason
On Thu, Apr 02, 2009 at 11:20:02AM +0200, Ivan Sergio Borgonovo wrote: > This is the work-flow I've in mind: > > 1a) take out *all* data from a table in chunks (M record for each > file, one big file?) (\copy??, from inside a scripting language?) What about using cursors here? > 2a) process each

[GENERAL] reducing IO and memory usage: sending the content of a table to multiple files

2009-04-02 Thread Ivan Sergio Borgonovo
This is the work-flow I've in mind: 1a) take out *all* data from a table in chunks (M record for each file, one big file?) (\copy??, from inside a scripting language?) 2a) process each file with awk to produce N files very similar each other (substantially turn them into very simple xml) 3a) gzip