Re: [PERFORM] trouble with a join on OS X

2007-02-05 Thread Shane Ambler
Kirk Wythers wrote: The 4G (32bit) limit may be where you hit the out of memory errors (or is postgres get around that with it's caching?). Any idea if postgres on OS X can truely access more that 4 gigs if the 64 bit version is built? I have tried building the 64 bit version of some other a

Re: [PERFORM] trouble with a join on OS X

2007-02-04 Thread Kirk Wythers
On Feb 3, 2007, at 9:59 AM, Shane Ambler wrote: just so you can look into it for your own curiosity ;-) - Mac OS X uses the startup disk for VM storage. You can find the files in - / var/vm You will find the swapfiles there, the size of the swapfiles progressively get larger - swapfile0

Re: [PERFORM] trouble with a join on OS X

2007-02-03 Thread Shane Ambler
Tom Lane wrote: Since this is just a bog-standard Mini with 512M memory, it was pretty thoroughly on its knees by this point :-(. I'm not sure how to find out about allocated swap space in OS X, but my bet is that the above message should be understood as "totally out of virtual memory". just

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers <[EMAIL PROTECTED]> writes: > The new table needs to be filled with the results of the join. If > there is a way to do this without a SELECT, please share. If it's an entirely new table, then you probably want to use INSERT ... SELECT. If what you want is to update existing rows us

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Kirk Wythers wrote: > > On Feb 2, 2007, at 8:32 PM, Tom Lane wrote: > >> Kirk Wythers <[EMAIL PROTECTED]> writes: >>> On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Now he's got to worry about how to page through 8GB of results in something less than geological time with the space bar

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 8:32 PM, Tom Lane wrote: Kirk Wythers <[EMAIL PROTECTED]> writes: On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Now he's got to worry about how to page through 8GB of results in something less than geological time with the space bar ;-) I actually have no intention of

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers <[EMAIL PROTECTED]> writes: > On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: >> Now he's got to worry about how to page through 8GB of results in >> something less than geological time with the space bar ;-) > I actually have no intention of paging through the results, but > rat

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Geoffrey wrote: > Joshua D. Drake wrote: >> Luke Lonergan wrote: \o /tmp/really_big_cursor_return ;) >>> Tough crowd :-D >> >> Yeah well Andrew probably would have said use sed and pipe it through >> awk to get the data you want. > > Chances are, if you're using awk, you shouldn't n

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Geoffrey
Joshua D. Drake wrote: Luke Lonergan wrote: \o /tmp/really_big_cursor_return ;) Tough crowd :-D Yeah well Andrew probably would have said use sed and pipe it through awk to get the data you want. Chances are, if you're using awk, you shouldn't need sed. :) -- Until later, Geoffrey Those

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Tom, On 2/2/07 2:18 PM, "Tom Lane" <[EMAIL PROTECTED]> wrote: as of 8.2 there's a psql variable FETCH_COUNT that can be set to make it happen behind the scenes.) FETCH_COUNT is a godsend and works beautifully for exactly this purpose. No

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Luke Lonergan wrote: >> \o /tmp/really_big_cursor_return >> >> ;) > > Tough crowd :-D Yeah well Andrew probably would have said use sed and pipe it through awk to get the data you want. Joshua D. Drake > > - Luke > > > > ---(end of broadcast)-

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 7:53 PM, Luke Lonergan wrote: Tough crowd :-D No kidding ;-)

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Luke Lonergan
> \o /tmp/really_big_cursor_return > > ;) Tough crowd :-D - Luke ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Luke Lonergan wrote: > Tom, > > On 2/2/07 2:18 PM, "Tom Lane" <[EMAIL PROTECTED]> wrote: > >> as of 8.2 there's a psql variable >> FETCH_COUNT that can be set to make it happen behind the scenes.) > > FETCH_COUNT is a godsend and works beautifully for exactly this purpose. > > Now he's got to w

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Luke Lonergan
Tom, On 2/2/07 2:18 PM, "Tom Lane" <[EMAIL PROTECTED]> wrote: > as of 8.2 there's a psql variable > FETCH_COUNT that can be set to make it happen behind the scenes.) FETCH_COUNT is a godsend and works beautifully for exactly this purpose. Now he's got to worry about how to page through 8GB of r

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers <[EMAIL PROTECTED]> writes: > However, setting ulimit to unlimited does not seem to solve the > issue. After some experimentation I'm left wondering exactly what ulimit's -d option is for on OS X, because it sure doesn't seem to be limiting process data size. (I should have been su

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 10:11 AM, Tom Lane wrote: "Steinar H. Gunderson" <[EMAIL PROTECTED]> writes: On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: Thanks Tom... Any suggestions as to how much to raise ulimit -d? And how to raise ulimit -d? Try multiplying it by 100 for a start:

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
At this point there are no silly questions. But I am running the query under the shell session that I adjusted. I did discover that ulimit -d only changes the shell session that you issue the command in. So I changed ulimit -d to unlimited, connected to the db with psql db_name, then ran th

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers <[EMAIL PROTECTED]> writes: > However, setting ulimit to unlimited does not seem to solve the > issue. Output from ulimit -a is: Possibly a silly question, but you are running the client code under the shell session that you adjusted ulimit for, yes? regards

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
Tom, I tried ulimit -d 614400, but the query ended with the same error. I thought then that the message: psql(21522) malloc: *** vm_allocate(size=8421376) failed (error code=3) psql(21522) malloc: *** error: can't allocate region psql(21522) malloc: *** set a breakpoint in szone_error to debug

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
"Steinar H. Gunderson" <[EMAIL PROTECTED]> writes: > On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: >> Thanks Tom... Any suggestions as to how much to raise ulimit -d? And >> how to raise ulimit -d? > Try multiplying it by 100 for a start: > ulimit -d 614400 Or just "ulimit -d

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Steinar H. Gunderson
On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: > Thanks Tom... Any suggestions as to how much to raise ulimit -d? And > how to raise ulimit -d? Try multiplying it by 100 for a start: ulimit -d 614400 /* Steinar */ -- Homepage: http://www.sesse.net/ --

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 9:46 AM, Tom Lane wrote: =?ISO-8859-1?Q?G=E1briel_=C1kos?= <[EMAIL PROTECTED]> writes: Richard Huxton wrote: Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. If you look at the message carefull

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
Thanks for the reply Steiner, On Feb 2, 2007, at 8:41 AM, Steinar H. Gunderson wrote: On Fri, Feb 02, 2007 at 07:52:48AM -0600, Kirk Wythers wrote: psql(15811) malloc: *** vm_allocate(size=8421376) failed (error code=3) psql(15811) malloc: *** error: can't allocate region psql(15811) malloc:

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
=?ISO-8859-1?Q?G=E1briel_=C1kos?= <[EMAIL PROTECTED]> writes: > Richard Huxton wrote: >> Kirk Wythers wrote: >>> I am trying to do fairly simple joins on climate databases that should >>> return ~ 7 million rows of data. > If you look at the message carefully, it looks like (for me) that the > c

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Gábriel Ákos
Richard Huxton wrote: Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. However, I'm getting an error message on a OS X (10.4 tiger server) machine that seems to imply that I am running out of memory. The errors are: p

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Richard Huxton
Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. However, I'm getting an error message on a OS X (10.4 tiger server) machine that seems to imply that I am running out of memory. The errors are: psql(15811) malloc: ***

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Steinar H. Gunderson
On Fri, Feb 02, 2007 at 07:52:48AM -0600, Kirk Wythers wrote: > psql(15811) malloc: *** vm_allocate(size=8421376) failed (error code=3) > psql(15811) malloc: *** error: can't allocate region > psql(15811) malloc: *** set a breakpoint in szone_error to debug It sounds like you are out of memory. Ha

[PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. However, I'm getting an error message on a OS X (10.4 tiger server) machine that seems to imply that I am running out of memory. The errors are: psql(15811) malloc: *** vm_allocate(size=8