Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Martijn van Oosterhout
On Tue, Nov 13, 2012 at 05:50:08PM -0500, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > Yeah. The thing that concerns me is that I think we have a pretty > > decent number of memory contexts where the expected number of > > allocations is very small ... and we have the co

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > Yeah. The thing that concerns me is that I think we have a pretty > decent number of memory contexts where the expected number of > allocations is very small ... and we have the context *just in case* > we do more than that in certain instances. I've

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Robert Haas
On Tue, Nov 13, 2012 at 3:21 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: >>> I wonder though if we ought to think about running output functions in >>> a short-lived memory context instead of the executor's main context. >>> We've considered tha

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Tom Lane
Robert Haas writes: > On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: >> I wonder though if we ought to think about running output functions in >> a short-lived memory context instead of the executor's main context. >> We've considered that before, I think, and it's always been the path >> of l

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Robert Haas
On Tue, Nov 13, 2012 at 12:18 AM, Tom Lane wrote: > I wonder though if we ought to think about running output functions in > a short-lived memory context instead of the executor's main context. > We've considered that before, I think, and it's always been the path > of least resistance to fix the

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> I think explicit calls like that actually wouldn't be a problem, >> since they'd be run in a per-tuple context anyway. The cases that >> are problematic are hard-coded I/O function calls. I'm worried >> about the ones like, say, plpgsql's built-in

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Dimitri Fontaine
Tom Lane writes: > I think explicit calls like that actually wouldn't be a problem, > since they'd be run in a per-tuple context anyway. The cases that > are problematic are hard-coded I/O function calls. I'm worried > about the ones like, say, plpgsql's built-in conversion operations. > We coul

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> Thoughts? > I've been using textin(record_out(NEW)) in generic partitioning > triggers, and you can find examples of that trick in the wiki, so I > think we have users of that in the field. I think explicit calls like that actually wouldn't be a pr

Re: [HACKERS] Memory leaks in record_out and record_send

2012-11-13 Thread Dimitri Fontaine
Tom Lane writes: > OTOH I can't see trying to back-patch a solution like that. If we want > to fix this in the back branches (and note the complaint linked above is > against 8.3), I think we have to do it as attached. > > Thoughts? I've been using textin(record_out(NEW)) in generic partitionin

[HACKERS] Memory leaks in record_out and record_send

2012-11-12 Thread Tom Lane
I looked into the problem complained of here: http://archives.postgresql.org/pgsql-general/2012-11/msg00279.php which turns out to have nothing to do with joins and everything to do with the fact that record_out() leaks memory like mad. It leaks both the strings returned by the per-column output f

Re: [HACKERS] Memory leaks on SRF rescan

2008-02-21 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > On Thu, 2008-02-21 at 21:42 -0500, Tom Lane wrote: >> Yeah. I think it's hopeless to expect these functions to all hew to >> the straight and narrow path. It seems to me that the right way is for >> the sub-select to somehow run in its own "per-query" con

Re: [HACKERS] Memory leaks on SRF rescan

2008-02-21 Thread Neil Conway
On Thu, 2008-02-21 at 21:42 -0500, Tom Lane wrote: > Given your point (2), is this worth fixing by itself? Right, probably not. > Yeah. I think it's hopeless to expect these functions to all hew to > the straight and narrow path. It seems to me that the right way is for > the sub-select to some

Re: [HACKERS] Memory leaks on SRF rescan

2008-02-21 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > Two different classes of allocations in the EState's per-query context > are leaked: > (1) In FunctionNext(), we call ExecMakeTableFunctionResult() to compute > the result set of the SRF, which allocates the TupleDesc out-parameter > in the per-query memor

[HACKERS] Memory leaks on SRF rescan

2008-02-21 Thread Neil Conway
Consider this test case: create table baz (a int, b int); insert into baz (select 1, generate_series(1, 300)); select baz.*, (select 1 from generate_series(1, 1) g(a) where g.a < baz.b) from baz; The final query consumes ~800 MB of memory on my syste

Re: [HACKERS] Memory leaks

2002-10-29 Thread mlw
Greg Copeland wrote: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? I

Re: [HACKERS] Memory leaks

2002-10-23 Thread Tom Lane
Greg Copeland <[EMAIL PROTECTED]> writes: > Ya, I'm currently looking to see how the memory is being used and why. > I'm trying to better understand it's life cycle. You implying that even > the short term memory should be using the palloc stuff? What about long > term? Blanket statement that p

Re: [HACKERS] Memory leaks

2002-10-23 Thread Greg Copeland
On Wed, 2002-10-23 at 08:48, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > Okay. I've started looking at plpython to better understand it's memory > > needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is > > basically malloc which also checks and reports on memo

Re: [HACKERS] Memory leaks

2002-10-23 Thread Tom Lane
Greg Copeland <[EMAIL PROTECTED]> writes: > Okay. I've started looking at plpython to better understand it's memory > needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is > basically malloc which also checks and reports on memory allocation > errors. Anyone know if the cases whe

Re: [HACKERS] Memory leaks

2002-10-23 Thread Tom Lane
"Nigel J. Andrews" <[EMAIL PROTECTED]> writes: > On Tue, 22 Oct 2002, Tom Lane wrote: >> Not everywhere. plpgsql is full of malloc's and I think the other PL >> modules are too --- and that's not to mention the allocation policies of >> the perl, tcl, etc, language interpreters... > I was going t

Re: [HACKERS] Memory leaks

2002-10-23 Thread Greg Copeland
On Tue, 2002-10-22 at 22:28, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > >So again, I'm not really sure it they are meaningful at > > this point. > > psql might well have some internal leaks; the backend memory-context > design doesn't apply to it. Okay. Thanks. I'll probably

Re: [HACKERS] Memory leaks

2002-10-23 Thread Nigel J. Andrews
On Tue, 22 Oct 2002, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > > Interesting. Having not looked at memory management schemes used in the > > pl implementations, can you enlighten me by what you mean by "integrate > > the memory-context notion"? Does that mean they are not

Re: [HACKERS] Memory leaks

2002-10-22 Thread Karel Zak
On Tue, Oct 22, 2002 at 11:28:23PM -0400, Tom Lane wrote: > > I then moved on to psql, again, just for fun. Here, I'm thinking that I > > started to find some other leaks...but again, I've not spent any real > > time on it. So again, I'm not really sure it they are meaningful at > > this point. >

Re: [HACKERS] Memory leaks

2002-10-22 Thread Tom Lane
Greg Copeland <[EMAIL PROTECTED]> writes: > On Tue, 2002-10-22 at 17:09, Tom Lane wrote: >> Yes, this has been dealt with before. > What tools, aside from noggin v1.0, did they use? Do we know? s/they/me/ ... none. But I don't know of any that I think would be useful. > I then moved on to psql

Re: [HACKERS] Memory leaks

2002-10-22 Thread Tom Lane
"Nigel J. Andrews" <[EMAIL PROTECTED]> writes: > I saw use of a couple of malloc (or Python specific malloc) calls the > other day but I also seem to recall that, after consideration, I > decided the memory needed to survive for the duration of the > backend. Should I have created a new child of th

Re: [HACKERS] Memory leaks

2002-10-22 Thread Nigel J. Andrews
On 22 Oct 2002, Greg Copeland wrote: > On Tue, 2002-10-22 at 17:09, Tom Lane wrote: > > > plpgsql has some issues too, I suspect, but not as bad as pltcl etc. > > Possibly the best answer is to integrate the memory-context notion into > > those modules; if they did most of their work in a temp co

Re: [HACKERS] Memory leaks

2002-10-22 Thread Greg Copeland
On Tue, 2002-10-22 at 17:09, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > I've started playing a little with Postgres to determine if there were > > memory leaks running around. After some very brief checking, I'm > > starting[1] to think that the answer is yes. Has anyone alre

Re: [HACKERS] Memory leaks

2002-10-22 Thread Neil Conway
Petru Paler <[EMAIL PROTECTED]> writes: > valgrind is a great tool I used -- didn't get the time to try it out on > Postgres yet, though. Besides leaks, it also catches uninitialized > variable access and stuff like that. I've used Valgrind with PostgreSQL a little bit, and it's been fairly useful

Re: [HACKERS] Memory leaks

2002-10-22 Thread Hannu Krosing
On Wed, 2002-10-23 at 03:09, Tom Lane wrote: > It's fairly difficult to get anywhere with standard leak-tracking tools, > since they don't know anything about palloc. What's worse, it is *not* > a bug for a routine to palloc space it never pfrees, if it knows that > it's palloc'ing in a short-live

Re: [HACKERS] Memory leaks

2002-10-22 Thread Tom Lane
Greg Copeland <[EMAIL PROTECTED]> writes: > I've started playing a little with Postgres to determine if there were > memory leaks running around. After some very brief checking, I'm > starting[1] to think that the answer is yes. Has anyone already gone > through a significant effort to locate and

Re: [HACKERS] Memory leaks

2002-10-22 Thread Petru Paler
On Tue, Oct 22, 2002 at 04:16:16PM -0500, Greg Copeland wrote: > I've started playing a little with Postgres to determine if there were > memory leaks running around. After some very brief checking, I'm > starting[1] to think that the answer is yes. Has anyone already gone > through a significant

[HACKERS] Memory leaks

2002-10-22 Thread Greg Copeland
I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Is this done periodicall