Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jan 13, 2015 at 8:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Not sure whether to just commit this to HEAD and call it a day, or to risk back-patching. I think we need to back-patch something; that's a pretty nasty regression, and I have some

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 8:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: But do we really need to backpatch any of this? Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB peak in 9.3 and up. That seems like a

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Heikki Linnakangas
On 01/13/2015 07:24 PM, Tom Lane wrote: It is, but FDWs are not at risk here: they merely reference ExplainStates that were allocated by core backend code. So as long as we add the new field at the end it's not a problem for them. Problematic usage would be like what auto_explain does:

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: A difficulty with either your patch or my idea is that they require adding another field to ExplainState, which is an ABI break for any third-party code that might be declaring variables of that struct type. That's fine for HEAD

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alvaro Herrera
Tom Lane wrote: A difficulty with either your patch or my idea is that they require adding another field to ExplainState, which is an ABI break for any third-party code that might be declaring variables of that struct type. That's fine for HEAD but would be risky to back-patch. Any thoughts

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 01/13/2015 07:24 PM, Tom Lane wrote: In hindsight, that's a bad API and we should change it to something like ExplainState *es = NewExplainState(); so that the sizeof the struct isn't embedded in extension code. But we definitely can't do

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
I wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: But do we really need to backpatch any of this? Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB peak in 9.3 and up. That seems like a pretty nasty regression. I did a bit more measurement of the time and

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alexey Bashtanov
On 13.01.2015 16:47, Heikki Linnakangas wrote: Hmm, something like the attached? Seems reasonable... - Heikki yes, i have tested something like this, it stopped eating memory Just one small notice to the patch you attached: maybe it would be more safe to switch to oldcxt before calling

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Heikki Linnakangas
On 01/13/2015 02:08 PM, Alexey Bashtanov wrote: I found that EXPLAIN command takes very much memory to execute when huge unions are used. For example the following sql -- begin sql create table t (a000 int, a001 int, ... a099 int); explain select * from ( select a001 a from t union

[HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alexey Bashtanov
Hello! I found that EXPLAIN command takes very much memory to execute when huge unions are used. For example the following sql -- begin sql create table t (a000 int, a001 int, ... a099 int); explain select * from ( select a001 a from t union all select a001 a from t union all

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 01/13/2015 02:08 PM, Alexey Bashtanov wrote: By varying the parameters and reading source code I determined that memory usage linearly depends on (plan nodes count)*(overall columns count), thus it quadratically depends on number of tables

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
I wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: Hmm, something like the attached? Seems reasonable... This looks pretty unsafe to me: it assumes, without much justification, that there is no memory allocated during show_expression() that will be needed later. I suspect the