Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-15 Thread Tom Lane
Robert Haas writes: > On Tue, Jan 13, 2015 at 8:16 PM, Tom Lane wrote: >> Not sure whether to just commit this to HEAD and call it a day, or to >> risk back-patching. > I think we need to back-patch something; that's a pretty nasty > regression, and I have some EDB-internal reports that might be

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 8:16 PM, Tom Lane wrote: > I wrote: >> Heikki Linnakangas writes: >>> But do we really need to backpatch any of this? > >> Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB >> peak in 9.3 and up. That seems like a pretty nasty regression. > > I did a

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
I wrote: > Heikki Linnakangas writes: >> But do we really need to backpatch any of this? > Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB > peak in 9.3 and up. That seems like a pretty nasty regression. I did a bit more measurement of the time and backend memory consump

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Heikki Linnakangas writes: > On 01/13/2015 07:24 PM, Tom Lane wrote: >> In hindsight, that's a bad API and we should change it to something like >> ExplainState *es = NewExplainState(); >> so that the sizeof the struct isn't embedded in extension code. But we >> definitely can't do that in back b

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Heikki Linnakangas
On 01/13/2015 07:24 PM, Tom Lane wrote: It is, but FDWs are not at risk here: they merely reference ExplainStates that were allocated by core backend code. So as long as we add the new field at the end it's not a problem for them. Problematic usage would be like what auto_explain does:

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> A difficulty with either your patch or my idea is that they require adding >> another field to ExplainState, which is an ABI break for any third-party >> code that might be declaring variables of that struct type. That's fine >> for HEAD but would be ri

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alvaro Herrera
Tom Lane wrote: > A difficulty with either your patch or my idea is that they require adding > another field to ExplainState, which is an ABI break for any third-party > code that might be declaring variables of that struct type. That's fine > for HEAD but would be risky to back-patch. Any thoug

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
I wrote: > Heikki Linnakangas writes: >> Hmm, something like the attached? Seems reasonable... > This looks pretty unsafe to me: it assumes, without much justification, > that there is no memory allocated during show_expression() that will be > needed later. > I suspect the O(N^2) consumption co

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Tom Lane
Heikki Linnakangas writes: > On 01/13/2015 02:08 PM, Alexey Bashtanov wrote: >> By varying the parameters and reading source code I determined that >> memory usage linearly depends on (plan nodes count)*(overall columns >> count), thus it quadratically depends on number of tables unionized. >> >>

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alexey Bashtanov
On 13.01.2015 16:47, Heikki Linnakangas wrote: Hmm, something like the attached? Seems reasonable... - Heikki yes, i have tested something like this, it stopped eating memory Just one small notice to the patch you attached: maybe it would be more safe to switch to oldcxt before calling Expl

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Heikki Linnakangas
On 01/13/2015 02:08 PM, Alexey Bashtanov wrote: I found that EXPLAIN command takes very much memory to execute when huge unions are used. For example the following sql -- begin sql create table t (a000 int, a001 int, ... a099 int); explain select * from ( select a001 a from t union al

[HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-13 Thread Alexey Bashtanov
Hello! I found that EXPLAIN command takes very much memory to execute when huge unions are used. For example the following sql -- begin sql create table t (a000 int, a001 int, ... a099 int); explain select * from ( select a001 a from t union all select a001 a from t union all