[GENERAL] aggregate functions on massive number of rows

[Todd Kover]

I have an aggregate function setup (attached) that I'm calling over a
massive amount of data and am running into:

ERROR:  cannot have more than 2^32-1 commands in a transaction
CONTEXT:  PL/pgSQL function float8_jitter_add line 16 at if

error.  Since I'm able to do count() and avg() over the same values
without this aggregate function, it's theoretically possible.

Something was making me think that it was the extract() that was doing
it (I used to have 'select extract(milliseconds from v_rtt_in) into
v_rtt' and something leaded me to believe the error was with that), but
I don't recall what that is, so it looks like it's just the if like it
says.

Since I'm not really doing anything transaction-oriented in
float8_jitter_add, is there a way to get around this?  Or is this a
limitation in pl/pgsql?

thanks in advance,
-Todd

---snip---
create or replace function float8_jitter_add(float8[], interval)
returns float8[] as '
declare
v_old_state ALIAS FOR $1;
v_rtt_in ALIAS FOR $2;
v_state float8[];
v_rtt float8;
BEGIN
v_state := v_old_state;
v_rtt := extract(milliseconds from v_rtt_in);

IF v_old_state is NULL THEN
v_state := ''{0,0,0,0}'';
v_state[1] = 0;
v_state[2] = 0;
v_state[3] = v_rtt;
v_state[4] = 1;
ELSIF v_rtt IS NOT NULL THEN
if v_old_state[4] = 1 THEN
v_state[1] := v_old_state[2] + (v_old_state[3] - v_rtt);
v_state[2] := v_old_state[2] + 1;
END IF;
v_state[3] := v_rtt;
v_state[4] := 1;
ELSE
v_state[4] := 0;
END IF;
return v_state;
END;
' language 'plpgsql';

create or replace function float8_jitter_sum(float8[])
returns float8 as '
declare
v_state ALIAS FOR $1;
v_avg float8;
BEGIN
v_avg := NULL;
if v_state[1] != 0 THEN
v_avg := v_state[1] / v_state[2];
END IF;
return v_avg;
END;
' language 'plpgsql';

create aggregate jitter (
basetype = interval,
sfunc = float8_jitter_add,
stype = float8[],
finalfunc = float8_jitter_sum
);


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [GENERAL] aggregate functions on massive number of rows

[Michael Fuhr]

On Wed, Mar 02, 2005 at 12:23:45PM -0500, Todd Kover wrote:
 
 I have an aggregate function setup (attached) that I'm calling over a
 massive amount of data and am running into:
 
   ERROR:  cannot have more than 2^32-1 commands in a transaction
   CONTEXT:  PL/pgSQL function float8_jitter_add line 16 at if
 
 error.  Since I'm able to do count() and avg() over the same values
 without this aggregate function, it's theoretically possible.
 
 Something was making me think that it was the extract() that was doing
 it (I used to have 'select extract(milliseconds from v_rtt_in) into
 v_rtt' and something leaded me to believe the error was with that), but
 I don't recall what that is, so it looks like it's just the if like it
 says.

Note the following from the PL/pgSQL Expressions documentation:

  All expressions used in PL/pgSQL  statements are processed using
  the server's regular SQL executor. In effect, a query like

SELECT expression

  is executed using the SPI manager.

I'd guess that you are indeed hitting the command limit.  You might
have more luck with one of the other procedural languages (PL/Perl,
PL/Tcl, PL/Python, etc.), but I'd consider coding something like
this in C if I were using it with so much data.

-- 
Michael Fuhr
http://www.fuhr.org/~mfuhr/

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match