Hi Ted,
Last I checked (when we wrote the book chapter on the subject), aggregate state
are limited to scalars and Drill-defined types. There is no support to spill
aggregate state, so that state will be lost if spilling is required to handle
large aggregate batches. The current solution works for simple cases such as
totals and averages.
Aggregate UDFs share no state, so it is not possible for one function to use
state accumulated by another. If, for example, you want sum, average and
standard deviation, you'll have to accumulate the total three times, average
twice, and so on. Note that the std dev function will require buffering all
data in one's own array (without any spilling or other support), to allow
computing the (X-bar - X)^2 part of the calculation.
A UDF can emit a byte array (have to check it this is true of aggregate UDFs).
A VarChar is simply a special kind of array, and UDFs can emit a VarChar.
All this is from memory and so is only approximately accurate. YMMV.
Thanks,
- Paul
On Monday, August 12, 2019, 07:35:47 AM PDT, Ted Dunning
<[email protected]> wrote:
What is the current state of building aggregators that have complex state
via UDFs?
Is it possible to define multi-level aggregators in a UDF?
Can the output of a UDF be a byte array?
(these are three different questions)
