This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 7fefccd3 Publish built docs triggered by
3481904fe9c770ffc218cc043d47d35e860148a6
7fefccd3 is described below
commit 7fefccd33183b5523d2b4c9e5c861066cf0b7ed1
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Feb 18 16:45:49 2026 +0000
Publish built docs triggered by 3481904fe9c770ffc218cc043d47d35e860148a6
---
_sources/autoapi/datafusion/index.rst.txt | 16 ++++++++++
.../autoapi/datafusion/user_defined/index.rst.txt | 16 ++++++++++
.../common-operations/udf-and-udfa.rst.txt | 33 +++++++++++++++----
autoapi/datafusion/index.html | 14 ++++++++
autoapi/datafusion/user_defined/index.html | 14 ++++++++
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 30 +++++++++---------
user-guide/common-operations/basic-info.html | 34 ++++++++++----------
user-guide/common-operations/functions.html | 20 ++++++------
user-guide/common-operations/udf-and-udfa.html | 37 ++++++++++++++++++----
10 files changed, 159 insertions(+), 57 deletions(-)
diff --git a/_sources/autoapi/datafusion/index.rst.txt
b/_sources/autoapi/datafusion/index.rst.txt
index d967e42e..24fecff4 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -107,6 +107,14 @@ Package Contents
Return the resultant value.
+ While this function template expects a PyArrow Scalar value return type,
+ you can return any value that can be converted into a Scalar. This
+ includes basic Python data types such as integers and strings. In
+ addition to primitive types, we currently support PyArrow, nanoarrow,
+ and arro3 objects in addition to primitive data types. Other objects
+ that support the Arrow FFI standard will be given a "best attempt" at
+ conversion to scalar objects.
+
.. py:method:: merge(states: list[pyarrow.Array]) -> None
@@ -123,6 +131,14 @@ Package Contents
Return the current state.
+ While this function template expects PyArrow Scalar values return type,
+ you can return any value that can be converted into a Scalar. This
+ includes basic Python data types such as integers and strings. In
+ addition to primitive types, we currently support PyArrow, nanoarrow,
+ and arro3 objects in addition to primitive data types. Other objects
+ that support the Arrow FFI standard will be given a "best attempt" at
+ conversion to scalar objects.
+
.. py:method:: update(*values: pyarrow.Array) -> None
diff --git a/_sources/autoapi/datafusion/user_defined/index.rst.txt
b/_sources/autoapi/datafusion/user_defined/index.rst.txt
index 83f52e66..21063829 100644
--- a/_sources/autoapi/datafusion/user_defined/index.rst.txt
+++ b/_sources/autoapi/datafusion/user_defined/index.rst.txt
@@ -62,6 +62,14 @@ Module Contents
Return the resultant value.
+ While this function template expects a PyArrow Scalar value return type,
+ you can return any value that can be converted into a Scalar. This
+ includes basic Python data types such as integers and strings. In
+ addition to primitive types, we currently support PyArrow, nanoarrow,
+ and arro3 objects in addition to primitive data types. Other objects
+ that support the Arrow FFI standard will be given a "best attempt" at
+ conversion to scalar objects.
+
.. py:method:: merge(states: list[pyarrow.Array]) -> None
@@ -78,6 +86,14 @@ Module Contents
Return the current state.
+ While this function template expects PyArrow Scalar values return type,
+ you can return any value that can be converted into a Scalar. This
+ includes basic Python data types such as integers and strings. In
+ addition to primitive types, we currently support PyArrow, nanoarrow,
+ and arro3 objects in addition to primitive data types. Other objects
+ that support the Arrow FFI standard will be given a "best attempt" at
+ conversion to scalar objects.
+
.. py:method:: update(*values: pyarrow.Array) -> None
diff --git a/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
b/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
index d554e1e2..f669721a 100644
--- a/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
+++ b/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
@@ -123,7 +123,7 @@ also see how the inputs to ``update`` and ``merge`` differ.
.. code-block:: python
- import pyarrow
+ import pyarrow as pa
import pyarrow.compute
import datafusion
from datafusion import col, udaf, Accumulator
@@ -136,16 +136,16 @@ also see how the inputs to ``update`` and ``merge``
differ.
def __init__(self):
self._sum = 0.0
- def update(self, values_a: pyarrow.Array, values_b: pyarrow.Array) ->
None:
+ def update(self, values_a: pa.Array, values_b: pa.Array) -> None:
self._sum = self._sum + pyarrow.compute.sum(values_a).as_py() -
pyarrow.compute.sum(values_b).as_py()
- def merge(self, states: List[pyarrow.Array]) -> None:
+ def merge(self, states: list[pa.Array]) -> None:
self._sum = self._sum + pyarrow.compute.sum(states[0]).as_py()
- def state(self) -> pyarrow.Array:
- return pyarrow.array([self._sum])
+ def state(self) -> list[pa.Scalar]:
+ return [pyarrow.scalar(self._sum)]
- def evaluate(self) -> pyarrow.Scalar:
+ def evaluate(self) -> pa.Scalar:
return pyarrow.scalar(self._sum)
ctx = datafusion.SessionContext()
@@ -156,10 +156,29 @@ also see how the inputs to ``update`` and ``merge``
differ.
}
)
- my_udaf = udaf(MyAccumulator, [pyarrow.float64(), pyarrow.float64()],
pyarrow.float64(), [pyarrow.float64()], 'stable')
+ my_udaf = udaf(MyAccumulator, [pa.float64(), pa.float64()], pa.float64(),
[pa.float64()], 'stable')
df.aggregate([], [my_udaf(col("a"), col("b")).alias("col_diff")])
+FAQ
+^^^
+
+**How do I return a list from a UDAF?**
+
+Both the ``evaluate`` and the ``state`` functions expect to return scalar
values.
+If you wish to return a list array as a scalar value, the best practice is to
+wrap the values in a ``pyarrow.Scalar`` object. For example, you can return a
+timestamp list with ``pa.scalar([...], type=pa.list_(pa.timestamp("ms")))`` and
+register the appropriate return or state types as
+``return_type=pa.list_(pa.timestamp("ms"))`` and
+``state_type=[pa.list_(pa.timestamp("ms"))]``, respectively.
+
+As of DataFusion 52.0.0 , you can pass return any Python object, including a
+PyArrow array, as the return value(s) for these functions and DataFusion will
+attempt to create a scalar type from the value. DataFusion has been tested to
+convert PyArrow, nanoarrow, and arro3 objects as well as primitive data types
+like integers, strings, and so on.
+
Window Functions
----------------
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index bdc9b610..49b2d73f 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -3572,6 +3572,13 @@ See <a class="reference external"
href="https://datafusion.apache.org/python";>ht
<dt class="sig sig-object py" id="datafusion.Accumulator.evaluate">
<em class="property"><span class="pre">abstract</span><span class="w">
</span></em><span class="sig-name descname"><span
class="pre">evaluate</span></span><span class="sig-paren">(</span><span
class="sig-paren">)</span> <span class="sig-return"><span
class="sig-return-icon">→</span> <span class="sig-return-typehint"><span
class="pre">pyarrow.Scalar</span></span></span><a class="headerlink"
href="#datafusion.Accumulator.evaluate" title="Link to this
definition">¶</a></dt>
<dd><p>Return the resultant value.</p>
+<p>While this function template expects a PyArrow Scalar value return type,
+you can return any value that can be converted into a Scalar. This
+includes basic Python data types such as integers and strings. In
+addition to primitive types, we currently support PyArrow, nanoarrow,
+and arro3 objects in addition to primitive data types. Other objects
+that support the Arrow FFI standard will be given a “best attempt” at
+conversion to scalar objects.</p>
</dd></dl>
<dl class="py method">
@@ -3584,6 +3591,13 @@ See <a class="reference external"
href="https://datafusion.apache.org/python";>ht
<dt class="sig sig-object py" id="datafusion.Accumulator.state">
<em class="property"><span class="pre">abstract</span><span class="w">
</span></em><span class="sig-name descname"><span
class="pre">state</span></span><span class="sig-paren">(</span><span
class="sig-paren">)</span> <span class="sig-return"><span
class="sig-return-icon">→</span> <span class="sig-return-typehint"><span
class="pre">list</span><span class="p"><span class="pre">[</span></span><span
class="pre">pyarrow.Scalar</span><span class="p"><span
class="pre">]</span></span></sp [...]
<dd><p>Return the current state.</p>
+<p>While this function template expects PyArrow Scalar values return type,
+you can return any value that can be converted into a Scalar. This
+includes basic Python data types such as integers and strings. In
+addition to primitive types, we currently support PyArrow, nanoarrow,
+and arro3 objects in addition to primitive data types. Other objects
+that support the Arrow FFI standard will be given a “best attempt” at
+conversion to scalar objects.</p>
</dd></dl>
<dl class="py method">
diff --git a/autoapi/datafusion/user_defined/index.html
b/autoapi/datafusion/user_defined/index.html
index 349cc862..4dbf00e9 100644
--- a/autoapi/datafusion/user_defined/index.html
+++ b/autoapi/datafusion/user_defined/index.html
@@ -1134,6 +1134,13 @@
<dt class="sig sig-object py"
id="datafusion.user_defined.Accumulator.evaluate">
<em class="property"><span class="pre">abstract</span><span class="w">
</span></em><span class="sig-name descname"><span
class="pre">evaluate</span></span><span class="sig-paren">(</span><span
class="sig-paren">)</span> <span class="sig-return"><span
class="sig-return-icon">→</span> <span class="sig-return-typehint"><span
class="pre">pyarrow.Scalar</span></span></span><a class="headerlink"
href="#datafusion.user_defined.Accumulator.evaluate" title="Link to this
definition">¶</a></dt>
<dd><p>Return the resultant value.</p>
+<p>While this function template expects a PyArrow Scalar value return type,
+you can return any value that can be converted into a Scalar. This
+includes basic Python data types such as integers and strings. In
+addition to primitive types, we currently support PyArrow, nanoarrow,
+and arro3 objects in addition to primitive data types. Other objects
+that support the Arrow FFI standard will be given a “best attempt” at
+conversion to scalar objects.</p>
</dd></dl>
<dl class="py method">
@@ -1146,6 +1153,13 @@
<dt class="sig sig-object py" id="datafusion.user_defined.Accumulator.state">
<em class="property"><span class="pre">abstract</span><span class="w">
</span></em><span class="sig-name descname"><span
class="pre">state</span></span><span class="sig-paren">(</span><span
class="sig-paren">)</span> <span class="sig-return"><span
class="sig-return-icon">→</span> <span class="sig-return-typehint"><span
class="pre">list</span><span class="p"><span class="pre">[</span></span><span
class="pre">pyarrow.Scalar</span><span class="p"><span
class="pre">]</span></span></sp [...]
<dd><p>Return the current state.</p>
+<p>While this function template expects PyArrow Scalar values return type,
+you can return any value that can be converted into a Scalar. This
+includes basic Python data types such as integers and strings. In
+addition to primitive types, we currently support PyArrow, nanoarrow,
+and arro3 objects in addition to primitive data types. Other objects
+that support the Arrow FFI standard will be given a “best attempt” at
+conversion to scalar objects.</p>
</dd></dl>
<dl class="py method">
diff --git a/searchindex.js b/searchindex.js
index e940047a..c255df34 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[19, null]], "Additional
Resources": [[37, "additional-resources"]], "Aggregate Functions": [[24,
"aggregate-functions"], [31, "aggregate-functions"], [33,
"aggregate-functions"]], "Aggregation": [[24, null]], "Alternative Approach":
[[20, "alternative-approach"]], "Apache Iceberg": [[35, "apache-iceberg"]],
"Arrays": [[26, "arrays"]], "Arrow": [[39, null]], "Attributes": [[4,
"attributes"], [7, "attributes"], [12, "attributes"], [18, "att [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[19, null]], "Additional
Resources": [[37, "additional-resources"]], "Aggregate Functions": [[24,
"aggregate-functions"], [31, "aggregate-functions"], [33,
"aggregate-functions"]], "Aggregation": [[24, null]], "Alternative Approach":
[[20, "alternative-approach"]], "Apache Iceberg": [[35, "apache-iceberg"]],
"Arrays": [[26, "arrays"]], "Arrow": [[39, null]], "Attributes": [[4,
"attributes"], [7, "attributes"], [12, "attributes"], [18, "att [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html
b/user-guide/common-operations/aggregations.html
index d3cc751b..e75ffa32 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -621,15 +621,15 @@ Pokemon set. Since there will be many entries of <code
class="docutils literal n
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Steel</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span class="p">[]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Poison</span><span
class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span
class="p">,</span> <span class="n">Fighting</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Rock</span><span
class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Psychic</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Ice</span><span
class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span
class="p">,</span> <span class="p">]</span>
<span class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
</pre></div>
@@ -644,14 +644,14 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">]</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Steel</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Fighting</span><span class="p">,</span>
<span class="n">Psychic</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Dark</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Rock</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span
class="n">Ground</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
@@ -663,15 +663,15 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
diff --git a/user-guide/common-operations/basic-info.html
b/user-guide/common-operations/basic-info.html
index 102c1a0b..abb24229 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -456,11 +456,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">642</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">979</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">425</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">249</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">533</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">850</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">312</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">621</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">792</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">812</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -471,8 +471,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">642</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">979</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">850</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">312</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -490,11 +490,11 @@ passing them to an Arrow table, and then converting them
to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div
class="highlight"><pre><span></span><span class="n">In</span> <span
class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span
class="n">df</span><span class="o">.</span><span
class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span
class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">642</span> <span
class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">979</span> <span
class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">425</span> <span
class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">249</span> <span
class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">533</span> <span
class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">850</span> <span
class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">312</span> <span
class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">621</span> <span
class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">792</span> <span
class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">812</span> <span
class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal"
href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe"
title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func
docutils literal notranslate"><span class="pre">describe()</span></code></a>
shows a quick statistic summary of your data:</p>
@@ -506,11 +506,11 @@ passing them to an Arrow table, and then converting them
to a pandas DataFrame.<
<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
<span class="o">|</span> <span class="n">count</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span
class="mf">5.0</span> <span class="o">|</span> <span
class="mi">5</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">null_count</span> <span
class="o">|</span> <span class="mf">0.0</span> <span
class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span
class="mf">0.0</span> <span class="o">|</span> <span
class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">565.6</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">272.8567389675395</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">249.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">979.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">533.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">677.4</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">222.3393802276151</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">312.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">850.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">792.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
</pre></div>
</div>
diff --git a/user-guide/common-operations/functions.html
b/user-guide/common-operations/functions.html
index 7240ad13..38315122 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -566,16 +566,16 @@ In here we will cover some of the more popular use cases.
If you want to view al
<span class="o">+-------------------------------+</span>
<span class="o">|</span> <span class="n">now</span><span class="p">()</span>
<span class="o">|</span>
<span class="o">+-------------------------------+</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">14</span><span class="n">T00</span><span class="p">:</span><span
class="mi">19</span><span class="p">:</span><span
class="mf">23.922353607</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">18</span><span class="n">T16</span><span class="p">:</span><span
class="mi">45</span><span class="p">:</span><span
class="mf">37.900528994</span> <span class="o">|</span>
<span class="o">+-------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
</pre></div>
diff --git a/user-guide/common-operations/udf-and-udfa.html
b/user-guide/common-operations/udf-and-udfa.html
index 07eb1060..ad7e7010 100644
--- a/user-guide/common-operations/udf-and-udfa.html
+++ b/user-guide/common-operations/udf-and-udfa.html
@@ -426,6 +426,13 @@
<a class="reference internal nav-link" href="#aggregate-functions">
Aggregate Functions
</a>
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#faq">
+ FAQ
+ </a>
+ </li>
+ </ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#window-functions">
@@ -582,7 +589,7 @@ different states. Finally <code class="docutils literal
notranslate"><span class
<p>In the following example we want to define a custom aggregate function that
will return the
difference between the sum of two columns. The state can be represented by a
single value and we can
also see how the inputs to <code class="docutils literal notranslate"><span
class="pre">update</span></code> and <code class="docutils literal
notranslate"><span class="pre">merge</span></code> differ.</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">import</span><span
class="w"> </span><span class="nn">pyarrow</span>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">import</span><span
class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span
class="k">as</span><span class="w"> </span><span class="nn">pa</span>
<span class="kn">import</span><span class="w"> </span><span
class="nn">pyarrow.compute</span>
<span class="kn">import</span><span class="w"> </span><span
class="nn">datafusion</span>
<span class="kn">from</span><span class="w"> </span><span
class="nn">datafusion</span><span class="w"> </span><span
class="kn">import</span> <span class="n">col</span><span class="p">,</span>
<span class="n">udaf</span><span class="p">,</span> <span
class="n">Accumulator</span>
@@ -595,16 +602,16 @@ also see how the inputs to <code class="docutils literal
notranslate"><span clas
<span class="k">def</span><span class="w"> </span><span
class="fm">__init__</span><span class="p">(</span><span
class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span
class="n">_sum</span> <span class="o">=</span> <span class="mf">0.0</span>
- <span class="k">def</span><span class="w"> </span><span
class="nf">update</span><span class="p">(</span><span
class="bp">self</span><span class="p">,</span> <span
class="n">values_a</span><span class="p">:</span> <span
class="n">pyarrow</span><span class="o">.</span><span
class="n">Array</span><span class="p">,</span> <span
class="n">values_b</span><span class="p">:</span> <span
class="n">pyarrow</span><span class="o">.</span><span
class="n">Array</span><span class="p">)</span> <span [...]
+ <span class="k">def</span><span class="w"> </span><span
class="nf">update</span><span class="p">(</span><span
class="bp">self</span><span class="p">,</span> <span
class="n">values_a</span><span class="p">:</span> <span
class="n">pa</span><span class="o">.</span><span class="n">Array</span><span
class="p">,</span> <span class="n">values_b</span><span class="p">:</span>
<span class="n">pa</span><span class="o">.</span><span
class="n">Array</span><span class="p">)</span> <span class="o" [...]
<span class="bp">self</span><span class="o">.</span><span
class="n">_sum</span> <span class="o">=</span> <span
class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span
class="o">+</span> <span class="n">pyarrow</span><span class="o">.</span><span
class="n">compute</span><span class="o">.</span><span class="n">sum</span><span
class="p">(</span><span class="n">values_a</span><span class="p">)</span><span
class="o">.</span><span class="n">as_py</span><span cla [...]
- <span class="k">def</span><span class="w"> </span><span
class="nf">merge</span><span class="p">(</span><span
class="bp">self</span><span class="p">,</span> <span
class="n">states</span><span class="p">:</span> <span
class="n">List</span><span class="p">[</span><span
class="n">pyarrow</span><span class="o">.</span><span
class="n">Array</span><span class="p">])</span> <span class="o">-></span>
<span class="kc">None</span><span class="p">:</span>
+ <span class="k">def</span><span class="w"> </span><span
class="nf">merge</span><span class="p">(</span><span
class="bp">self</span><span class="p">,</span> <span
class="n">states</span><span class="p">:</span> <span
class="nb">list</span><span class="p">[</span><span class="n">pa</span><span
class="o">.</span><span class="n">Array</span><span class="p">])</span> <span
class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span
class="n">_sum</span> <span class="o">=</span> <span
class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span
class="o">+</span> <span class="n">pyarrow</span><span class="o">.</span><span
class="n">compute</span><span class="o">.</span><span class="n">sum</span><span
class="p">(</span><span class="n">states</span><span class="p">[</span><span
class="mi">0</span><span class="p">])</span><span class=" [...]
- <span class="k">def</span><span class="w"> </span><span
class="nf">state</span><span class="p">(</span><span
class="bp">self</span><span class="p">)</span> <span class="o">-></span>
<span class="n">pyarrow</span><span class="o">.</span><span
class="n">Array</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">pyarrow</span><span
class="o">.</span><span class="n">array</span><span class="p">([</span><span
class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span
class="p">])</span>
+ <span class="k">def</span><span class="w"> </span><span
class="nf">state</span><span class="p">(</span><span
class="bp">self</span><span class="p">)</span> <span class="o">-></span>
<span class="nb">list</span><span class="p">[</span><span
class="n">pa</span><span class="o">.</span><span class="n">Scalar</span><span
class="p">]:</span>
+ <span class="k">return</span> <span class="p">[</span><span
class="n">pyarrow</span><span class="o">.</span><span
class="n">scalar</span><span class="p">(</span><span
class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span
class="p">)]</span>
- <span class="k">def</span><span class="w"> </span><span
class="nf">evaluate</span><span class="p">(</span><span
class="bp">self</span><span class="p">)</span> <span class="o">-></span>
<span class="n">pyarrow</span><span class="o">.</span><span
class="n">Scalar</span><span class="p">:</span>
+ <span class="k">def</span><span class="w"> </span><span
class="nf">evaluate</span><span class="p">(</span><span
class="bp">self</span><span class="p">)</span> <span class="o">-></span>
<span class="n">pa</span><span class="o">.</span><span
class="n">Scalar</span><span class="p">:</span>
<span class="k">return</span> <span class="n">pyarrow</span><span
class="o">.</span><span class="n">scalar</span><span class="p">(</span><span
class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span
class="p">)</span>
<span class="n">ctx</span> <span class="o">=</span> <span
class="n">datafusion</span><span class="o">.</span><span
class="n">SessionContext</span><span class="p">()</span>
@@ -615,11 +622,27 @@ also see how the inputs to <code class="docutils literal
notranslate"><span clas
<span class="p">}</span>
<span class="p">)</span>
-<span class="n">my_udaf</span> <span class="o">=</span> <span
class="n">udaf</span><span class="p">(</span><span
class="n">MyAccumulator</span><span class="p">,</span> <span
class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span
class="n">float64</span><span class="p">(),</span> <span
class="n">pyarrow</span><span class="o">.</span><span
class="n">float64</span><span class="p">()],</span> <span
class="n">pyarrow</span><span class="o">.</span><span class="n">float64 [...]
+<span class="n">my_udaf</span> <span class="o">=</span> <span
class="n">udaf</span><span class="p">(</span><span
class="n">MyAccumulator</span><span class="p">,</span> <span
class="p">[</span><span class="n">pa</span><span class="o">.</span><span
class="n">float64</span><span class="p">(),</span> <span
class="n">pa</span><span class="o">.</span><span class="n">float64</span><span
class="p">()],</span> <span class="n">pa</span><span class="o">.</span><span
class="n">float64</span><span cl [...]
<span class="n">df</span><span class="o">.</span><span
class="n">aggregate</span><span class="p">([],</span> <span
class="p">[</span><span class="n">my_udaf</span><span class="p">(</span><span
class="n">col</span><span class="p">(</span><span
class="s2">"a"</span><span class="p">),</span> <span
class="n">col</span><span class="p">(</span><span
class="s2">"b"</span><span class="p">))</span><span
class="o">.</span><span class="n">alias</span><span class="p">(</span><spa [...]
</pre></div>
</div>
+<section id="faq">
+<h3>FAQ<a class="headerlink" href="#faq" title="Link to this
heading">¶</a></h3>
+<p><strong>How do I return a list from a UDAF?</strong></p>
+<p>Both the <code class="docutils literal notranslate"><span
class="pre">evaluate</span></code> and the <code class="docutils literal
notranslate"><span class="pre">state</span></code> functions expect to return
scalar values.
+If you wish to return a list array as a scalar value, the best practice is to
+wrap the values in a <code class="docutils literal notranslate"><span
class="pre">pyarrow.Scalar</span></code> object. For example, you can return a
+timestamp list with <code class="docutils literal notranslate"><span
class="pre">pa.scalar([...],</span> <span
class="pre">type=pa.list_(pa.timestamp("ms")))</span></code> and
+register the appropriate return or state types as
+<code class="docutils literal notranslate"><span
class="pre">return_type=pa.list_(pa.timestamp("ms"))</span></code> and
+<code class="docutils literal notranslate"><span
class="pre">state_type=[pa.list_(pa.timestamp("ms"))]</span></code>,
respectively.</p>
+<p>As of DataFusion 52.0.0 , you can pass return any Python object, including a
+PyArrow array, as the return value(s) for these functions and DataFusion will
+attempt to create a scalar type from the value. DataFusion has been tested to
+convert PyArrow, nanoarrow, and arro3 objects as well as primitive data types
+like integers, strings, and so on.</p>
+</section>
</section>
<section id="window-functions">
<h2>Window Functions<a class="headerlink" href="#window-functions" title="Link
to this heading">¶</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
