milenkovicm commented on code in PR #215:
URL: https://github.com/apache/datafusion-site/pull/215#discussion_r4636136819
##########
content/blog/2026-10-29-ballista-python-udfs.md:
##########
@@ -0,0 +1,298 @@
+---
+layout: post
+title: Distributed Python UDFs in Apache DataFusion Ballista
+date: 2026-10-29
+author: pmc
+categories: [subprojects]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Python client for [Apache DataFusion Ballista] can now run Python
user-defined functions (UDFs) across a
+cluster. Scalar, aggregate, and window UDFs written in Python are serialized
into the query plan, shipped to
+the scheduler, and executed on Ballista executors, from both the DataFrame API
and SQL.
+
+This comes alongside a larger change in how the Ballista Python client works.
Instead of wrapping
+[DataFusion Python], Ballista now plugs into a standard
`datafusion.SessionContext` through DataFusion's FFI
+layer. If you know the DataFusion Python API, you already know how to use
Ballista.
+
+Neither change would have been possible without a long series of improvements
to DataFusion Python and the
+`datafusion-ffi` crate. This post explains what changed, how it works, and who
made it happen.
+
+[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/
+[DataFusion Python]: https://datafusion.apache.org/python/
+
+## What it looks like
+
+Python UDFs are most useful when logic is awkward to write with built-in SQL
functions but natural to express
+in Python, such as business rules kept in a Python data structure. To show
what this looks like on a cluster,
+we will rewrite a query from the [TPC-H] benchmark, the same benchmark
Ballista uses to measure its own
+performance, so you can try it on data you generate yourself.
+
+[TPC-H Query 19] computes the revenue from parts that qualify for a discount.
The rules differ by brand, and
+the reference SQL spells them out as three nearly identical blocks joined with
`OR`. With a Python UDF, the
+rules can live in an ordinary Python dictionary, and the query is reduced to a
join, the shared filters, and a
+single call to the UDF.
+
+To follow along, generate the two tables the query needs with [tpchgen-cli]:
+
+```shell
+pip install tpchgen-cli
+tpchgen-cli parquet --scale-factor 1 --tables lineitem,part --output-dir data
+```
+
+Then start a local cluster. The `ballista` Python package can run both the
scheduler and the executor. Start
+each in its own terminal, from the same directory as `data/`, so the executor
can find the files:
+
+```shell
+# terminal 1: scheduler, listening on localhost:50050
+python -c "from ballista import BallistaScheduler; s = BallistaScheduler();
s.start(); s.wait_for_termination()"
+
+# terminal 2: executor
+python -c "from ballista import BallistaExecutor; e = BallistaExecutor();
e.start(); e.wait_for_termination()"
+```
+
+Alternatively, `ballista.setup_test_cluster()` starts both inside the current
Python process and returns the
+host and port to connect to. Until the next release, the `ballista` package
needs to be built from source, as
+described in [Trying it out](#trying-it-out).
+
+The UDF is written with [pyarrow.compute], so each batch stays in Arrow rather
than being converted to Python
+objects:
+
+```python
+import functools
+import os
+
+import pyarrow as pa
+import pyarrow.compute as pc
+from datafusion import col, lit, udf
+from datafusion import functions as f
+from ballista import BallistaSessionContext
+
+ctx = BallistaSessionContext("df://localhost:50050")
+ctx.register_parquet("lineitem", "data/lineitem.parquet")
+ctx.register_parquet("part", "data/part.parquet")
+
+# brand -> (allowed containers, minimum quantity, maximum size)
+DISCOUNT_RULES = {
+ "Brand#12": (["SM CASE", "SM BOX", "SM PACK", "SM PKG"], 1, 5),
+ "Brand#23": (["MED BAG", "MED BOX", "MED PKG", "MED PACK"], 10, 10),
+ "Brand#34": (["LG CASE", "LG BOX", "LG PACK", "LG PKG"], 20, 15),
+}
+
+
+def is_discounted(
+ brand: pa.Array, container: pa.Array, quantity: pa.Array, size: pa.Array
+) -> pa.Array:
+ print(f"is_discounted: {len(brand)} rows in process {os.getpid()}",
flush=True)
+ matches = [
+ functools.reduce(
+ pc.and_,
+ [
+ pc.equal(brand, b),
+ pc.is_in(container, pa.array(containers)),
+ pc.greater_equal(quantity, min_qty),
+ pc.less_equal(quantity, min_qty + 10),
+ pc.greater_equal(size, 1),
+ pc.less_equal(size, max_size),
+ ],
+ )
+ for b, (containers, min_qty, max_size) in DISCOUNT_RULES.items()
+ ]
+ return functools.reduce(pc.or_, matches)
+
+
+is_discounted_udf = udf(
+ is_discounted,
+ [pa.string(), pa.string(), pa.decimal128(15, 2), pa.int32()],
+ pa.bool_(),
+ "immutable",
+ name="is_discounted",
+)
+
+# DataFrame API
+lineitem = ctx.table("lineitem").filter(
+ f.in_list(col("l_shipmode"), [lit("AIR"), lit("AIR REG")]),
+ col("l_shipinstruct") == lit("DELIVER IN PERSON"),
+)
+
+revenue = (
+ lineitem.join(ctx.table("part"), left_on="l_partkey", right_on="p_partkey")
+ .filter(
+ is_discounted_udf(
+ col("p_brand"), col("p_container"), col("l_quantity"),
col("p_size")
+ )
+ )
+ .aggregate(
+ [], [f.sum(col("l_extendedprice") * (lit(1) -
col("l_discount"))).alias("revenue")]
+ )
+)
+revenue.show()
+
+# SQL
+ctx.register_udf(is_discounted_udf)
+ctx.sql("""
+ select sum(l_extendedprice * (1 - l_discount)) as revenue
+ from lineitem join part on p_partkey = l_partkey
+ where l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ and is_discounted(p_brand, p_container, l_quantity, p_size)
+""").show()
+```
+
+The join and filters run as native DataFusion operators on the executors, and
only the rows that survive them
+are handed to `is_discounted`. The UDF does not need to be registered on the
cluster ahead of time. Its code,
+along with the `DISCOUNT_RULES` dictionary it references, travels with the
query.
+
+You can see this happen. The `print` call at the top of `is_discounted` writes
one line per batch, and those
+lines appear in the executor's terminal, not in the terminal where you ran the
query:
+
+```text
+is_discounted: 8192 rows in process 61867
+is_discounted: 8192 rows in process 61867
+...
+```
+
+The client only prints the query results. The function was pickled into the
plan, sent to the executor, and
+run there, in a different Python process.
+
+[TPC-H]: https://www.tpc.org/tpch/
+[TPC-H Query 19]:
https://github.com/apache/datafusion-ballista/blob/main/benchmarks/queries/q19.sql
+[pyarrow.compute]: https://arrow.apache.org/docs/python/api/compute.html
+[tpchgen-cli]: https://github.com/datafusion-contrib/tpcgen-rs
+
+## A native DataFusion Python experience
+
+Earlier versions of the Ballista Python client subclassed DataFusion Python's
`SessionContext` and `DataFrame`.
+The main execution methods, such as `collect`, `show`, and `to_pandas`, were
intercepted, the logical plan was
+serialized, and the query was resubmitted through a separate Ballista session.
This worked for the common
+cases, but it had real drawbacks:
+
+- Methods the wrapper did not know about ran locally instead of on the cluster.
+- Results came back as a Ballista-specific `DistributedDataFrame`, and whether
a query ran on the cluster
+ depended on the wrapper re-wrapping every DataFrame it returned. Write
options such as `partition_by` were
+ silently ignored.
+- The wrapper had to track every change to the DataFusion Python API, which
caused churn in the Ballista
+ repository each time a feature was added upstream. For example, it mirrored
DataFusion Python's Parquet
+ writer options field by field.
+- Python objects did not always cross the boundary between the two libraries
cleanly. The issue that started
+ this effort, [#1142], reported a `TypeError: argument 'group_by': 'Expr'
object cannot be converted to 'Expr'`
+ from a simple aggregate.
+
+DataFusion Python 55 can install a custom query planner and extension codecs
through FFI. Ballista now uses
+this to configure a standard `datafusion.SessionContext`. DataFusion Python
plans each query as usual, and
+Ballista's query planner sends the plan to the scheduler. Any operation that
executes a plan, including
+`collect`, `show`, `to_pandas`, `write_parquet`, and `EXPLAIN ANALYZE`, runs
on the cluster. The object you get
+back from `BallistaSessionContext` is a `datafusion.SessionContext`, and the
DataFrames it creates are regular
+`datafusion.DataFrame`s.
+
+Because these are the standard DataFusion Python objects, the full DataFusion
Python API and its
+documentation, including the [user guide], [API reference], and examples,
apply to Ballista as-is. Previously,
+any method the wrapper did not intercept quietly ran locally instead of on the
cluster. One known gap remains:
+`DataFrame.cache()` is not yet supported, because the in-memory table it
creates cannot be serialized into the
+plan sent to the scheduler.
+
+[user guide]: https://datafusion.apache.org/python/user-guide/introduction.html
+[API reference]: https://datafusion.apache.org/python/autoapi/index.html
+
+[#1142]: https://github.com/apache/datafusion-ballista/issues/1142
+
+## How Python UDFs travel with the plan
+
+Ballista exchanges plans between the client, scheduler, and executors as
protobuf, using DataFusion's
+extension codecs for anything the built-in serialization does not cover. A
Python UDF is a reference to a live
+Python object, so it cannot be described by name alone.
+
+DataFusion Python 55 added codecs that solve this. When a plan contains a
Python UDF, the codec pickles the
+function with [cloudpickle] and embeds the payload inline in the plan. On the
receiving side, the codec
+recognizes the payload and reconstructs the UDF. Ballista composes these
codecs with its own:
+
+- The client, scheduler, and executors started from the `ballista` Python
package all use Python-aware
+ logical and physical codecs, with Ballista's default codecs underneath.
+- Plans that do not contain Python UDFs serialize exactly as before, so there
is no overhead for queries that
+ do not use them.
+
+Because the function body is shipped as a pickle, a few requirements follow:
+
+- The scheduler and executors must be started from the `ballista` Python
package (`BallistaScheduler` and
+ `BallistaExecutor`), since decoding a Python UDF requires a Python
interpreter.
+- They must run the same Python minor version as the client.
+- Any module the UDF imports must be installed wherever it runs.
+
+One known limitation remains: Python UDFs used as shuffle partitioning keys
are not yet inlined.
+
+[cloudpickle]: https://github.com/cloudpipe/cloudpickle
+
+## The building blocks
+
+Most of the work behind this feature happened upstream, in DataFusion Python
and the `datafusion-ffi` crate.
+[Tim Saucer] has led the DataFusion FFI effort for a long time, and over the
DataFusion 54 and 55 cycles he
+added the pieces Ballista needed:
+
+- [Logical and physical extension codecs][dfp-1541] in DataFusion Python
+- [Pickling of expressions][dfp-1544], then of [aggregate and window
UDFs][dfp-1545]
+- [Per-session control over UDF inlining][dfp-1546], with a strict mode that
refuses to serialize UDFs
+- [FFI query planner support][dfp-1677]
+- [Composable extension codecs][dfp-1678], so projects like Ballista can layer
their own codecs on top
+- [Hardening the extension API][dfp-1759] before it shipped in 55
+- [A user guide and runnable examples][dfp-1547] for distributing expressions
+
+[Marko Milenković] then brought these together in Ballista. He rebuilt the
Python client on the FFI query
+planner ([#2588]) and wired the Python-aware codecs through the client,
scheduler, and executors to support
+UDFs ([#2619]).
+
+[Tim Saucer]: https://github.com/timsaucer
+[Marko Milenković]: https://github.com/milenkovicm
+[dfp-1541]: https://github.com/apache/datafusion-python/pull/1541
+[dfp-1544]: https://github.com/apache/datafusion-python/pull/1544
+[dfp-1545]: https://github.com/apache/datafusion-python/pull/1545
+[dfp-1546]: https://github.com/apache/datafusion-python/pull/1546
+[dfp-1547]: https://github.com/apache/datafusion-python/pull/1547
+[dfp-1677]: https://github.com/apache/datafusion-python/pull/1677
+[dfp-1678]: https://github.com/apache/datafusion-python/pull/1678
+[dfp-1759]: https://github.com/apache/datafusion-python/pull/1759
+[#2588]: https://github.com/apache/datafusion-ballista/pull/2588
+[#2619]: https://github.com/apache/datafusion-ballista/pull/2619
+
+This is a good example of how the DataFusion ecosystem compounds. Extension
points built to make DataFusion
+Python more composable turned out to be exactly what a distributed engine
needed to support Python natively.
+
+## Trying it out
+
+Python UDF support is on the `main` branch of Ballista and will be included in
the next release of the
+`ballista` Python package. Until then, you can build the package from source
by following the instructions in
+the [Python client README]. The [example script] starts a local test cluster
and walks through both the
+DataFrame and SQL forms shown above.
+
+Progress on the Python client is tracked in [#2618]. If you try it out, we
would love to hear how it goes.
+Please report bugs and feature requests on [GitHub], or join the conversation
on the
+[DataFusion communication channels].
+
+[Python client README]:
https://github.com/apache/datafusion-ballista/blob/main/python/README.md
+[example script]:
https://github.com/apache/datafusion-ballista/blob/main/python/examples/ballista_execution.py
+[#2618]: https://github.com/apache/datafusion-ballista/issues/2618
Review Comment:
This needs to be changed to
https://github.com/apache/datafusion-ballista/issues/2781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]