andygrove opened a new issue, #7252:
URL: https://github.com/apache/datafusion-comet/issues/7252
### What is the problem the feature request solves?
A `GenerateExec` whose generator is user-defined falls back to Spark. This
covers Hive `GenericUDTF`s (`CREATE FUNCTION ... AS 'com.example.MyUDTF'` used
in `LATERAL VIEW`, planned as `HiveGenericUDTF`) and custom Catalyst
`Generator` expressions injected through session extensions. The fallback costs
a columnar-to-row transition before the generator and a row-to-columnar
transition after it, and the rest of the stage, often an aggregate over the
generated rows, runs in Spark.
The generator would still run on the JVM one row at a time, so the generator
itself would not get faster. The gain is keeping the stage native around it.
Comet already runs scalar `ScalaUDF`s in the native pipeline through codegen
dispatch, but that path rejects any `Generator`
(`CometBatchKernelCodegen.canHandle`), because its kernel contract is one
output value per input row.
### Describe the potential solution
A native generate operator that calls back into the JVM through the existing
UDF bridge:
1. Native code sends an input batch. The JVM side evaluates the bound
generator's interpreted `eval` once per row, reading through the Arrow-backed
row getters that codegen dispatch already uses, and writes the generated rows
into Arrow vectors with the existing output writers.
2. The JVM returns the generated columns plus a parent-row index for each
output row. Native code builds the output by running `take` on the
carried-forward columns with those indices and appending the generated columns.
`outer` (one row of NULLs when the generator yields nothing) is an index with a
null generated row.
3. The bridge gains two lifecycle pieces it lacks today:
- per-partition generator state that lives across batches, with
`Nondeterministic.initialize(partitionIndex)`, as the per-plan kernel cache
already does for `Rand`;
- an end-of-input call so `terminate()` can emit its rows. `GenerateExec`
pairs those rows with the carried-forward columns of the partition's last input
row, matching Hive, so native code must keep that row.
`HiveGenericUDTF` is `CodegenFallback` and converts through its own
`ObjectInspector`s inside `eval`, so it needs nothing Hive-specific beyond what
`eval` already does.
### Additional context
- Python UDTFs (`@udtf`) plan as their own Python UDTF exec nodes, not
`GenerateExec`, and belong with the Arrow Python UDF work (#5386, #5123).
- #7251 (for #5125) runs `inline` and `stack` natively. The
generated-columns-plus-parent-index shape here could also replace the list
unnesting those use, though `ExplodeExec` already handles them well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
