Holden Karau created SPARK-59880:
------------------------------------
Summary: Broaden transpilation input type categories to timestamp,
map, and array
Key: SPARK-59880
URL: https://issues.apache.org/jira/browse/SPARK-59880
Project: Spark
Issue Type: Sub-task
Components: PySpark, SQL
Affects Versions: 4.4.0
Reporter: Holden Karau
`_annotation_category` knows `str`, `int`, `float`, `bool`, `bytes`. Anything
else comes back `None`, which means untyped, so `_param_category_combos` tries
the parameter as numeric *and* string and `_transpile_func` tags each option
with
that guess. `optionMatchesTypes` is strict and ends in `case _ => false`, so a
`datetime` parameter gets matched `("numeric", TimestampType)` and
`("string", TimestampType)`, both fail, every option is pruned, and the Python
UDF stays.
**Fix.** Teach the categories the types a variety can lower, and leave the
matcher strict:
- `python/pyspark/sql/transpile.py:937` `_annotation_category` — `datetime` →
`"timestamp"`, `dict[str, str]` → `"map"`, `list[X]` → `"array"`. Pinning them
also shrinks the combo matrix, since they stop counting as untyped.
- `python/pyspark/sql/transpile.py:960` `_param_category_combos` — nothing to do
once the annotations resolve; the `untyped > 3` collapse just fires less.
- `.../catalyst/analysis/ResolveTranspiledPythonUDFOptions.scala:95`
`optionMatchesTypes` — add the arms, keep `case _ => false`.
Granularity is the actual decision. `"array"` matching any `ArrayType` lets an
`array<string>` lowering match an `array<int>` column; `"array<string>"` is
precise and starts putting a type grammar inside a category string. J2 makes the
question go away, and then this ticket is three lines in `_annotation_category`.
We should add one UDF per new category in `test_udf_transpile_unit.py`,
asserting the
option survives against a matching column type and is pruned against a
non-matching one. The built-in transpiler still refuses these bodies, so assert
on `_transpiled_input_categories` or stand up a stub variety.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
