holdenk opened a new pull request, #59128:
URL: https://github.com/apache/spark/pull/59128
### What changes were proposed in this pull request?
Adjusts the three tests SPARK-58626 added that assert master's behavior for
a transpiled UDF call inside a higher-order function's lambda, so they assert
branch-4.3's behavior instead: analysis rejects the query with
`UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF` — the same error the
query gets with transpilation off.
- `test_udf_transpile_declines_inside_a_higher_order_function` and the
lambda half of `test_udf_transpile_position_can_rule_out_sharing`
(`test_udf_transpile_unit.py`)
- `TranspiledUDFParameterSuite.keeps the Python UDF inside a higher-order
function's lambda`
Expecting the rejection is itself the assertion that matters on this branch:
a call lowered inside the lambda would analyze and run, so pinning the error
pins "not lowered".
### Why are the changes needed?
The SPARK-58626 cherry-pick (#58189) broke the branch-4.3 build. The three
tests expect the query to *run* — which it does on master because SPARK-27052
("Support Python UDFs inside higher-order function lambdas") rewrites the
lambda to apply the UDF over the whole array. SPARK-27052 is master-only; on
branch-4.3 `CheckAnalysis` rejects any Python UDF inside a higher-order
function's lambda, transpiled or not, deterministically.
The branch build looked intermittent only because `dev/run-tests` skips
unchanged modules on some pushes ("There are no modules to test, exiting
without testing"), so those pushes went green without running these tests.
The alternative fix is backporting SPARK-27052 to branch-4.3. That is a
feature backport (new config, analyzer/optimizer rules) and a
release-management call, not something to fold into a build fix.
Two adjacent notes, not addressed here:
- branch-4.x is NOT affected: SPARK-27052 and its follow-ups (SPARK-58695,
SPARK-58728) are all on branch-4.x, and the two test files plus
`CheckAnalysis.scala` are byte-identical to master there. Only branch-4.3 lacks
the lambda support.
- The branch-4.3 build has a second, unrelated red job:
`UdfSerializationSuite` (Decimal serialVersionUID audit, SPARK-59708 area).
- Pre-existing stale comments reference the master-only
`ExtractPythonUDFFromLambda` rule on this branch (`Optimizer.scala` and
`ConvertToCatalystSuite.scala`); left alone here as out of scope.
### Does this PR introduce _any_ user-facing change?
No this is test-only (and transpilation has not yet been released either
way). User-facing behavior on branch-4.3 is unchanged for this shape: the query
fails with the same error whether transpilation is on or off.
### How was this patch tested?
On branch-4.3 (Java 21):
- `python -u -m unittest pyspark.sql.tests.test_udf_transpile_unit` — 82
tests, OK
- `build/sbt -Phive 'sql/testOnly *TranspiledUDFParameterSuite'` — 10 tests,
OK
- `RUN_HYPOTHESIS=1 RUN_HYPOTHESIS_MAX_EXAMPLES=200 python -u -m unittest
pyspark.sql.tests.test_udf_transpile_hypothesis` — OK
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (kimi-k3)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
