linhongliu-db opened a new pull request, #57509: URL: https://github.com/apache/spark/pull/57509
### What changes were proposed in this pull request? This PR propagates `SPARK_TRACEBACK_WITH_LOCALS` from `PythonPlannerRunner` when `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled. It also adds a UDTF `analyze` regression test that raises from the planner-side Python worker and verifies the surfaced traceback includes the local variable. ### Why are the changes needed? `PythonPlannerRunner` already reads `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled`, but it did not add `SPARK_TRACEBACK_WITH_LOCALS` to the Python worker environment. As a result, planner-driven Python paths such as UDTF `analyze` did not honor the traceback-locals config, unlike regular Python UDF execution. ### Does this PR introduce _any_ user-facing change? Yes. When `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled, Python exceptions raised through planner-driven Python paths can now include local variables in their tracebacks. ### How was this patch tested? Passed: ``` python3 -m py_compile python/pyspark/sql/tests/test_udtf.py python3 -m py_compile python/pyspark/sql/tests/test_udf.py git diff --check ``` Attempted but not run locally because the Spark assembly could not be built in this environment: ``` ./build/sbt assembly/package JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 ./build/mvn -DskipTests -pl :spark-assembly_2.13 -am package python3 python/run-tests.py --testnames 'pyspark.sql.tests.test_udtf UDTFTests.test_udtf_analyze_traceback_with_locals' python3 python/run-tests.py --testnames 'pyspark.sql.tests.test_udf UDFTests.test_udf_traceback_with_locals' ``` The SBT launcher fetch failed, and Maven dependency resolution failed because Maven Central hostnames did not resolve from this local environment. The PySpark test runner then failed at startup with `Cannot find assembly build directory, please build Spark first.` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
