Hyukjin Kwon created SPARK-60915:
------------------------------------
Summary: Send the call sites of DataFrame API calls in Spark
Connect Rust, so that errors carry DataFrame query contexts as in PySpark
Key: SPARK-60915
URL: https://issues.apache.org/jira/browse/SPARK-60915
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect records, for each expression that a function of
{{pyspark.sql.functions}} or a {{Column}} operator builds, the name of the
function and its call site in user code ({{_with_origin}} and
{{with_origin_to_class}}), and sends them as the expression's
{{python_origin}}. For an error in that expression, the server then reports a
{{DataFrameQueryContext}} whose {{fragment()}} is the function and whose
{{callSite()}} is the {{file:line}} of the user code.
{{spark.python.sql.dataFrameDebugging.enabled}}, true by default, turns this
on. The Rust-backed Python client sends no origins, so these PySpark 4.3.0
results differ:
* For {{spark.range(1).select("no_such_col").collect()}}, and for
{{select(F.col("nope") + 1)}}, {{getQueryContext()}} returns a
{{DataFrameQueryContext}} with the fragment {{col}} and the call site of the
{{select}}. The client returns none.
* For {{select(F.lit(1) / F.lit(0))}}, it returns one with the fragment
{{__truediv__}}. The client returns none.
Record the origins and send them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
