[
https://issues.apache.org/jira/browse/SPARK-60915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60915:
------------------------------------
Assignee: Hyukjin Kwon
> Send the call sites of DataFrame API calls in Spark Connect Rust, so that
> errors carry DataFrame query contexts as in PySpark
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60915
> URL: https://issues.apache.org/jira/browse/SPARK-60915
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect records, for each expression that a function of
> {{pyspark.sql.functions}} or a {{Column}} operator builds, the name of the
> function and its call site in user code ({{_with_origin}} and
> {{with_origin_to_class}}), and sends them as the expression's
> {{python_origin}}. For an error in that expression, the server then reports a
> {{DataFrameQueryContext}} whose {{fragment()}} is the function and whose
> {{callSite()}} is the {{file:line}} of the user code.
> {{spark.python.sql.dataFrameDebugging.enabled}}, true by default, turns this
> on. The Rust-backed Python client sends no origins, so these PySpark 4.3.0
> results differ:
> * For {{spark.range(1).select("no_such_col").collect()}}, and for
> {{select(F.col("nope") + 1)}}, {{getQueryContext()}} returns a
> {{DataFrameQueryContext}} with the fragment {{col}} and the call site of the
> {{select}}. The client returns none.
> * For {{select(F.lit(1) / F.lit(0))}}, it returns one with the fragment
> {{__truediv__}}. The client returns none.
> Record the origins and send them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
