[
https://issues.apache.org/jira/browse/SPARK-61019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61019:
---------------------------------
Description:
In PySpark, {{df.explain(extended, mode)}} prints
{{df._explain_string(extended, mode)}} with Python's {{print}}, so that the
plan is written to {{sys.stdout}}, which a notebook or
{{contextlib.redirect_stdout}} captures, and
{{StreamingQuery.explain(extended)}} prints its plan and returns None.
{{_explain_string}} raises {{CANNOT_SET_TOGETHER}} for both {{extended}} and
{{mode}}, {{NOT_EXPECTED_TYPE}} for an argument that is neither a bool nor a
string, and {{VALUE_NOT_ALLOWED}} for a mode other than 'simple', 'extended',
'codegen', 'cost' and 'formatted'.
The Rust-backed Python client's {{explain}} writes the plan to the standard
output of the process from Rust, which neither a notebook nor
{{redirect_stdout}} captures, and {{StreamingQuery.explain}} returns the plan
as a string. {{_explain_string}} does not exist. {{explain(mode="FORMATTED")}}
is accepted, {{explain(mode="bogus")}} raises a PySparkValueError without an
error class, and {{explain(True, "simple")}} and {{explain(1)}} raise a
ValueError and a TypeError. Do as PySpark does.
Also, an error of a PySpark error class that the client raises before it calls
the server, such as the {{VALUE_NOT_ALLOWED}} of {{explain}} or the
{{CANNOT_BE_EMPTY}} of {{spark.tvf.json_tuple(col)}} without fields, is raised
with only its message: its {{getCondition()}} and {{getMessageParameters()}}
are None. Raise it with its error class and message parameters as PySpark does.
was:
In PySpark, {{df.explain(extended, mode)}} prints
{{df._explain_string(extended, mode)}} with Python's {{print}}, so that the
plan is written to {{sys.stdout}}, which a notebook or
{{contextlib.redirect_stdout}} captures, and
{{StreamingQuery.explain(extended)}} prints its plan and returns None.
{{_explain_string}} raises {{CANNOT_SET_TOGETHER}} for both {{extended}} and
{{mode}}, {{NOT_EXPECTED_TYPE}} for an argument that is neither a bool nor a
string, and {{VALUE_NOT_ALLOWED}} for a mode other than 'simple', 'extended',
'codegen', 'cost' and 'formatted'.
The Rust-backed Python client's {{explain}} writes the plan to the standard
output of the process from Rust, which neither a notebook nor
{{redirect_stdout}} captures, and {{StreamingQuery.explain}} returns the plan
as a string. {{_explain_string}} does not exist. {{explain(mode="FORMATTED")}}
is accepted, {{explain(mode="bogus")}} raises a PySparkValueError without an
error class, and {{explain(True, "simple")}} and {{explain(1)}} raise a
ValueError and a TypeError. Do as PySpark does.
> Print the plans of explain and StreamingQuery.explain with Python's print,
> and raise the errors of explain, in the Python client as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61019
> URL: https://issues.apache.org/jira/browse/SPARK-61019
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{df.explain(extended, mode)}} prints
> {{df._explain_string(extended, mode)}} with Python's {{print}}, so that the
> plan is written to {{sys.stdout}}, which a notebook or
> {{contextlib.redirect_stdout}} captures, and
> {{StreamingQuery.explain(extended)}} prints its plan and returns None.
> {{_explain_string}} raises {{CANNOT_SET_TOGETHER}} for both {{extended}} and
> {{mode}}, {{NOT_EXPECTED_TYPE}} for an argument that is neither a bool nor a
> string, and {{VALUE_NOT_ALLOWED}} for a mode other than 'simple', 'extended',
> 'codegen', 'cost' and 'formatted'.
> The Rust-backed Python client's {{explain}} writes the plan to the standard
> output of the process from Rust, which neither a notebook nor
> {{redirect_stdout}} captures, and {{StreamingQuery.explain}} returns the plan
> as a string. {{_explain_string}} does not exist.
> {{explain(mode="FORMATTED")}} is accepted, {{explain(mode="bogus")}} raises a
> PySparkValueError without an error class, and {{explain(True, "simple")}} and
> {{explain(1)}} raise a ValueError and a TypeError. Do as PySpark does.
> Also, an error of a PySpark error class that the client raises before it
> calls the server, such as the {{VALUE_NOT_ALLOWED}} of {{explain}} or the
> {{CANNOT_BE_EMPTY}} of {{spark.tvf.json_tuple(col)}} without fields, is
> raised with only its message: its {{getCondition()}} and
> {{getMessageParameters()}} are None. Raise it with its error class and
> message parameters as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
