[
https://issues.apache.org/jira/browse/SPARK-61496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-61496.
---------------------------------
Resolution: Fixed
Issue resolved by pull request 543
[https://github.com/apache/spark-connect-rust/pull/543]
> Type the join type and direction of the as-of join in the Rust client, and
> set an Observation whose relation has sent copies
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61496
> URL: https://issues.apache.org/jira/browse/SPARK-61496
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Two code-quality findings in crates/spark-connect/src/plan.rs of
> spark-connect-rust.
> 1. The AsOfJoin plan node keeps its join type and its direction as strings.
> Give them enums in the core (AsOfJoinType: inner, left outer;
> AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf
> checks neither `how` nor `direction` and sends them as they are, and the
> server reads more spellings than these names (any case, underscores) and
> refuses the rest in the analysis, so any other string is kept and sent as it
> is: no request changes, and the client raises no error that PySpark does not.
> 2. PlanIds::node_of returns whichever node of a plan ID the scan of its map
> meets first. A plan ID is of more than one node whenever a relation that the
> session sent has a DataFrame of its own (DataFrame::from_sent_relation), such
> as an unpickled copy of a DataFrame. When the scan meets the copy, the
> observed metrics of that plan ID are dropped, and Observation.get of an
> observed DataFrame that has such a copy returns nothing. Give every live node
> of the ID, and set the Observation from the one that observes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]