[
https://issues.apache.org/jira/browse/SPARK-60949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60949.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 186
[https://github.com/apache/spark-connect-rust/pull/186]
> Access struct fields as attributes of a Column, and slice a Column, in Spark
> Connect Rust as PySpark does
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60949
> URL: https://issues.apache.org/jira/browse/SPARK-60949
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, an attribute that a Column does not have is the field of
> that name ({{Column.__getattr__}} returns {{self[item]}}): {{F.col("a").b}}
> is {{Column<'a['b']'>}}, and {{F.col("a").b.c}} is {{Column<'a['b']['c']'>}},
> so that {{df.person.name}} selects a field of a struct column. A name that
> starts with {{__}} raises {{PySparkAttributeError}}
> {{ATTRIBUTE_NOT_SUPPORTED}}, and {{_jc}} raises
> {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
> The Rust-backed Python client raises {{AttributeError}} for every such
> attribute, so {{df.select(F.col("s").field)}} fails. Resolve the attributes
> that a Column does not have as PySpark does.
> {{Column.__getitem__}} also takes a slice as {{substr}}: {{F.col("a")[1:3]}}
> is {{Column<'substr(a, 1, 3)'>}}, and a slice with a step raises
> {{PySparkValueError}} {{SLICE_WITH_STEP}}. {{Column.substr}} raises
> {{PySparkTypeError}}: {{NOT_SAME_TYPE}} for arguments of different types,
> such as {{substr(1, F.lit(2))}} or the slice {{[:3]}}, and
> {{NOT_EXPECTED_TYPE}} for other than a Column or an int. The client raises
> {{TypeError}} for a slice, and builds {{substr}} of any arguments. Slice a
> Column, and check the arguments of {{substr}}, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
