[
https://issues.apache.org/jira/browse/SPARK-60986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60986:
-----------------------------------
Labels: pull-request-available (was: )
> Take datetime.timedelta values in lit() and createDataFrame in Spark Connect
> Rust as PySpark does
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-60986
> URL: https://issues.apache.org/jira/browse/SPARK-60986
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark takes a {{datetime.timedelta}} as a value of {{DayTimeIntervalType}}
> (INTERVAL DAY TO SECOND), of {{floor(total_seconds())}} seconds and its
> microseconds: {{F.lit(datetime.timedelta(days=1, seconds=2))}} is such a
> literal, and {{createDataFrame([(datetime.timedelta(seconds=1),)])}} infers
> {{interval day to second}}. The Rust-backed Python client raises
> {{UNSUPPORTED_DATA_TYPE}} for the literal and {{TypeError: unsupported value
> type for createDataFrame: timedelta}} for the rows. Take a {{timedelta}} as
> PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
