[
https://issues.apache.org/jira/browse/SPARK-60858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60858.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 146
[https://github.com/apache/spark-connect-rust/pull/146]
> Close remaining Spark 4.3.0 client API gaps in the Rust drop-in
> ---------------------------------------------------------------
>
> Key: SPARK-60858
> URL: https://issues.apache.org/jira/browse/SPARK-60858
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Follow-up to SPARK-59305, which retargeted spark-connect-rust to Apache Spark
> 4.3.0. Close the remaining 4.3.0 client API gaps in the Rust-backed drop-in:
> * Nanosecond timestamp types: map the TimestampNTZNanos / TimestampLTZNanos
> proto kinds to their own DataTypes (instead of TIME / microsecond TIMESTAMP),
> with proto, JSON, DDL and Arrow support and picklable Python types.
> * functions: instr(occurrence=...) without start searches from position 1,
> and None means omitted for instr / regexp_replace; variant_delete() with no
> paths raises CANNOT_BE_EMPTY.
> * SparkSession.__enter__ / __exit__ (exit stops the session); stop() is
> idempotent and best-effort, and clears the active session and the cached
> SQLContext.
> * Spark Connect SQLContext / HiveContext (pyspark.sql.connect.context), with
> SQLContext.getOrCreate() routing to it.
> * Pipelines create_auto_cdc_flow: SCD type 2, track_history_column_list /
> track_history_except_column_list and spark_conf, with the 4.3.0 client-side
> validation.
> * operation_id: ExecutionInfo.operation_id (DataFrame.executionInfo returns a
> pyspark.sql.metrics.ExecutionInfo) and SparkConnectException.operation_id.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
