[
https://issues.apache.org/jira/browse/SPARK-60723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60723.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59931
[https://github.com/apache/spark/pull/59931]
> Settle version 1 of the native data source interface before 4.4.0
> -----------------------------------------------------------------
>
> Key: SPARK-60723
> URL: https://issues.apache.org/jira/browse/SPARK-60723
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> A follow-up to SPARK-60101 (https://github.com/apache/spark/pull/59424).
> Version 1 of spark_datasource.h first ships in 4.4.0, so the changes from the
> review in https://github.com/apache/spark/pull/59424#issuecomment-6480975990
> that change existing functions or the meaning of a flag have to land before
> then:
> - read and create_data_writer get size-prefixed SparkDsReadInfo and
> SparkDsDataWriterInfo structs with the schema, and read also gets whether the
> read is batch or streaming.
> - SPARK_DS_TRUNCATE is split into SPARK_DS_BATCH_TRUNCATE and
> SPARK_DS_STREAMING_TRUNCATE.
> - partitions, stream_partitions and create_writer return the state of the
> reader or the writer, replacing serialize_reader, serialize_stream_reader and
> serialize_writer.
> - create_stream_reader and latest_offset take size-prefixed structs, with the
> checkpoint location and the start offset, so that admission control can be
> added later.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
